{"slug":"architecture/RUNTIME_SUBSTRATE_AND_SURFACE_CONTRACT","title":"Runtime Substrate and Workbench Surface Contract","section":"architecture","sectionLabel":"Architecture","status":"unclassified","type":"document","path":"architecture/RUNTIME_SUBSTRATE_AND_SURFACE_CONTRACT.mdx","bytes":20478,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n**Status:** Derived implementation contract — required for platform completion; does not supersede frozen architecture\n\n**Date:** 2026-07-28\n**Authority:** `ARCHITECTURE_LOCK.md` and `IMPLEMENTATION_CONTRACT_v1.1.md`\n\n## 1. Binding rules\n\n1. **Schemas are pure entity-based.** Persisted records represent a business or\n   platform entity, relationship, event, evidence artifact, policy, capability,\n   execution, or versioned definition. They are never created merely because a\n   sidebar label exists. There is no `home`, `inbox`, `work-view-1`, or\n   `finance-manager` source-of-truth table.\n2. **Projections are RBAC-based.** Gateway derives tenant, membership, role,\n   permissions, policy, entitlement and permitted connector context from the\n   authenticated identity. Projection Engine then applies role, department,\n   industry, data availability, freshness and user preference. The browser may\n   request a view; it cannot select a broader data scope.\n3. A Workbench surface never maps one-to-one to a physical table. It resolves a\n   versioned projection contract through Gateway and Projection Engine.\n4. A physical table may be a **primary source**, but every projection may use\n   multiple supporting sources, permission rules, connector availability and\n   capability state.\n5. Product never reads or writes platform tables directly. A mutation is a\n   semantic command through Gateway → Governance → Hermes where required →\n   Pipeline promotion → projection invalidation.\n6. The following runtime guarantees are mandatory Base Spine/platform\n   capabilities: event transport, idempotency, capability resolution,\n   discovery, queues, retry/replay, sync/reconciliation, observability, cache\n   discipline and object/evidence storage. Industry and connector extensions\n   may add mappings; they must not redefine these guarantees.\n7. The Twin is silent by default. A conversational intake may be offered in a\n   contextual surface, but it neither becomes a mandatory chat panel nor writes\n   truth directly. Submitted content is triaged, evidenced, governed where\n   required, then promoted through the Pipeline.\n\n## 2. Runtime substrate\n\n```mermaid\nflowchart LR\n    I[\"Ingress: webhook, poll, stream, API, MCP, user intent\"] --> V[\"Validate + archive raw artifact\"]\n    V --> E[\"Immutable event + idempotency/deduplication\"]\n    E --> B[\"Event bus / stream / queue\"]\n    B --> D[\"Discovery and capability resolution\"]\n    D --> G[\"Governance\"]\n    G --> H[\"Hermes / worker runtime\"]\n    H --> P[\"Provider or local execution\"]\n    P --> O[\"Outcome event\"]\n    O --> S[\"Sync, verify and reconcile\"]\n    S --> W[\"Pipeline promotion to Spine\"]\n    W --> X[\"Projection invalidation and Workbench\"]\n```\n\nThe Projection Engine and Workbench sit above this substrate. They can observe\nits state through semantic projections; they do not manage queue delivery,\nstream offsets, provider execution or retries directly.\n\n## 3. Runtime guarantees and table-status map\n\nThe table names in the **Required completion records** column are the target\ncontract. A name is not evidence that a migration exists. The migration-035\nstatus below is a repository reading, not a live database verification.\n\n| Runtime guarantee             | Existing migration-035 evidence                                                               | Required completion records / service state                                                     | Release rule                                                                                                                                   |\n| ----------------------------- | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |\n| Event bus and Pub/Sub         | `spine_event_types`, `spine_events`, topics, subscriptions, deliveries, offsets, dead letters | publisher registry, subscription filters, per-tenant ordering and replay policy                 | Immutable envelope, delivery attempt tracking, isolated subscriber failure and poison-event quarantine.                                        |\n| Idempotency and deduplication | event `idempotency_key`; sync mutation idempotency key                                        | `spine_idempotency_keys`, deduplication records, operation fingerprints                         | Same logical operation + scope + key produces one accepted effect. Provider timeout is reconciled before a retry.                              |\n| Capability matrix             | capabilities, versions, provider/policy bindings, runs and steps                              | implementation, requirement, role, connector and availability matrix records                    | Resolve capability by tenant/workspace/role/connector/auth/data/policy; installation expands implementations, never redefines base capability. |\n| Discovery                     | discovery runs, assets, capability candidates, domain manifests                               | snapshots, discovered objects/fields/events/operations, classification and extension candidates | Discovery describes and maps a provider. It never mutates physical schema automatically.                                                       |\n| Queues and workers            | event deliveries and continuity retries; Cloudflare runtime bindings are outside schema       | runtime queues, message/lease/attempt/dead-letter metadata and queue policies                   | User work queues and platform execution queues are separate planes.                                                                            |\n| Streaming                     | event consumer offsets                                                                        | stream/partition/checkpoint/consumer-group metadata where a provider needs ordered replay       | Use for ordered/high-volume changes; do not confuse it with Pub/Sub fan-out.                                                                   |\n| Webhook ingress               | integration webhook endpoints                                                                 | verified deliveries, signatures, classified events and failure records; raw payload in R2       | Verify signature, archive raw payload, check replay/idempotency, then publish canonical event.                                                 |\n| Polling ingress               | connector cursors and sync runs                                                               | poll definitions, schedules, leases, results/failures                                           | Advance a cursor only after emitted events are durably accepted.                                                                               |\n| Sync and reconciliation       | sync runs, mutations, conflicts, verifications                                                | definitions, checkpoints, pending mutations, dead letters and reconciliation records            | An API success is not closure; re-ingest, compare expected/actual, reconcile, then close.                                                      |\n| Replay and reprocessing       | delivery dead letters and continuity retries                                                  | reprocess request/run/item, replay checkpoint and dead-letter resolution                        | Replay is a governed operation with dry-run/execution mode, target consumers, side-effect policy and verification method.                      |\n| Logging and traceability      | audit log, event correlation/causation, entity trace/correlation fields                       | structured log references, runtime errors, trace spans, causation/lineage records               | Redacted, tenant-aware, queryable and retention-controlled. Logs are not audit lineage.                                                        |\n| Observability and health      | connection health and selected runtime state                                                  | service health, health checks, metrics, SLOs, incidents, alert rules/deliveries                 | Cover infrastructure plus connector, queue, capability, sync and projection freshness.                                                         |\n| Cache discipline              | projection snapshots; D1/KV bounded-state contract                                            | cache namespace, invalidation rule, refresh and health metadata                                 | Cache is reproducible, versioned and non-authoritative; never cache secrets.                                                                   |\n| Object/evidence store         | evidence links and object references                                                          | blob/version/link/retention/access/integrity metadata, with bytes in R2                         | Object store owns bytes; Neon owns metadata/ownership; knowledge and vector indexes are derived.                                               |\n\n## 4. Canonical runtime envelope\n\nEvery ingress path normalizes to an envelope equivalent to this contract before\nit enters the event bus or queue:\n\n```ts\ninterface SpineEventEnvelope<T = unknown> {\n  eventId: string;\n  eventType: string;\n  eventVersion: string;\n  tenantId: string;\n  workspaceId?: string;\n  source: {\n    type: \"webhook\" | \"poll\" | \"stream\" | \"api\" | \"mcp\" | \"user\" | \"agent\" | \"system\";\n    provider?: string;\n    connectionId?: string;\n    externalEventId?: string;\n  };\n  subject: { entityType?: string; entityId?: string; externalEntityId?: string };\n  traceId: string;\n  correlationId: string;\n  causationId?: string;\n  occurredAt: string;\n  receivedAt: string;\n  idempotencyKey: string;\n  payload: T;\n  provenance: { rawPayloadRef?: string; evidenceRefs: string[]; schemaVersion: string };\n}\n```\n\n`traceId` covers the end-to-end operation, `correlationId` groups related work,\nand `causationId` identifies the direct parent command/event. These identifiers\nmust flow from user intent or ingress through approval, capability execution,\nprovider response, re-ingestion, verification and final Spine mutation.\n\n## 5. Reliability policy: until successful, safely\n\n“Until successful” means retry until success, terminal non-retryable state,\npolicy expiration, dead letter, or required human intervention. It never means\nan unbounded blind retry loop.\n\n```text\naccepted → idempotency reserved → attempt\nattempt → succeeded → verify → reconcile → close\nattempt → retryable failure → backoff → scheduled attempt\nattempt → ambiguous provider outcome → query/reconcile before retry\nattempt → terminal failure | policy expired | retries exhausted → dead letter / escalation\n```\n\nEvery retry policy needs maximum attempts and elapsed time, backoff/jitter,\nretryable versus terminal errors, circuit-breaker thresholds, dead-letter\nhandling, and human escalation conditions. Provider mutations such as payment,\nemail, provisioning, deployment or record update must not be repeated until\nthe remote effect is proven absent.\n\n## 6. Adaptive Spine expansion law\n\nThe Spine adapts to **more** data without flattening the base model or forcing\nthe Product to hard-code provider schemas:\n\n```text\nBase schema module\n→ connector discovery\n→ versioned provider schema\n→ approved mapping to canonical entity/relationship/field definition\n→ additive domain extension candidate where no canonical mapping exists\n→ governed approval and migration where physical expansion is required\n→ pipeline promotion, provenance and projection recompilation\n```\n\nThe Base Spine remains the minimum standard: identity, entity, work, event,\nknowledge, evidence, memory, governance, capability and continuity. A new\nindustry, role, department or connector can add fields, entity subtypes,\nrelationships, projections and capabilities, but cannot weaken tenant\nisolation, governance, provenance, idempotency or the Pipeline-writer law.\n\n## 7. Surface backend contract\n\n```ts\ninterface SurfaceBackendContract {\n  navId: string;\n  projectionId: string;\n  primarySources: string[];\n  supportingSources: string[];\n  requiredContext: [\"tenant\", \"membership\", \"role\", \"policy\", \"freshness\"];\n  commands: Array<{ capability: string; approval: \"none\" | \"policy\" | \"human\" }>;\n  runtime?: {\n    publishTopics?: string[];\n    consumeTopics?: string[];\n    queue?: string;\n    syncDefinition?: string;\n    idempotencyScope?: string;\n    reprocessingPolicy?: string;\n  };\n}\n```\n\n### Shared navigation and changing work surface\n\nThe sidebar structure is common. What changes is the **Work** projection:\n\n```text\nBase manifest + active tenant + role + department + industry\n+ connector/schema availability + policy + user preference\n→ Projection Engine → permitted Workbench surface\n```\n\nThe twelve role/domain projections are Workbench views—not user-facing “L1”\nlabels. A view definition stores filters, grouping, layout and capabilities; it\ndoes not store the operational truth displayed in that view.\n\n### Canonical sidebar shell\n\nThe following navigation shell is common across the Product. RBAC determines\nwhether a user sees an item and whether it is read-only, operational, or\nadministrative. The **Work** branch is the adaptive branch: its actual views\nare composed from role × department × industry × tenant policy × connector\navailability, rather than being hard-coded per provider or user.\n\n```text\nIW\n├── Home\n├── My Work\n│   ├── Inbox\n│   ├── Tasks\n│   ├── My Queue\n│   ├── Approvals\n│   ├── Notifications\n│   ├── Recent Work\n│   └── Favorites\n├── Communications\n│   ├── Activity Feed\n│   ├── Email\n│   ├── Calendar\n│   ├── Meetings\n│   ├── Chat\n│   └── Calls\n├── Documents Shared\n├── Personal\n│   ├── Notes\n│   ├── Knowledge Base\n│   └── Brainstorming Sessions\n├── Activity\n├── Finance Manager\n├── Work\n│   └── Adaptive views: role × department × industry\n│       ├── View 1\n│       └── View N\n├── Platform\n│   ├── Integration Manager\n│   ├── Connector Marketplace\n│   ├── Activation Hub\n│   ├── Capability Manager\n│   ├── MCP Pool (internal Platform operators only)\n│   ├── AI Models\n│   ├── Knowledge\n│   ├── Memory & Continuity\n│   ├── Automation\n│   ├── Pipelines\n│   ├── Governance\n│   └── Observability\n└── Administration\n    ├── Organizations\n    ├── Users\n    ├── Roles\n    ├── Permissions\n    ├── Audit\n    ├── Billing\n    ├── Deployments\n    ├── API Keys\n    └── Security\n```\n\n`Chat` is a communication or contextual-intake surface. It does not turn the\nSilent Twin into a permanent sidebar destination, and any contextual Twin\nengagement still uses the Four Buttons and governed Pipeline paths.\n\n### Surface groups\n\n| Group                                           | Primary projection responsibility                        | Typical supporting truth                                                                                                |\n| ----------------------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |\n| Home                                            | Composite operational summary                            | events, work, approvals, notifications, calendar, timeline, readiness.                                                  |\n| My Work / Inbox / Tasks / Queue                 | Prioritized work and attention                           | work items, queue membership read model, messages, approvals, notifications, exceptions.                                |\n| Communications                                  | Human and provider communication continuity              | messages, threads, calendar, meetings, calls, evidence and entity links.                                                |\n| Documents / Notes / Knowledge                   | Document, evidence and derived knowledge access          | blobs, documents, shares, evidence, knowledge/chunks, retrieval indexes.                                                |\n| Personal                                        | User-scoped preferences, favorites and permitted context | memberships, favorites, tuning profile, notifications, permitted memory.                                                |\n| Brainstorming                                   | Domain extension, not a replacement chat history         | session, messages, canvas state, ideas/insights, decisions, promoted knowledge.                                         |\n| Organizational Memory / Company Wiki            | First-class operating handbook projection                | promoted memory, knowledge pages, policies, rituals, decisions, linked subpages, embedded views, and approved outcomes. |\n| Work                                            | Role/department/industry operational projection          | canonical entities, work, signals, relationships, capabilities and governance.                                          |\n| Integration / Activation                        | Tenant connectivity and readiness                        | catalog, installation/connection, discovery, schemas, webhooks, sync, health, activation/hydration.                     |\n| Capability / MCP / AI                           | Permitted technical capability availability              | registry, implementations, bindings, policy, health and runs. MCP Pool remains internal.                                |\n| Knowledge / Continuity / Automation / Pipelines | Platform operational control views                       | memory candidates/promotions, continuity, definitions/runs, event/queue/replay state.                                   |\n| Governance / Observability                      | Accountability and health projections                    | decisions, approvals, audit, traces, health, incidents, SLOs and freshness.                                             |\n| Administration                                  | Tenant administration                                    | organisations, users, roles, permissions, billing, deployments, API-key metadata and security events.                   |\n\n### Canvas and intake\n\nCanvas views may render permitted context as cards, tables, timelines, images,\ndocuments, graph fragments and safely formatted JSON data. They are\nrepresentations of a projection, never direct database or object-store access.\n\nOrganizational memory is a canvas-capable projection surface. In practice it\nbehaves like a company wiki or operating handbook: document-like pages, linked\nsubpages, embedded operational views, and approved write-back to Spine when the\nuser and governance allow it. It is a projection over memory and knowledge, not\na second truth store.\n\nA contextual intake/chat surface may accept text, files or links. It must:\n\n1. archive the original artifact to the evidence/object-store path;\n2. create an observation/event with tenant, provenance and idempotency;\n3. have the Twin triage and propose classifications silently or on request;\n4. use governance for consequential routing or actions; and\n5. promote only validated outcomes to knowledge, work or canonical Spine truth.\n\n## 8. Delivery checkpoint\n\nBefore a testing-environment event, provide evidence for:\n\n1. one ingress-to-reconciliation trace with event, idempotency, approval,\n   execution, re-ingestion and projection invalidation identifiers;\n2. one retry/ambiguous-provider scenario that proves no duplicate side effect;\n3. one dead-letter and governed reprocessing scenario;\n4. one connector discovery-to-schema-mapping-to-projection scenario;\n5. cross-tenant and cross-environment isolation for queue, cache, object,\n   credential, webhook and projection access; and\n6. a rendered Workbench showing shared navigation plus a role-specific Work\n   projection, a canvas projection and explicit partial/degraded state.\n"}