{"slug":"architecture/integratewise-data-flow-kt","title":"Integratewise Data Flow Kt","section":"architecture","sectionLabel":"Architecture","status":"unclassified","type":"document","path":"architecture/integratewise-data-flow-kt.mdx","bytes":14946,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"Status: Canonical internal reference\nClassification: Internal\nProduct Statement: IntegrateWise is a Knowledge Workspace over the Spine, empowered by AI and governed by approvals.\n\n---\n\nDocument Control\n\nColumn 1 Column 2\nField Value\nDocument IntegrateWise Data Flow KT\nVersion v1.2\nDate 2026-03-23\nOwner IntegrateWise LLP\nAudience Product, Engineering, AI, Connector, Workflow, GTM\nScope End-to-end data flow from landing and onboarding through Spine, Workspace, Cognitive, approval, action, and re-ingestion\nCanonical runtime path Landing → Auth → L0 Onboarding → L1 Workspace → L2 Cognitive → L3 Platform\nCanonical engine loop Load → Normalize → Store → Think → Review & Approve → Act → Repeat\n\n---\n\n1. Purpose\n\nThis KT defines how data should flow through IntegrateWise so every team uses the same mental model for truth, context, intelligence, approval, and execution.\n\nThe platform rule is simple:\n\nAll durable business value in IntegrateWise must be created through a Spine-based flow.\n\nThat means every meaningful path must follow this sequence:\n\nIngest → Normalize → Store in Spine → Render in Workspace / Entity 360 → Think → Govern / HITL → Act → Re-ingest\n\n---\n\n2. Canonical Principles\n\n2.1 Spine is the only truth layer\n\nThe Spine is the canonical, tenant-partitioned source of structured truth. It holds entity identity, operational state, relationships, schema observations, and the durable data used by the workspace and cognitive layers.\n\n2.2 No data bypasses Layer 3\n\nAll external data must enter through the controlled backend path before it can appear in the workspace or cognitive layer. There are no direct UI-side truth paths and no side databases for business truth.\n\n2.3 Flow C never writes directly to Spine truth\n\nFlow A and Flow B can create truth through the pipeline. Flow C contributes governed knowledge and decision context. Truth changes from Flow C happen only after approved action and re-ingestion.\n\n2.4 Workspace is not a truth layer\n\nL1 is the operating surface, not the ingestion surface and not the source of truth. It renders Spine-backed runtime projections.\n\n2.5 No action executes without approval\n\nGovernance is a hard gate. If policy requires approval, no execution occurs without the proper approval path.\n\n---\n\n3. Runtime Architecture\n\nThe official runtime order is:\n\nExperience → Gateway → Workspace Runtime → External Connectivity → Data Plane → SSOT / Spine → Cognitive → AI Providers\n\nLayer summary\n\n- L0 Onboarding — Tenant identity, business context, schema direction, connector relevance\n- L1 Workspace — Where users work: dashboard, modules, approvals, entity views, and signals\n- L2 Cognitive — Context, IQ Hub, Evidence, Signals, Think, Act, Approval, Governance, Adjust, Audit, Agents, Twin\n- L3 Platform — Gateway, connectors, loader, pipeline, Spine, knowledge, workflow, and background processing\n\n---\n\n4. End-to-End Flow\n\nflowchart TB\nA[Landing] --> B[Auth]\nB --> C[L0 Onboarding]\nC --> D[Connector Selection and Auth]\nD --> E[Creamy Hydration]\nE --> F[Loader and Queue]\nF --> G[8-Stage Pipeline]\nG --> H[Spine Write]\nH --> I[L1 Workspace Ready]\nH --> J[Entity 360]\nJ --> K[L2 Cognitive]\nK --> L[Think]\nL --> M[Govern / HITL]\nM --> N[Act]\nN --> O[Re-ingest]\nO --> G\n\nExpanded meaning\n\n1. User lands and authenticates\n2. Onboarding seeds tenant configuration and schema direction\n3. Connector setup establishes real external bindings\n4. Creamy hydration fetches the first useful data\n5. Loader enqueues records into the data plane\n6. The 8-stage pipeline normalizes every record\n7. Spine stores canonical truth\n8. Workspace becomes usable from Spine-backed data\n9. Entity 360 fuses truth, context, and signals\n10. Cognitive services propose actions\n11. Govern and HITL gate execution\n12. Approved actions execute externally\n13. Results return through re-ingestion so truth stays current\n\n---\n\n5. Onboarding and Tenant Setup (L0)\n\nOnboarding determines:\n\n- who the tenant is\n- industry and department context\n- desired workspace mode and goals\n- connector relevance\n- schema direction\n- whether approved knowledge can be linked into Entity 360\n\nL0 output\n\nOnboarding seeds tenant runtime configuration, especially tenant_spine_config, which later drives:\n\n- workspace navigation\n- readiness\n- allowed entity types\n- depth matrix behavior\n- module visibility\n- schema routing\n\nCore rule\n\nL0 introduces reality. What is declared during onboarding determines what the system is allowed to ingest, normalize, and render later.\n\n---\n\n6. Connector Setup and Activation\n\nThe connector stage should work like this:\n\n1. Connector list is filtered by industry and department\n2. User authorizes a provider\n3. Installation is bound to tenant\n4. Connector state is persisted\n5. Initial sync starts with phase: creamy\n\nImportant point\n\nConnector choice affects scope and schema eligibility, but it does not change the core platform path.\n\nAll connectors still follow:\n\nConnect → Authenticate → Extract → Queue → Normalize → Spine → Workspace / Entity 360\n\n---\n\n7. The Three Data Flows\n\n7.1 Flow A — Structured Truth Ingestion\n\nFlow A handles structured data from systems like CRM, billing, support, project tools, and operational SaaS apps.\n\nflowchart LR\nA[Structured Connector or Webhook] --> B[Loader]\nB --> C[PIPELINE_QUEUE]\nC --> D[8-Stage Pipeline]\nD --> E[Normalizer]\nE --> F[Spine Truth]\nF --> G[Entity 360]\nF --> H[L1 Workspace Views]\n\nRole of Flow A\n\n- create canonical business truth\n- populate domain tables\n- record schema observations\n- power lists, dashboards, views, and signals\n- support write-back reconciliation after approved actions\n\nSync phases\n\n- Creamy — first useful value, fast\n- Needed — next meaningful coverage\n- Delta — continuous incremental freshness\n\nNo default full-sync-everything behavior should exist.\n\n---\n\n7.2 Flow B — Unstructured Context Integration\n\nFlow B handles documents, emails, PDFs, chats, transcripts, and related artifacts.\n\nflowchart LR\nA[Docs, Mail, PDF, Chat] --> B[Ingest]\nB --> C[Loader and Pipeline]\nC --> D[Entity Linking]\nD --> E[Spine References]\nD --> F[Knowledge Chunks and Embeddings]\nE --> G[Entity 360]\nF --> G\n\nRole of Flow B\n\n- enrich context without replacing structured truth\n- link artifacts to Spine entities\n- create evidence-backed views\n- support retrieval, IQ Hub, and grounded reasoning\n\nRule\n\nFlow B may enrich truth with references and structured metadata, but it must not overwrite canonical structured business fields with inferred content.\n\n---\n\n7.3 Flow C — AI Sessions, MCP, and Approved Memory\n\nFlow C handles AI sessions, MCP sessions, external AI content, and human-AI memory.\n\nflowchart LR\nA[MCP, AI Chat, Custom AI] --> B[D1 Buffer or Capture Layer]\nB --> C[Triage]\nC --> D{User Decision}\nD -->|Approve| E[Compounding Space]\nD -->|Discard| F[Archive]\nD -->|Defer| C\nE -. optional .-> G[Entity 360]\nC -. action path .-> H[Think -> Govern -> HITL -> Act]\nH --> I[Pipeline Re-ingest]\nI --> J[Spine]\n\nRule\n\nFlow C never writes directly to Spine truth.\n\nIt contributes knowledge, context, and decision support. If it triggers a business action, that action must go through approval, execution, and then re-enter the standard data plane before truth updates.\n\n---\n\n8. The Mandatory 8-Stage Pipeline\n\nEvery meaningful record must pass through the same canonical data plane:\n\n1. Analyze\n2. Classify\n3. Filter\n4. Refine\n5. Extract\n6. Validate\n7. Sanity\n8. Sectorize\n\nWhy this matters\n\nThis is the platform’s normalization contract. It ensures:\n\n- no raw source payload becomes truth directly\n- tenant and schema rules are applied consistently\n- only canonical shapes reach the Spine\n- Flow A and Flow B remain compatible with Entity 360 and Cognitive\n\n---\n\n9. Spine and Storage Responsibilities\n\nSpine\n\nStores canonical structured truth:\n\n- entities\n- relationships\n- schema observations\n- operational truth\n- approval and action records where needed\n- linked references\n\nKnowledge\n\nStores retrieval-optimized context:\n\n- chunks\n- embeddings\n- linked unstructured artifacts\n- evidence content\n\nCompounding Space / Approved Memory\n\nStores approved Flow C knowledge and memory. This is not SSOT. It is optional contextual augmentation.\n\nAudit / Decisions / Actions\n\nStores:\n\n- approvals\n- denials\n- action history\n- traceability\n- decision memory\n\nImportant rule\n\nThese stores are not interchangeable. The workspace must not treat Knowledge or approved memory as the primary truth source.\n\n---\n\n10. Workspace Layer Data Loading (L1)\n\nThis section clarifies the missing piece that causes the most confusion.\n\nThe Workspace Layer does not load from connectors directly.\n\nL1 becomes usable only after Spine-backed data exists.\n\n10.1 The workspace loading principle\n\nL1 is:\n\n- the operating surface\n- the place where users work\n- the place where signals, approvals, and context are surfaced\n\nL1 is not:\n\n- the ingestion layer\n- the truth layer\n- a direct connector-read layer\n\nIt renders data from:\n\n- Spine truth\n- workflow / BFF projections\n- Knowledge links\n- Signals\n- optional approved memory when explicitly linked into Entity 360\n\n  10.2 What makes L1 ready\n\nThe workspace becomes usable after Creamy has successfully passed through:\n\nconnector auth → loader → queue → 8-stage pipeline → normalizer → Spine write\n\nOnly then should the workspace hydrate from canonical data.\n\n10.3 The exact L1 loading chain\n\nflowchart LR\nA[Auth and Onboarding] --> B[Connector Auth]\nB --> C[Creamy Sync]\nC --> D[Loader]\nD --> E[Queue]\nE --> F[Pipeline]\nF --> G[Spine Write]\nG --> H[BFF / Readiness / Tenant Config]\nH --> I[WorkspaceShell]\nI --> J[Dashboard and Modules]\nG --> K[Entity 360]\nK --> L[Signals and L2 Activation]\n\n10.4 The four loading surfaces inside L1\n\nA. Workspace shell load\n\nIncludes:\n\n- shell\n- sidebar / nav\n- tenant context\n- active view\n- personal vs work mode\n- readiness state\n\nB. Home dashboard load\n\nIncludes:\n\n- dashboard cards\n- approvals count\n- intelligence count\n- focus items\n- summary widgets\n\nC. Module data load\n\nIncludes workspace module views such as:\n\n- Home\n- Accounts / Projects\n- Contacts\n- Meetings\n- Docs\n- Tasks\n- Calendar\n- Notes\n- Knowledge Space\n- Team\n- Pipeline\n- Risks\n- Expansion\n- Intelligence-linked module states\n\nD. Intelligence overlays back into L1\n\nIncludes:\n\n- banners\n- signal indicators\n- attention queues\n- approvals\n- feed summaries\n\n  10.5 Official L1 data sources\n\nThe workspace reads from:\n\n1. Spine\n2. Knowledge\n3. Approved memory / compounding space when explicitly linked\n4. Signals generated from normalized truth and Think\n\n   10.6 Phased workspace loading\n\nPhase 1 — Shell ready\n\nAuth, tenant, nav, and readiness resolve.\n\nPhase 2 — Creamy workspace ready\n\nFirst useful data lands, and the user can work.\n\nPhase 3 — Module hydration\n\nDashboard and modules fill from runtime projections.\n\nPhase 4 — Needed expansion\n\nHistorical and broader entity coverage expands.\n\nPhase 5 — Delta freshness\n\nIncremental sync keeps L1 current.\n\n10.7 Immediate vs later loads\n\nImmediately after Creamy\n\n- shell and nav\n- dashboard cards\n- first entity projections\n- first approvals and signal feed\n\nLater through Needed / Delta\n\n- deeper history\n- broader entity coverage\n- richer module views\n- stronger Entity 360\n- better cognitive quality\n\n  10.8 L1 and L2 dependency\n\n- L1 can render after Creamy\n- L2 should only fully activate after Creamy and enough meaningful connectors are loaded\n\n  10.9 User-visible interpretation\n\n1. Onboarding decides the lens\n2. Connector auth makes the data real\n3. Creamy gives first value fast\n4. Spine becomes the source\n5. Workspace renders that truth\n6. Needed and Delta deepen the workspace silently\n7. L2 activates when enough truth exists\n\n---\n\n11. Entity 360\n\nEntity 360 is the read-time fusion layer for the platform.\n\nIt combines:\n\n- Spine truth\n- Knowledge context\n- Signals\n- optional approved memory when linked\n\nRule\n\nEntity 360 is a read model, not a write model.\n\nIt exists to support:\n\n- workspace understanding\n- grounded reasoning\n- evidence-backed proposals\n- consistent entity views\n\n---\n\n12. Cognitive Layer (L2)\n\nThe cognitive loop is:\n\nEntity 360 → Think → Govern → HITL → Act → Adjust\n\nflowchart LR\nA[Entity 360] --> B[Think]\nB --> C[Govern]\nC --> D[HITL]\nD --> E[Act]\nD --> F[Adjust]\nE --> G[Re-ingest]\nG --> H[Spine]\nF --> B\n\nThink\n\nReads canonical truth and context to generate signals, proposals, and evidence.\n\nGovern\n\nEvaluates policy and determines whether the proposal can proceed.\n\nHITL\n\nCaptures explicit human approval, rejection, or deferment.\n\nAct\n\nExecutes approved actions via controlled integration paths.\n\nAdjust\n\nLearns from rejections, outcomes, and decision memory.\n\n---\n\n13. Approval and Re-ingestion Model\n\nNo action is complete until its result has been reconciled through the platform.\n\nCorrect action path\n\n1. Proposal created\n2. Govern checks policy\n3. User approves in UI\n4. Act executes external mutation\n5. Result re-enters the pipeline\n6. Spine updates\n7. Workspace refreshes from canonical truth\n\nThis prevents internal truth from drifting away from real external state.\n\n---\n\n14. Anti-Patterns to Avoid\n\n1. Direct source-to-truth writes\n1. AI directly mutating Spine truth\n1. Shadow schemas in modules or connectors\n1. Workspace reading raw connector APIs directly\n1. Entity 360 becoming a hidden write surface\n1. Approval bypass for governed actions\n\n---\n\n15. Acceptance Checklist\n\nTenant and onboarding\n\n- tenant config is seeded correctly\n- onboarding choices define scope and schema direction\n- workspace shell uses tenant config and readiness\n\nConnector and sync\n\n- connector auth binds correctly to tenant\n- connection success triggers Creamy\n- source data enters the approved queue path\n\nPipeline and truth\n\n- all records pass through all 8 stages\n- truth writes land in Spine\n- Flow B writes linked Knowledge context\n- Flow C remains knowledge-first until approved action\n\nWorkspace loading\n\n- L1 shell reads BFF / readiness / tenant config\n- dashboard uses Spine-backed runtime projections\n- connectors do not directly populate L1\n- Creamy is the gate for usable workspace\n- Needed and Delta deepen the workspace in the background\n- signals in L1 come from normalized data and Think\n- L2 activation is gated until enough truth exists\n\nCognitive and action\n\n- Entity 360 is read-time fusion only\n- Think reads canonical truth\n- Govern and HITL gate action\n- Act results re-enter the pipeline\n- final workspace state reflects reconciled truth\n\n---\n\n16. One-Sentence KT Summary\n\nIntegrateWise data must always enter through controlled backend ingest paths, pass through the mandatory normalization pipeline, become canonical truth in the Spine, surface into the workspace through Spine-backed runtime projections, power Entity 360 and the cognitive loop, route actions through approval, and return through re-ingestion so the Spine remains the only source of truth.\n\n---\n"}