{"slug":"architecture/integratewise-data-flow","title":"Integratewise Data Flow","section":"architecture","sectionLabel":"Architecture","status":"unclassified","type":"document","path":"architecture/integratewise-data-flow.mdx","bytes":18100,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"IntegrateWise — Data Flow KT v1.1\n\nStatus: Canonical internal reference\nPurpose: Define how data should flow through IntegrateWise from onboarding to workspace, Spine, cognitive loop, approval, action, and re-ingestion.\n\n---\n\n1. Purpose\n\nThis KT note defines the correct end-to-end data flow for IntegrateWise so product, engineering, connector, data, and AI teams all work from the same model.\n\nThe governing rule is:\n\nAll durable business value in IntegrateWise must be created through a Spine-based flow.\n\nThe canonical runtime chain is:\n\nLanding → Auth → L0 Onboarding → L1 Workspace → L2 Cognitive → L3 Platform\n\nThe canonical system loop is:\n\nLOAD → NORMALIZE → STORE → THINK → REVIEW & APPROVE → ACT → REPEAT\n\n---\n\n2. Canonical Product Statement\n\nIntegrateWise is a Knowledge Workspace over the Spine, empowered by AI and governed by approvals.\n\nThat means:\n\n- Workspace is where users operate\n- Spine is the structured source of truth\n- Knowledge carries linked unstructured context and approved compounding context\n- AI / Cognitive reasons over canonical data\n- Approvals gate execution\n- Actions must reconcile back through the platform so the Spine stays current\n\n---\n\n3. Core Principles\n\n3.1 The Spine is the single source of truth\n\nThe official architecture is explicit that the Spine is the foundation and that ingestion, context, intelligence, backend services, and AI all derive value only when they operate on Spine-shaped data.\n\n3.2 No data may bypass Layer 3\n\nEvery piece of information must pass through the controlled L3 pipeline before it can appear in L2 or L1. There are no shortcuts or side write paths.\n\n3.3 Flow C never writes directly to the Spine\n\nFlow A and Flow B can create truth through pipeline-normalized writes. Flow C contributes knowledge and decision context; truth updates from Flow C only happen through approved actions that go back through Act → pipeline re-ingestion → Spine.\n\n3.4 The workspace is not a truth layer\n\nL1 is where people work, but it is dynamically generated from Spine data, Depth Matrix, BFF projections, and linked knowledge. L1 does not load raw connector data directly and does not maintain a separate SSOT.\n\n3.5 No action executes without approval\n\nThe cognitive loop is always Think → Govern → HITL → Act / Adjust. No durable external mutation is valid without the approval path when policy requires it.\n\n---\n\n4. Official Runtime Architecture\n\nThe official runtime layer order is:\n\nExperience → Gateway → Workspace Runtime → External Connectivity → Data Plane → SSOT / Spine → Cognitive → AI Providers\n\nLayer roles\n\n- L0 Onboarding: collects context, seeds tenant_spine_config, and prepares workspace + schema direction\n- L1 Workspace: personalized work layer with modules, dashboards, approvals, and entity views driven by Spine + Depth Matrix\n- L2 Cognitive: Think, Govern, HITL, Act, Adjust operating on Spine + linked context\n- L3 Backend: Gateway, connectors, loader, pipeline, Spine, knowledge, and background services; the only entry point for external data into the system\n\n---\n\n5. Canonical End-to-End Loop\n\nThe platform loop is:\n\nLoad → Normalize → Store → Think → Govern → HITL → Act → Adjust → Repeat\n\nExpanded meaning\n\n1. Load\n   Data enters through connectors, files, webhooks, MCP sessions, or AI session capture.\n2. Normalize\n   The pipeline transforms raw source payloads into tenant-valid, schema-valid, entity-valid structures.\n3. Store\n   Canonical truth is written to Spine; linked unstructured context goes to Knowledge.\n4. Think\n   Entity 360, signals, and approved context are used to generate proposals and insights.\n5. Govern / HITL\n   Policies are checked and humans approve or reject actions.\n6. Act\n   Approved actions execute through controlled integration channels.\n7. Adjust / Repeat\n   Outcomes are recorded, decision memory improves the next cycle, and re-ingestion reconciles truth.\n\n---\n\n6. End-to-End Flow from Landing to Cognitive\n\nThe full system flow is:\n\n1. User lands on marketing or app entry.\n2. Auth resolves session and onboarding status.\n3. L0 onboarding seeds tenant_spine_config.\n4. User selects and authenticates connectors.\n5. The first phase: \"creamy\" hydration starts.\n6. Loader enqueues records to PIPELINE_QUEUE.\n7. Normalizer runs all 8 stages.\n8. spine-v2 writes canonical truth and schema observations.\n9. The initial workspace becomes usable from Spine-backed creamy data.\n10. Needed and Delta continue in the background.\n11. Entity 360 is read from Spine + Knowledge + Signals.\n12. L2 cognitive logic activates after creamy and connector threshold rules are met.\n\n---\n\n7. Onboarding and Schema Seeding (L0)\n\nOnboarding is the point where tenant reality is declared.\n\nIt determines:\n\n- who the tenant is\n- industry\n- department / domains\n- desired workspace\n- connector relevance\n- whether knowledge UI should link into Entity 360\n- the initial Depth Matrix / Spine direction\n\nThe system stores this in tenant-scoped configuration, especially tenant_spine_config, which later drives workspace navigation, readiness, entity types, and allowed schema behavior.\n\nOnboarding output\n\nOnboarding does not only collect profile information. It actively seeds the future runtime by creating the initial workspace and tenant schema basis. The fixed runtime path explicitly states that Landing / Auth / Onboarding create the Initial Workspace and seed tenant_spine_config.\n\n---\n\n8. Connector Selection, Auth, and Sync Kickoff\n\nThe connect stage must work like this:\n\n1. Connectors are shown based on industry + department\n2. User authenticates a connector\n3. Installation is bound to the tenant\n4. Connected connector state is persisted\n5. The first phase: \"creamy\" sync is triggered immediately on success\n\nThe internal landing-flow doc explicitly describes the chain:\n\n- connectors filtered by industry + department\n- authentication callback\n- resource types + traits determine what data is allowed\n- creamy runs through loader → queue → 8-stage pipeline → normalizer → Spine\n- workspace then reads BFF dashboard, readiness, tenant-config, modules, and Entity 360 sources\n\n---\n\n9. The Three Data Flows\n\n9.1 Flow A — Structured data\n\nFlow A is structured operational data from external systems such as CRM, billing, support, productivity, and related business tools. It lands as truth in the Spine.\n\nFlow A path\n\nConnection success → connector-sync → connector / loader → PIPELINE_QUEUE → Normalizer 8 stages → Spine truth → Initial Workspace → Needed / Delta → Entity 360 → Think / Twin\n\nFlow A role\n\nFlow A is the main truth creation path for operational records. It is the path that powers dashboard lists, entity records, workspace views, and most signals.\n\n---\n\n9.2 Flow B — Unstructured content\n\nFlow B handles documents, emails, files, and other unstructured context. It goes through ingestion, linking, and Knowledge storage while also creating Spine references that can be used in Entity 360. The full landing-flow diagram describes Flow B as: Ingest → entity linking → Spine + Knowledge → Entity 360.\n\nFlow B rule\n\nFlow B enriches context and evidence. It does not replace structured truth.\n\n---\n\n9.3 Flow C — AI sessions, MCP, and compounding knowledge\n\nFlow C handles MCP / AI / chat / triage / compounding knowledge. It follows the knowledge-first path and never writes directly to Spine truth. The landing-flow architecture shows Flow C as: MCP/AI → D1 → Triage Center Bot → triage space → approve → compounding.\n\nFlow C rule\n\nOnly approved actions from Flow C can influence truth, and they must do so through Act → connector write-back → pipeline re-ingestion → Spine.\n\n---\n\n10. Workspace Layer Data Loading (L1)\n\nThis is the missing section now added to the KT.\n\nThe workspace does not load raw connector data directly. L1 is a projection layer that becomes usable only after normalized, Spine-backed data exists. The docs are explicit: all information from external sources must pass through L3 and be written to Spine or Knowledge before it can appear in L2 or L1.\n\n10.1 The workspace loading principle\n\nL1 is the user-facing work surface, but it is not:\n\n- the ingestion layer\n- the truth layer\n- a direct connector read layer\n\nIts job is to render context-aware views of already normalized data coming from:\n\n- Spine truth\n- BFF projections\n- linked Knowledge\n- Signals\n- optional compounding space when linked into Entity 360\n\n  10.2 What makes the workspace “ready”\n\nThe workspace becomes usable after the initial Creamy phase has passed through:\n\nconnector auth → loader → queue → 8-stage pipeline → normalizer → Spine write\n\nThe fixed runtime path explicitly says:\n\n- initial workspace is created from onboarding\n- real connector auth succeeds\n- phase: \"creamy\" hydration is triggered\n- canonical truth is written\n- the Initial Workspace becomes usable after Creamy\n- Needed and Delta continue later in the background\n\n  10.3 The exact L1 loading chain\n\nThe correct loading sequence is:\n\nLanding / Auth / Onboarding → Connector selection → Authentication → Creamy hydration → Spine write → BFF / tenant-config / readiness / dashboard → Workspace shell + modules → Entity 360 + signals → L2 activation\n\n10.4 The four loading surfaces inside L1\n\nA. Workspace shell load\n\nThis is the first visible workspace load.\n\nIt includes:\n\n- workspace shell\n- sidebar / nav\n- tenant context\n- active domain / view\n- work vs personal mode\n- readiness state\n\nThe landing-flow doc states that L1 is built from WorkspaceShell, nav from tenant_spine_config, BFF dashboard/readiness/HITL/analytics, and ContentRouter to domain modules.\n\nB. Home dashboard load\n\nThis is the next visible layer.\n\nIt includes:\n\n- summary cards\n- approvals count\n- intelligence count\n- focus items\n- dashboard widgets\n- readiness / dashboard state\n\nThe BFF surface explicitly serves:\n\n- GET /api/v1/workspace/tenant-config\n- GET /api/v1/workspace/dashboard\n- GET /api/v1/workspace/readiness\n- HITL queue and actions\n\nC. Module data load\n\nThis is where module views load.\n\nExamples seen in current workspace design and docs:\n\n- Accounts / Projects\n- Meetings\n- Docs\n- Tasks\n- Calendar\n- Notes\n- Knowledge Space\n- Team\n- Pipeline\n- Risks\n- Expansion\n- Intelligence surfaces\n\nThese are domain projections rendered from Spine-backed runtime data, not live connector API reads. The full flow docs describe the workspace as dashboard, list views, modules, and Entity 360 built from Spine and internal stores.\n\nD. Intelligence overlays back into L1\n\nL1 also shows intelligence before the user opens L2, including:\n\n- banners\n- signal indicators\n- approvals\n- feed summaries\n- attention queues\n\nThe phased flow doc says dashboard and list views are built from Spine and BFF, while proposals from Think appear in the workspace and the user approves, edits, or rejects there.\n\n10.5 Official L1 data sources\n\nThe workspace reads from these internal sources:\n\n1. Spine — domain tables, core truth, projections\n2. Knowledge — Flow B entity-linked chunks\n3. Knowledge UI / compounding space — only when link_knowledge_ui_to_entity360 is true\n4. Signals — derived from Normalizer + Think\n\n   10.6 Workspace loading happens in phases\n\nL1 loading is phased.\n\nPhase 1 — Shell ready\n\nAuth, tenant, onboarding completion, nav, and workspace shell resolve.\n\nPhase 2 — Creamy workspace ready\n\nFirst useful normalized data lands; user can work.\n\nPhase 3 — Module hydration\n\nDashboard and module views begin filling from BFF + Spine projections.\n\nPhase 4 — Needed expansion\n\nThe next layer of business truth expands entity and historical coverage.\n\nPhase 5 — Delta freshness\n\nIncremental refresh keeps L1 current.\n\n10.7 What should load immediately vs later\n\nLoad immediately after Creamy\n\n- shell and nav\n- tenant identity\n- dashboard cards\n- initial approvals and intelligence feed\n- first account/task/meeting/contact projections\n- first entity-ready work surfaces\n\nLoad later through Needed / Delta\n\n- deeper history\n- wider entity coverage\n- more complete module views\n- richer Entity 360\n- stronger signal quality\n- better cognitive recommendations\n\n  10.8 L1 / L2 activation dependency\n\nThe official rule is:\n\n- L1 can render once Creamy truth exists\n- L2 intelligence should activate only after Creamy is complete and at least 2–3 meaningful connectors are loaded\n\nThis is the dependency that prevents empty or weak intelligence on shallow data.\n\n10.9 User-visible interpretation\n\nThe user-visible meaning of workspace loading is:\n\n1. Onboarding decides the lens\n2. Connector auth makes the data real\n3. Creamy gives first value fast\n4. Spine becomes the source\n5. Workspace renders that truth\n6. Needed and Delta deepen the workspace silently\n7. L2 activates once enough truth exists\n\n   10.10 One-line workspace rule\n\nThe Workspace Layer does not load from connectors directly; it loads from Spine-backed runtime projections after Creamy hydration has passed through connector auth, loader, pipeline, normalizer, and canonical Spine write.\n\n---\n\n11. The Mandatory 8-Stage Pipeline\n\nEvery meaningful incoming record must pass through the 8-stage pipeline:\n\n1. Analyze\n2. Classify\n3. Filter\n4. Refine\n5. Extract\n6. Validate\n7. Sanity\n8. Sectorize\n\nThis is the mandatory normalization path for all canonical truth creation. Raw external events do not bypass it.\n\n---\n\n12. Entity 360\n\nEntity 360 is a read-time fusion view.\n\nIt is built from:\n\n- Spine\n- Knowledge\n- Signals\n- optional compounding space when link_knowledge_ui_to_entity360 is true\n\nRule\n\nEntity 360 is a read model, not a materialized truth table and not a write surface. The docs explicitly describe it as built at read time.\n\n---\n\n13. Cognitive Loop (L2)\n\nThe cognitive loop is:\n\nEntity 360 → Think → Govern → HITL → Act / Adjust → Spine updated\n\nThink\n\nConsumes Entity 360 and linked knowledge to generate proposals and evidence refs.\n\nGovern\n\nPerforms policy checks and ensures approval token gating.\n\nHITL\n\nUser approves, edits, or rejects proposals in the workspace.\n\nAct / Adjust\n\nApprove leads to execution; deny leads to feedback / decision memory. The docs describe loop closure as Act updating the system and Adjust feeding future thinking.\n\n---\n\n14. Approval and Action Model\n\nNo action is complete until the result has been reconciled through the platform.\n\nCorrect action path\n\n1. Proposal created\n2. Govern checks policy\n3. User approves in workspace\n4. Act executes external mutation\n5. Result re-enters pipeline\n6. Spine updates\n7. Workspace refreshes from canonical truth\n\nThis prevents internal truth from drifting away from actual external state.\n\n---\n\n15. Storage Responsibility Model\n\nSpine\n\nCanonical structured truth, entity tables, relationships, schema observations.\n\nKnowledge\n\nUnstructured chunks, evidence, entity-linked contextual data.\n\nKnowledge UI / compounding space\n\nApproved Flow C knowledge and memory layer, optionally linked into Entity 360.\n\nAudit / actions / decisions\n\nApprovals, denials, action records, and traceability.\n\nL1 rule\n\nThe dashboard is powered from internal Spine-backed data via the BFF. It does not make live multi-system calls for each initial load; it reads the normalized internal copy and internal projections.\n\n---\n\n16. Anti-Patterns to Avoid\n\nDo not allow these:\n\n1. Direct source-to-truth writes\n   Connectors or webhooks must not directly write business truth outside the pipeline.\n2. AI direct-to-Spine writes\n   Flow C must never mutate truth directly.\n3. Shadow schemas\n   No parallel truth models in modules, connectors, or AI services. The Spine is the foundation.\n4. Workspace reading external systems directly\n   L1 must read from BFF / Spine-backed projections, not from raw connector APIs.\n5. Entity 360 as write surface\n   Entity 360 is for fusion and reasoning only.\n6. Approval bypass\n   No governed action may execute without the correct approval path.\n\n---\n\n17. Code / Runtime Map\n\nFor engineers tracing implementation, the main runtime map is:\n\n- onboarding seeds tenant config and schema direction\n- connector auth binds installation\n- sync triggers creamy\n- loader enqueues to pipeline\n- normalizer runs 8 stages\n- Spine writes truth\n- BFF exposes tenant-config, readiness, dashboard\n- WorkspaceShell + ContentRouter render L1\n- Entity 360 reads Spine + Knowledge + Signals\n- Think / Govern / HITL / Act run on top of that truth\n\n---\n\n18. Acceptance Checklist\n\nTenant and onboarding\n\n- tenant config is seeded correctly\n- onboarding choices define runtime scope\n- workspace shell uses tenant config and readiness\n\nConnector and sync\n\n- connector auth binds to tenant\n- connection success triggers creamy\n- source data enters the queue path\n\nPipeline and truth\n\n- all records pass through all 8 stages\n- truth writes land in Spine\n- Flow B writes linked Knowledge context\n- Flow C remains knowledge-first until approval-driven action\n\nWorkspace loading\n\n- L1 shell reads BFF / tenant-config / readiness\n- dashboard uses Spine-backed projections\n- connectors do not directly populate L1\n- creamy is the gate for usable workspace\n- needed and delta deepen the workspace in background\n- signals in L1 come from normalized data and Think\n- L2 activation is gated until creamy + connector threshold\n\nCognitive and action\n\n- Entity 360 is read-time fusion only\n- Think reads canonical truth\n- Govern and HITL gate action\n- Act results re-enter the pipeline\n- final workspace state reflects reconciled truth\n\n---\n\n19. One-Sentence KT Summary\n\nIntegrateWise data must always enter through controlled L3 ingest paths, pass through the mandatory normalization pipeline, become canonical truth in the Spine, surface into the workspace through BFF and Spine-backed projections, power Entity 360 and the cognitive loop, route actions through approval, and return through re-ingestion so the Spine remains the only source of truth.\n\nIf you want, the next step is turning this into a polished internal artifact with a title page, version block, diagrams, and a short “L1/L2 screenshots mapping” appendix.\n"}