{"slug":"business/SYSTEM_FLOW_DEFINITIVE","title":"IntegrateWise — Definitive System Flow","section":"business","sectionLabel":"Business","status":"unclassified","type":"document","path":"business/SYSTEM_FLOW_DEFINITIVE.mdx","bytes":23435,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n> Founder's design. Canonical. All code and docs must conform to this.\n> Last updated based on complete architectural review conversation.\n\n---\n\n## Current Surface-Binding & MCP Nervous System Doctrine (v3.0)\n\n- **MCP as the System Nervous System**: The Model Context Protocol (MCP) is not just a gateway or a set of wrappers; it is the primary communication fabric of IntegrateWise. All tool integrations, workflow triggers, memory access, and worker coordination occur via standard MCP schemas.\n- **Loader & Normalizer Integration**: Loader and Normalizer are implemented as internal execution steps under the unified MCP framework, ensuring that any incoming data payload conforms to the system's strict protocol contracts before spine insertion.\n- **Zero-Trust Database Isolation**: No agent, worker, or twin can write directly to the Supabase database. All database writes are delegated through the `PIPELINE` Fetcher binding's `/api/v1/db-proxy` endpoint, preserving complete tenant and schema boundaries.\n- **Operational Workbench & Twin**: The primary human interface is the Operational Workbench with the collapsible Twin sidebar. The Twin generates playbooks (JSON action sequences) and context locally, and hands them off to the client environment for execution, enforcing a strict boundary on compute costs.\n- **Triage Bot Sole Writer Rules**: Only the Triage Bot has write permissions to organizational and shared memory. All AI summaries and session learnings must route through the Triage Bot for confidence scoring and staging.\n\n**Canonical architecture reference:** `docs/ARCHITECTURE_CANONICAL.md` — supersedes all prior descriptions.\n\n---\n\n## THE CORE LOOP\n\n```\nLoad → Store → Think → Act → Govern → Repeat → Adjust\n```\n\nREPEAT only happens on structured data (Flow A). Not Flow B. Not Flow C.\n\n---\n\n## ONBOARDING — SCHEMA ACTIVATION\n\nUser selects Department + Industry during onboarding.\n\n- `getSpineConfig(industry, department)` composes the final schema\n- 12 domain base schemas × 11 industry overlays = adaptive by design\n- Schema controls everything downstream: what Loader extracts, what Normalizer filters, what L1 shows, what L2 reasons over\n- Writes: Spine DB `tenant_spine_config`\n\n### 12 Domain Schemas\n\n| Schema         | Department       | Entity types (examples)                                                                                                                                                          |\n| -------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `cs`           | Customer Success | account_master, success_plan, risk_register, engagement_log, renewal, stakeholder_outcome, initiative, capability, value_stream, api_portfolio, people_team, strategic_objective |\n| `sales`        | Sales            | deal, opportunity, lead, pipeline_stage, quote, sales_sequence, call_log, competitor_intel                                                                                       |\n| `marketing`    | Marketing        | campaign, audience, landing_page, email_campaign, social_post, ab_test, attribution_touchpoint, audience_segment                                                                 |\n| `revops`       | RevOps           | quota, forecast, territory, comp_plan, revenue_metric, attribution, segment_rule                                                                                                 |\n| `eng`          | Engineering      | sprint, incident, deploy, bug, release, repository, pull_request, deployment                                                                                                     |\n| `product`      | Product          | feature, usage_metric, feedback, roadmap_item                                                                                                                                    |\n| `support`      | Support          | ticket, csat, sla_policy, escalation, queue, agent_performance, knowledge_article                                                                                                |\n| `finance`      | Finance          | invoice, payment, expense, budget, revenue_entry, tax_filing, financial_report, vendor, cost_center                                                                              |\n| `it`           | IT Admin         | system, device, user, change_request, vulnerability, backup, certificate, network_device, license                                                                                |\n| `bizops`       | BizOps           | activity, workflow, okr, cross_dept_initiative, ops_metric                                                                                                                       |\n| `procurement`  | Procurement      | purchase_order, spend_category, savings_initiative, approval_request, compliance_check                                                                                           |\n| `industry_edu` | Education        | course, assignment, grade, attendance, discussion, learning_objective, intervention                                                                                              |\n\n### 11 Industry Overlays\n\nSaaS, Healthcare, Manufacturing, Automotive, Logistics, Retail, Financial Services, Media, Public Sector, Education, Professional Services\n\nEach overlay adds industry-specific entity types on top of the domain schema. A CS manager in Healthcare gets the `cs` schema plus `industry_hc` entities (patient, claim, provider).\n\n### Personal Schema\n\n| Schema  | Entity types                                     |\n| ------- | ------------------------------------------------ |\n| `spine` | calendar_event, bookmark, daily_reflection, note |\n\n---\n\n## THREE DATA FLOWS\n\n### Flow A: Structured Data → Truth (HAS REPEAT LOOP)\n\n**Sources:** CRMs, databases, billing, product tools — the allowed tools per tenant schema.\n\n```\nSource System API (Salesforce, HubSpot, Stripe, Jira, etc.)\n    │\n    ▼\nCONNECTOR\n    │  OAuth 2.0 flow\n    │  Writes: Spine DB `connectors` (via PIPELINE db-proxy)\n    │  Caches: KV `CONNECTOR_STATUS`\n    │\n    ▼\nLOADER (Internal Sync / Webhook Ingress)\n    │  Creamy load (30 days, bounded) or delta/full sync\n    │  Reads: Source API using stored OAuth tokens\n    │  Invokes: pipeline.ingest tool (MCP Server protocol boundary)\n    │  Sends: PipelineMessage to PIPELINE_QUEUE\n    │\n    ▼\nNORMALIZER (8-stage pipeline - integrated in env.PIPELINE)\n    │\n    │  S1 ANALYZE: Detect entity type, resolve schema from Spine DB tenant_spine_config\n    │  S2 CLASSIFY: Assign priority\n    │  S3 FILTER: Drop if entity type not in schema. Idempotency check (D1 fingerprint)\n    │  S4 REFINE: Normalize field names to canonical\n    │  S5 EXTRACT: Keep only schema-allowed fields. Drop unknown fields.\n    │  S6 VALIDATE: Type checks, required fields\n    │  S7 SANITY: Business rules. Score < 70 → DLQ\n    │  S7.5 IDENTITY RESOLUTION: Signature to D1, match check, merge_candidates if >= 0.7\n    │  S8 SECTORIZE: Route to correct department table\n    │\n    ▼\nSPINE DB (Zero-Trust Spine Membrane)\n    │  All writes routed strictly via env.PIPELINE.fetch(\"http://pipeline/v1/spine/...\")\n    │  INSERT INTO spine.entities (...) ON CONFLICT DO UPDATE (version++)\n    │  INSERT INTO {schema}.{table} (department-routed write)\n    │  Enqueue: INTELLIGENCE_QUEUE (for Twin activation)\n    │\n    ▼\nENTITY 360 (convergence — D1 edge cache + pgvector Supabase + markdown vault)\n    │\n    ▼\nREPEAT LOOP:\n    When an approved playbook action executes on an external tool (via local execution claw),\n    that tool's data changes. Loader triggers sync -> Normalizer processes -> Spine updates.\n    The system sees its own actions as new structured data.\n    This is the continuous loop. Only Flow A has this.\n```\n\n**Failure handling:** 3 retries with backoff (30s, 60s, 90s). After 3 → DLQ_QUEUE → Spine DB `normalization_errors`.\n\n---\n\n### Flow B: Unstructured Data → Context (NO REPEAT LOOP)\n\n**Sources:** Notion, Google Drive, Confluence, Gmail, Slack, SharePoint, Dropbox\n\n```\nSource System API\n    │\n    ▼\nCONNECTOR → LOADER → PIPELINE_QUEUE → NORMALIZER (same 8 stages)\n    │\n    │  At S8: Writes metadata to Spine DB spine.entities\n    │  Sends content to KNOWLEDGE_QUEUE\n    │\n    ▼\nKNOWLEDGE SERVICE\n    │  Generates embeddings\n    │  Writes: Spine DB `context_extractions` (entity_type + entity_id linking)\n    │  Documents, emails, PDFs stay intact and are connected to entities\n```\n\n**No repeat loop.** Unstructured data flows in once and is linked. It does not loop back.\n\n---\n\n### Flow C: AI Sessions → Knowledge Memory (NO REPEAT LOOP, NEVER WRITES TO SPINE)\n\n**Sources:** ChatGPT, Claude, Gemini, Perplexity, Grok, MCP-based tools, Twin itself\n\nFlow C is not just \"AI session summaries stored somewhere.\" It follows a definitive structure:\n\n```\nAI conversation or Twin reasoning\n    │\n    ▼\nTRIAGE BOT (sole writer to memory)\n    │\n    │  The Triage Bot is the GATEKEEPER.\n    │  No AI, no user writes directly to the memory store.\n    │  Only the Triage Bot has write access.\n    │\n    │  What gets written:\n    │  1. Session summaries from AI conversations\n    │     (continuously updated as sessions progress)\n    │  2. Truth + Context summaries from the Twin\n    │     (Twin reads Spine truth and Context, writes summarized\n    │      knowledge back into memory — but through Triage Bot, not directly)\n    │\n    │  Triage Bot processes:\n    │    Entity extraction, sentiment analysis, key fact extraction,\n    │    confidence scoring (0.0 — 1.0)\n    │\n    │  Writes: D1 (edge buffer) → Spine DB `ai_memories`\n    │    memory_type: decision | preference | insight | action | rule | fact\n    │    status: pending (human review) or approved (auto-approve rules)\n    │\n    ▼\nMEMORY CONSOLIDATOR (scheduled worker)\n    │  Reads approved ai_memories\n    │  Writes: Spine DB `consolidated_memories`\n    │  Session, daily, and weekly summaries\n    │\n    ▼\nKNOWLEDGE MEMORY STORE\n    │\n    │  Read access: controlled by tenant_id + OAuth approval\n    │    - Personal memory: readable by that person only\n    │    - Shared memory: readable by any AI/agent IF tenant has granted\n    │      OAuth approval for that specific access\n    │\n    │  This memory GROWS DAILY.\n    │  It accumulates over time, becoming richer.\n    │  This is exactly why it's called a Knowledge Workspace —\n    │  the accumulated knowledge memory IS what makes the workspace\n    │  intelligent over time.\n    │\n    │  NEVER writes to Spine.\n    │  NEVER loops back through the pipeline.\n    │  AI memory is never written back into the repeat loop.\n```\n\n---\n\n## ENTITY 360 — WHERE ALL THREE FLOWS MEET\n\nEntity 360 is the convergence point. All three flows contribute to it.\n\n```\nEntity 360 = Spine (Flow A — Truth)\n           + Context (Flow B — context_extractions)\n           + Knowledge Memory (Flow C — consolidated_memories, governed)\n           + Signals (generated by Twin from the above)\n```\n\n**How it assembles (Spine-v2 Worker, parallel reads from Spine DB):**\n\n1. Truth: `SELECT * FROM spine.entities WHERE spine_id = $id` + department table\n2. Context: `SELECT * FROM context_extractions WHERE entity_id = $id`\n3. Knowledge Memory: `SELECT * FROM consolidated_memories WHERE entity_id = $id`\n4. Signals: `SELECT * FROM signals WHERE entity_id = $id AND status = 'active'`\n\nAssembly time: < 200ms. Cache: KV, 60s TTL.\n\n**The critical distinction:**\n\n- Entity 360 READS from all three flows\n- But only Flows A and B WRITE to Spine tables\n- Flow C writes to the knowledge memory store (separate from Spine)\n- The Spine stays clean of AI-generated content\n\n---\n\n## TWIN ACTIVATION — THINK\n\nOnce Entity 360 is assembled, the Twin activates.\n\n**Twin access:**\n\n- READ-ONLY on Spine\n- Reads the full Entity 360 (Truth + Context + Knowledge Memory + Signals)\n- Generates proactive insights and suggestions\n\n**What the Twin produces:**\n\n- Human-readable observations with evidence:\n  - What is happening\n  - Why it matters\n  - What you could do about it\n  - What happens if you ignore it\n  - Evidence chain (which data, from which sources)\n  - Confidence level\n\n**Twin can also write to memory:**\n\n- Twin reads Spine (truth) and Context, and can write summarized knowledge back into the memory store\n- But THROUGH the Triage Bot, not directly\n- This is how the Twin contributes to the growing knowledge memory\n\n**Writes insights:** Spine DB `signals` table. Max 3 per entity per day.\n\n---\n\n## HUMAN DECIDES — GOVERN / HITL\n\nBased on the Twin's proactive insights and suggestions, the human decides.\n\n- Sees the insight with evidence\n- Evaluates: Is this accurate? Is this actionable? Should we proceed?\n\n**If approved:**\n\n- Action moves to ACT\n- ACT executes on external tools (create task, send email, update record)\n- The tool's data changes → flows back into Flow A → Loader picks it up → Spine updates\n- This is the REPEAT loop\n- Twin learns from the approval (ADJUST)\n\n**If modified:**\n\n- Human adjusts the proposed action\n- Modified action goes to ACT\n- Twin learns from the modification\n\n**If dismissed:**\n\n- Insight logged as dismissed\n- Twin learns — future proposals improve\n- No action taken\n\n**Twin → Spine write:**\n\n- Twin is READ-ONLY on Spine by default\n- Twin can write to Spine ONLY after HITL approval activates the write\n- The human approval is what unlocks the Spine write, not the Twin itself\n\n---\n\n## ACT — EXECUTION\n\nAgents and the Twin live ONLY in ACT. They do not participate in truth formation.\n\n**What ACT can do:**\n\n- Create a task in the connected tool\n- Send an email\n- Update a record\n- Schedule a meeting\n- Trigger a connector sync\n\n**Scoped, not full access:**\n\n- AI does NOT have access to the complete ecosystem\n- It can only act on what's permitted through Govern/HITL approval\n- Every action is scoped and governed\n\n**Actions flow back:**\n\n- Action changes the external tool\n- That tool's data flows back into Flow A (structured data)\n- Loader picks it up → Normalizer processes → Spine updates → Entity 360 updates\n- This is the continuous loop — the system sees its own actions as new data\n\n---\n\n## ADJUST — LEARNING\n\n- Denied or edited actions update decision memory (via Triage Bot → consolidated_memories)\n- Twin learns from patterns: what gets approved, dismissed, modified\n- Future proposals improve based on past human decisions\n- No retraining needed — context is always current, always growing, never lost\n\n---\n\n## THE KEY DIFFERENTIATOR — NO RETRAINING\n\nUnlike other AI systems where you have to retrain or re-explain context every time:\n\n- IntegrateWise already knows the context (from the Spine)\n- It already knows the truth (from structured data)\n- It acts based on accumulated knowledge (from the growing memory)\n- When the system updates, that update flows back through Flow A, and the cycle continues\n- The context is always current, always growing, never lost\n\n---\n\n## TWO PRODUCT LAYERS\n\n### L1 — The Workspace (12 domain-specific views)\n\nThis is what the user sees and works in. Schema-driven from the Spine.\n\n- Each domain has its own views shaped by its schema\n- A salesperson sees pipeline, deals, contacts, activities\n- A CS manager sees accounts, health, renewals, risk registers\n- A finance person sees invoices, budgets, revenue, expenses\n- Same Spine data, different lens per domain\n\n**L1 hydrates section-by-section per connector:**\n\n- Connect Gmail → tasks section hydrates\n- Connect Salesforce → accounts + contacts hydrate\n- Connect Stripe → revenue section hydrates\n- Empty sections stay empty until the feeding connector is connected\n\n### L2 — The Cognitive Overlay\n\n> **Canonical enumeration:** see `docs/tech/L2_TWIN_SURFACE_CONTRACT.md` for the full L2 surface contract (Cognitive Domains, Interaction Surfaces, Runtime Primitives, Automation/Workflow Fabric). The list below is a partial summary of Categories 1+2 only; counts are not part of the contract.\n\nSits alongside L1. Non-intrusive. Extends into work only when needed.\n\n- Signals (real-time sensing from connected tools)\n- Think (AI-detected situations with evidence)\n- Approvals/HITL (approve/reject with evidence)\n- Knowledge Hub (Triage Inbox, Memories, Sessions, Topics, Search)\n- AI Chat (grounded in Entity 360 context)\n- Evidence Drawer (provenance tracing)\n- Goal Framework (everything traces to growth or outcomes)\n- Bridge (mission control — Sense/Think/Act unified view)\n- Layer Audit (cross-layer integrity monitoring)\n\n**L2 activates as data accumulates:**\n\n- 1 connector: L2 mostly empty\n- 2-3 connectors: Entity 360 starts assembling meaningful profiles\n- 4+ connectors: Twin sees cross-system patterns, insights become valuable\n- Full stack: Full L2 active, deeply contextual\n\nAccessible via Cmd+J or from the Intelligence Overlay.\n\n---\n\n## WORKSPACE READS — SPINE_DB DIRECT\n\nFrontend reads from Spine DB directly. NOT through Gateway. RLS enforces tenant isolation.\n\n| View                                | Table                                                  |\n| ----------------------------------- | ------------------------------------------------------ |\n| Accounts, Tasks, Contacts, Calendar | spine.entities (filtered by entity_type)               |\n| Department entities                 | {schema}.{table} (cs.account_master, sales.deal, etc.) |\n| Signals                             | spine.signals                                          |\n| Documents                           | context_extractions                                    |\n| AI Memories (triage)                | ai_memories                                            |\n| Verified Knowledge                  | consolidated_memories                                  |\n| Audit trail                         | governance_audit_log                                   |\n\nRealtime updates via Spine DB Realtime subscriptions.\n\n---\n\n## WORKSPACE WRITES — THROUGH GATEWAY\n\nAll writes go through Gateway. Writes have side effects (validation, audit, downstream).\n\n| Action           | API                             | Why                              |\n| ---------------- | ------------------------------- | -------------------------------- |\n| Create task      | POST /api/v1/workspace/entities | Validation + Spine write + audit |\n| Approve triage   | POST /api/v1/triage/approve     | Govern + memory write + audit    |\n| Merge duplicates | POST /api/v1/identity/merge     | Govern + multi-table + audit     |\n| Connect tool     | POST /api/v1/connectors/connect | OAuth + connector service        |\n| Execute action   | POST /api/v1/act/execute        | Govern gate + orchestration      |\n\n---\n\n## SPINE WRITE ACCESS SUMMARY\n\n| Actor              | Spine read           | Spine write              | Memory write         | Memory read            |\n| ------------------ | -------------------- | ------------------------ | -------------------- | ---------------------- |\n| Twin               | YES (via Entity 360) | ONLY after HITL approval | YES (via Triage Bot) | YES                    |\n| Triage Bot         | NO                   | NO                       | YES (sole writer)    | NO                     |\n| External AIs       | NO                   | NO                       | NO                   | YES (per tenant OAuth) |\n| User (via Gateway) | YES                  | YES (validated writes)   | YES (approve/reject) | YES                    |\n| Normalizer         | NO                   | YES (S8 sectorize)       | NO                   | NO                     |\n| Loader             | NO                   | NO (sends to queue)      | NO                   | NO                     |\n\n---\n\n## COMPLETE DB INVENTORY\n\n### Spine DB (SSOT)\n\n| Table                 | Who writes               | Who reads                      | Flow           |\n| --------------------- | ------------------------ | ------------------------------ | -------------- |\n| spine.entities        | Normalizer S8            | Frontend, Entity 360           | A, B           |\n| {dept}.{table}        | Normalizer S8            | Frontend, Entity 360           | A              |\n| context_extractions   | Knowledge Service        | Frontend, Entity 360           | B              |\n| signals               | Think Service            | Frontend, Entity 360           | Twin-generated |\n| ai_memories           | Triage Bot (sole writer) | Frontend (Knowledge Hub)       | C              |\n| consolidated_memories | Memory Consolidator      | Frontend, Entity 360, MCP read | C (approved)   |\n| connectors            | Connector Worker         | Frontend                       | Setup          |\n| governance_audit_log  | Govern Worker            | Frontend                       | All decisions  |\n| tenant_spine_config   | Onboarding               | Normalizer S1                  | Setup          |\n| normalization_errors  | DLQ consumer             | Admin                          | Failures       |\n| policies              | Admin                    | Govern Worker                  | Governance     |\n\n### Cloudflare D1 (Edge operational)\n\n| Table             | Who writes        | Who reads               |\n| ----------------- | ----------------- | ----------------------- |\n| entity_signatures | Normalizer S7.5   | Normalizer S7.5         |\n| merge_candidates  | Normalizer S7.5   | Identity Resolution API |\n| ai_sessions       | Knowledge Service | Triage Bot              |\n\n### Cloudflare KV (Edge cache)\n\n| Namespace        | TTL  | Purpose               |\n| ---------------- | ---- | --------------------- |\n| SCHEMA_CACHE     | 5min | Resolved schemas      |\n| CONNECTOR_STATUS | 5min | Installation mappings |\n| SIGNAL_CACHE     | 60s  | Recent signals        |\n| RATE_LIMITS      | 60s  | IP rate counters      |\n\n### Cloudflare Queues\n\n| Queue              | Producer                | Consumer          |\n| ------------------ | ----------------------- | ----------------- |\n| PIPELINE_QUEUE     | Loader, Webhook Ingress | Normalizer        |\n| KNOWLEDGE_QUEUE    | Normalizer S5/S8        | Knowledge Service |\n| INTELLIGENCE_QUEUE | Normalizer S8           | Think Service     |\n| SIGNAL_QUEUE       | Think Service           | Workflow Service  |\n| DLQ_QUEUE          | Any stage on max retry  | DLQ consumer      |\n\n---\n\n## INTEGRATION BEST PRACTICES\n\n**Idempotency:** Content hash SHA-256(tenant_id + entity_type + source + content). D1 fingerprints. Check before processing.\n\n**Retry:** 3 attempts, backoff 30s → 60s → 90s. After 3 → DLQ.\n\n**DLQ:** Full context preserved. Spine DB normalization_errors. Replayable via admin API.\n\n**Version tracking:** Monotonic on spine.entities. Upsert increments. Highest version wins.\n\n**Trace correlation:** trace_id (UUID) flows through entire pipeline. All logs reference it.\n\n---\n\n_This is the founder's design. Entity 360 is where all three flows meet. Twin is read-only on Spine, writes only via HITL. Triage Bot is the sole writer to memory. Repeat loop is Flow A only. Memory grows daily — that's why it's a Knowledge Workspace. All code must conform to this._\n"}