{"slug":"spine/ingestion-normalization-pipeline","title":"19 · Ingestion & Normalization Pipeline","section":"spine","sectionLabel":"Spine","status":"unclassified","type":"document","path":"spine/ingestion-normalization-pipeline.mdx","bytes":3054,"description":"External Systems → Loader → Normalizer → Pipeline → Governed Writer → Business Context Spine. Flows, contracts, and why provider schema is never canonical.","updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n## How External State Becomes Business Context\n\n```text\nExternal Systems\n      │\n      ▼\n   Loader              (connector-typed, tenant-scoped)\n      │\n      ▼\n  Normalizer           (provider schema → canonical entities/relationships)\n      │\n      ▼\n   Pipeline            (four flows — governed routing)\n      │\n      ▼\n Governing Writer      (only governed writes update the Spine)\n      │\n      ▼\nBusiness Context Spine (THE organization's memory)\n      │\n      ▼\n  Projection → Human Workbench / AI Workbench (Active Context Bundle)\n```\n\n\n**Provider schema is never canonical.** A raw Salesforce or M365 payload does not become Spine truth until normalized, deduplicated, linked, and governed.\n\n\n## The Four Flows\n\nThe pipeline distinguishes four governed flows (detail in the Founder Canon and [Codebase Alignment](/implementation/codebase-alignment)):\n\n* **Initial load / hydration** — bulk bring-up of a tenant's connected ecosystems.\n* **Incremental sync** — continuous delta ingestion with deduplication and relationship linking.\n* **Event-driven ingestion** — webhook / event-stream triggered updates, with idempotency.\n* **Execution re-ingestion** — outcomes from Hermes-re-executed writes, verified before promotion.\n\nEach flow has its own contracts, idempotency, provenance capture, and governed-writer policy.\n\n## Loader · Normalizer · Governed Writer\n\n| Component | Responsibility |\n|---|---|\n| **Loader** | Connector-scoped fetch of provider payloads, with tenant isolation and rate-limit awareness |\n| **Normalizer** | Maps provider types to canonical entities, resolves relationships, captures provenance |\n| **Pipeline** | Routes through the correct flow with lineage and audit |\n| **Governed Writer** | The *only* path that may mutate Spine truth — guarded by tenancy, provenance, and governance |\n\n## From Outcome Back to Spine\n\nWrites follow the same pipeline in reverse:\n\n```text\nWork → Governance → Hermes → External System → Outcome\n  → Re-ingest (Loader) → Normalize → Verify → Reconcile → Governed Writer → Spine\n```\n\nThis is why [Execution Closure](/closure/execution-closure) is not provider success but verified Spine update.\n\n## Where to Go Next\n\n<CardGroup cols={2}>\n  \n    How ingested outcomes are verified and reconciled to Spine truth.\n  \n  \n    What the Spine holds once promotion succeeds.\n  \n  \n    How that truth is projected to participants.\n  \n</CardGroup>\n"}