Unclassified
19 · Ingestion & Normalization Pipeline
External Systems → Loader → Normalizer → Pipeline → Governed Writer → Business Context Spine. Flows, contracts, and why provider schema is never canonical.
How External State Becomes Business Context
External Systems
│
▼
Loader (connector-typed, tenant-scoped)
│
▼
Normalizer (provider schema → canonical entities/relationships)
│
▼
Pipeline (four flows — governed routing)
│
▼
Governing Writer (only governed writes update the Spine)
│
▼
Business Context Spine (THE organization's memory)
│
▼
Projection → Human Workbench / AI Workbench (Active Context Bundle)
Provider schema is never canonical. A raw Salesforce or M365 payload does not become Spine truth until normalized, deduplicated, linked, and governed.
The Four Flows
The pipeline distinguishes four governed flows (detail in the Founder Canon and Codebase Alignment):
- Initial load / hydration — bulk bring-up of a tenant's connected ecosystems.
- Incremental sync — continuous delta ingestion with deduplication and relationship linking.
- Event-driven ingestion — webhook / event-stream triggered updates, with idempotency.
- Execution re-ingestion — outcomes from Hermes-re-executed writes, verified before promotion.
Each flow has its own contracts, idempotency, provenance capture, and governed-writer policy.
Loader · Normalizer · Governed Writer
| Component | Responsibility |
|---|---|
| Loader | Connector-scoped fetch of provider payloads, with tenant isolation and rate-limit awareness |
| Normalizer | Maps provider types to canonical entities, resolves relationships, captures provenance |
| Pipeline | Routes through the correct flow with lineage and audit |
| Governed Writer | The only path that may mutate Spine truth — guarded by tenancy, provenance, and governance |
From Outcome Back to Spine
Writes follow the same pipeline in reverse:
Work → Governance → Hermes → External System → Outcome
→ Re-ingest (Loader) → Normalize → Verify → Reconcile → Governed Writer → Spine
This is why Execution Closure is not provider success but verified Spine update.
Where to Go Next
<CardGroup cols={2}>How ingested outcomes are verified and reconciled to Spine truth.
What the Spine holds once promotion succeeds.
How that truth is projected to participants.