{"slug":"business/27-data-pipeline","title":"27 — Data Pipeline","section":"business","sectionLabel":"Business","status":"unclassified","type":"document","path":"business/27-data-pipeline.mdx","bytes":2236,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n> **Extracted from:** IntegrateWise Platform Specification Suite\n> **Region:** third_region\n> **Original line:** 3281\n> **Lines:** 49 | **Chars:** 1,946\n> **Status:** Raw extraction — requires review and canonicalization\n\n27 — Data Pipeline\n27.1 Pipeline (preserves user’s ASCII)\nCopyIngestion\n↓\nNormalization\n↓\nValidation\n↓\nDeduplication\n↓\nCanonical Mapping\n↓\nEnrichment\n↓\nProjection\n↓\nStorage\n27.2 Responsibilities\nRun the pipeline for every Connector Delta (08), Memory intake (07), and Knowledge import (07/L8).\nIdempotency keyed on (source, source*record_id, observed_at, hash).\nStrict stage ordering; failure of one stage quarantines the artifact (DLQ).\n27.3 Inputs\nConnectorDeltaReceived, MemoryIntake, KnowledgeImportRequested, admin batch uploads.\n27.4 Outputs\nCanonical Spine updates (02) + Signal side-effects (10) + Memory write-backs (07).\n27.5 Events produced\nPipelineStarted, PipelineStageCompleted, PipelineStageFailed, PipelineCompleted, PipelineDuplicateDetected, PipelineEnrichmentApplied, PipelineAborted.\n27.6 Events consumed\nConnectorDeltaReceived, MemoryIntake, KnowledgeImportRequested, EntityTypeRegistered.\n27.7 APIs\nPOST /pipeline/run, GET /pipeline/runs/{pipe_id}, POST /pipeline/dlq/{item_id}/replay.\n27.8 Stage definitions\nIngestion: parse vendor payload; schema check.\nNormalization: apply tenant_spine_config.field_extensions, normalize dates/currency.\nValidation: cross-field, cross-entity (12).\nDeduplication: hash + semhash; resolve via Spine ?as_of.\nCanonical Mapping: map to ent* via 12 inheritance.\nEnrichment: Signals (10) may attach; capability cap_enrich_x may run pre-storage.\nProjection: build per-persona projection shape (04).\nStorage: write to Spine (02); archive snapshot to R2.\n27.9 State transitions\nreceived → ingested → normalized → validated → deduped → mapped → enriched → projected → stored. Failure path: … → DLQ. Replay path: DLQ → received.\n\n27.10 Failure handling\nStage failure → stop, retry policy per stage (configurable), finally DLQ.\nDLQ item presents replay UI in Workbench under Pipeline Admin.\n27.11 Extension points\nCustom pipeline steps via PIPELINE_STEP(name).\nCustom canonical mappers MAPPER(name).\n"}