{"slug":"documentation/DOCS_PUBLICATION_PIPELINE","title":"IntegrateWise — Docs Publication Pipeline (internal + public)","section":"documentation","sectionLabel":"Documentation","status":"unclassified","type":"document","path":"documentation/DOCS_PUBLICATION_PIPELINE.mdx","bytes":10306,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n> **Version:** 1.1 | **Date:** 2026-06-09 | **Status:** Canonical (design + doctrine)\n> **Authority:** Nirmal (Founder)\n> **Implements:** AGENTS.md \"Connection 5 — Live → BrandDocumentations → integratewise-docs\"\n> **Companions:** `ARCHITECTURE_INDEX.md`, `MCP_ADK_ROUTING_SPINE_MEMORY_ARCHITECTURE.md`\n\n---\n\n## 1. Principle — ALL docs are a governed projection of org_memory\n\nBoth **internal** docs (engineering / product / ops reference) and **public** docs\n(`docs.integratewise.ai`) are projections of org_memory, rendered by the\ndocs-publish flow. Neither doc track is hand-published. The only difference between\nthem is **classification** → which decides the access gate and the sanitization\nlevel. The path is identical:\n\n```\nNo agent (and no human) hand-publishes a doc — internal or public.\nContent is PROPOSED → GOVERNED → PROMOTED → PROJECTED → PUBLISHED.\ndocs-publish is the only writer of the published projection (humans never hand-edit).\n```\n\n> **Initial load scoping (the one carve-out):** the **one-time bulk seed** of the\n> existing docs corpus runs on a **separate bulk path — NOT the live pipeline**\n> (`pipeline-process`). It uses an isolated, lower-priority bulk queue (mirroring\n> `BULK_MEMORY_QUEUE`) so backfilling the corpus never creates backpressure on live\n> operations. **Steady-state (ongoing) publication** uses the normal governed flow\n> below. This carve-out applies to the initial load only.\n\nThe docs repos are **outputs**, not inputs. Edits happen at the source\n(org_memory), never in the rendered repo.\n\n---\n\n## 2. Two tracks, one publication path — routed by classification\n\n```\nclassification = internal                  classification = public\n  ─────────────────────────                  ─────────────────────────\n  Audience: team / operators / agents        Audience: anyone\n  Surface:  internal docs portal             Surface: docs.integratewise.ai\n            (CF Access gated)                           (public, no auth)\n  Repo:     integratewise-docs/docs/internal integratewise-docs/docs/ (public)\n  Sanitize: strip secrets + raw credentials  strip secrets + PII + internal IDs\n            (internal IDs/arch allowed)                 + tenant data + internal-only arch\n  Gate:     CF Access (team identity)         none (public)\n  Threshold: approved + classification=internal  approved + classification=public\n                                                 + public-sanitization PASSED\n```\n\nSame Triage governance. Same docs-publish writer. Same no-archive + lineage rules.\nOnly the **filter, sanitization depth, and destination/gate** differ.\n\n---\n\n## 2. The publication flow (end to end)\n\n```\nUPSTREAM (how the source itself is earned)\n  conversation → conversational_memory (raw capture)\n              → propose → Triage (govern) → org_memory (approved/published)\n  org_memory is NEVER authored directly. It is the governed promotion of\n  conversational_memory. Docs project only from this earned layer.\n\nSOURCE (record of truth for projection)\n  org_memory WHERE governance_state = 'approved' | 'published'\n  (D1 / Supabase fortress for promoted tenants; filesystem vault mirror)\n        │\n        ▼\nSELECT (publishable subset — by classification)\n  PUBLIC track:   classification = public   AND governance_state IN (approved, published)\n  INTERNAL track: classification = internal AND governance_state IN (approved, published)\n  Exclude (both): staging, rejected, superseded-without-replacement\n        │\n        ▼\nTRANSFORM (docs-publish flow — projection stage)\n  • render org_memory entries → doc pages (frontmatter + body)\n  • resolve lineage (supersedes / superseded_by) into version notes\n  • sanitize by track:\n      PUBLIC   → strip secrets + PII + internal IDs + tenant data + internal-only arch\n      INTERNAL → strip secrets + raw credentials (internal IDs/arch allowed)\n  • map to Document Manifest entry (slug, section, nav order, classification)\n        │\n        ▼\nPUBLISH (GitHub API → integratewise-docs)\n  • docs-publish service commits rendered pages via GitHub API:\n      PUBLIC   → integratewise-docs/docs/        (public site root)\n      INTERNAL → integratewise-docs/docs/internal/ (CF Access gated)\n  • Updates IntegrateWise_Document_Manifest.md (the docs index)\n  • Marks source org_memory entry governance_state = 'published'\n        │\n        ▼\nBUILD & SERVE\n  PUBLIC   → docs.integratewise.ai            (Cloudflare, public)\n  INTERNAL → docs.integratewise.ai/internal   (Cloudflare + CF Access team gate)\n        │\n        ▼\nINDEX (AI Search — derived retrieval, not a source)\n  • embed each published page → AI Search / Vectorize\n  • namespace by classification:  docs-public  |  docs-internal\n  • metadata: slug, classification, tenant scope, governance_state, lineage\n  • retrieval is access-scoped: internal namespace only reachable by gated\n    (CF Access) agents/users; public namespace open to any authorized reader\n  • re-embed on re-projection; drop on supersede/takedown (lineage preserved)\n```\n\n> AI Search is a **derived index of the published projection**, never authored\n> directly. org_memory remains the source of truth; the rendered page is the\n> projection; the embedding is a retrieval accelerator. If the AI Search index is\n> wiped it rebuilds from the published pages — same rebuildable-cache discipline\n> as Spine Cache. This is what lets the Twin answer \"what do our docs say about X\"\n> via `kb.search` / `memory.search_org` without ever reading the docs repo directly.\n\n> The transform/publish stages run in the **docs-publish flow** (a projection +\n> GitHub-publish job). **Initial-load carve-out:** the one-time bulk seed of the\n> existing corpus runs on a **separate bulk queue, not the live `pipeline-process`\n> queue** (backpressure isolation, like `BULK_MEMORY_QUEUE`). Steady-state\n> publication uses the normal governed flow. Either way, docs are a projection of\n> governed org_memory and `integratewise-docs` is an output repo, never hand-edited.\n\n---\n\n## 3. Governance gate (same Triage Bot, classification-aware threshold)\n\n```\nPROPOSE    memory.propose with classification ∈ {internal, public} → proposals (D1)\nGOVERN     Triage Bot scores. Both tracks require:\n             • governance_state = approved\n             • classification set (internal | public)\n             • sanitization check PASSED for that track\n                 internal → no secrets / raw credentials\n                 public   → no secrets / PII / internal IDs / tenant data\n           ≥ 0.85 auto-approve · 0.60–0.84 HITL (L2) · < 0.60 reject\nPROMOTE    approved → eligible for projection on its track\nPUBLISH    docs-publish service projects → integratewise-docs → internal or public surface\n           → source entry flipped to governance_state = published\n```\n\nNo doc — internal or public — exists without a promoted, governed org_memory entry.\n\n---\n\n## 4. No-archive + lineage (carried into all docs)\n\n```\nEvery page (internal + public) inherits the no-archive rule (D-2026-06-05-013):\n  • active        → published and current\n  • superseded    → replaced; old page shows \"replaced by [X]\" with lineage link\n  • rejected      → never published\nNo archive tab. No hidden pages. Unpublishing requires a superseding entry or an\nexplicit governed takedown decision (logged with reason).\n```\n\n---\n\n## 5. Update flow (how a doc changes — internal or public)\n\n```\n1. Edit happens at the SOURCE (org_memory), never in integratewise-docs.\n2. Re-propose → Triage → approve.\n3. docs-publish service re-projects the page (new version; old version linked as lineage).\n4. Manifest updated. Site rebuilds. docs.integratewise.ai reflects the change.\n\nTo align existing drifted public docs (v1.1 facts), use the promotion package:\n  docs/promotions/ARCHITECTURE_ALIGNMENT_v1.1.md  → submit through Triage.\n```\n\n---\n\n## 6. Invariants\n\n```\n• docs-publish is the only writer of the published projection (internal + public);\n  humans never hand-edit. INITIAL LOAD ONLY uses a separate bulk queue (not the\n  live pipeline-process queue) for backpressure isolation; steady-state uses the\n  normal governed flow.\n• integratewise-docs is an OUTPUT repo — never hand-edited (internal or public).\n• Every page (internal or public) traces to an approved/published org_memory entry.\n• Sanitization is a hard gate before publish — depth set by classification.\n• Internal surface is CF Access gated; public surface is open.\n• Document Manifest is the index (carries classification); updated by docs-publish.\n• AI Search is a DERIVED retrieval index of published pages (namespaced\n  docs-public / docs-internal), never a source of truth; access-scoped by\n  classification; rebuildable from the published projection.\n• Same governance model as the Spine: propose → govern → promote → publish.\n```\n\n---\n\n## 7. Build status\n\n```\nPENDING (Phase 4 — Docs):\n  □ docs-publish flow — org_memory → pages (steady-state)\n  □ INITIAL LOAD: separate bulk-seed queue (NOT live pipeline-process; backpressure isolation)\n  □ classification routing (internal → /docs/internal gated; public → /docs)\n  □ GitHub API publish step (commit to integratewise-docs + Manifest update)\n  □ sanitization gate (per-track scrub: internal vs public) in transform\n  □ CF Access gate on the internal docs surface (docs.integratewise.ai/internal)\n  □ governance_state lifecycle: approved → published flip on publish\n  □ AI Search indexing of published pages (namespaces: docs-public / docs-internal;\n    access-scoped retrieval; re-embed on re-projection; drop on supersede)\n  □ wire docs build to consume the projected output only (both tracks)\n```\n\n---\n\n_All docs — internal and public — are projections of org_memory, published by the\ndocs-publish flow. The one-time initial corpus load runs on a separate bulk queue\n(not the live pipeline). Classification decides the gate and the sanitization\ndepth. Connect once, govern once, publish the truth. The repo is the output,\nnever the input._\n"}