{"slug":"connectors/PIPELINE_AUTH_TO_MEMORY","title":"IntegrateWise Complete Pipeline: Auth → IW Bridge → Spine Hydration → Workbench → Memory","section":"connectors","sectionLabel":"Connectors","status":"unclassified","type":"document","path":"connectors/PIPELINE_AUTH_TO_MEMORY.mdx","bytes":20854,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n\n> **Tier C — Historical.** This document is a point-in-time snapshot preserved for reference. For current architecture, see [CANONICAL_PLATFORM_ARCHITECTURE.md](../architecture/CANONICAL_PLATFORM_ARCHITECTURE.md).\n\n## Executive Overview\n\nThis document specifies the **complete end-to-end pipeline** for IntegrateWise, building on the existing infrastructure:\n\n- ✅ **OAuth + Clerk Auth** (already in gateway)\n- ✅ **MCP Server** (spine-mcp-server.ts production-ready)\n- ✅ **Connector Registry** (domain-master pattern in place)\n- ✅ **L1 Projections** (role-aware, 16 workbenches)\n- ✅ **Domain Workspaces** (5 domains, 30+ views)\n- ❌ **Missing: Onboarding → Connector Approval → Spine Hydration → Workspace Loader → AI Memory**\n\n## 1. Auth + Onboarding Flow\n\n### 1.1 Current State\n- Gateway handles OAuth (Google/GitHub)\n- Users resolve to tenants via email domain or metadata\n- Clerk stores user metadata (role, org, preferences)\n\n### 1.2 What's Missing: Onboarding Sequence\n\n**Flow:**\n```\nOAuth Callback\n  ↓\nCheck if user exists in tenant\n  ↓\nIf NEW → Onboarding Wizard\n  ├─ Map role → functional category (Sales, CS, Support, etc.)\n  ├─ Display suggested connectors per role\n  ├─ Collect connector preferences (which to activate)\n  └─ Store in tenant_spine_config\n  ↓\nIf EXISTING → Skip to Workspace Load\n  ↓\nIssue MCP token (15 min, HMAC-signed)\n  ↓\nRedirect to /workspace/{domain}/{defaultView}\n```\n\n### 1.3 Implementation Tasks\n\n1. **Create Onboarding API** (`POST /api/v1/onboarding/start`)\n   - Input: user profile from OAuth\n   - Output: role options, suggested connectors, wizard config\n   - Storage: write to D1 tenant_spine_config\n\n2. **Create Connector Selection UI** (`components/onboarding/connector-selector.tsx`)\n   - Display role → connectors matrix\n   - Multi-select with descriptions\n   - Store selections in session\n\n3. **Create Onboarding Completion API** (`POST /api/v1/onboarding/complete`)\n   - Input: selected connectors, role confirmation\n   - Action: activate connectors in registry (not authorize yet)\n   - Issue MCP token\n   - Redirect to workspace\n\n**Files to Create:**\n- `services/gateway/src/onboarding.ts`\n- `apps/web/app/api/v1/onboarding/start/route.ts`\n- `apps/web/app/api/v1/onboarding/complete/route.ts`\n- `apps/web/components/onboarding/connector-selector.tsx`\n- `apps/web/components/onboarding/onboarding-wizard.tsx`\n\n---\n\n## 2. IW Bridge + MCP Server (Already Built)\n\n### 2.1 Current State\n✅ `services/mcp-connector/src/spine-mcp-server.ts` is production-ready with:\n- `spine.query`, `spine.get`, `spine.summary`\n- `memory.search`, `memory.propose`\n- `spine.connected_sources`\n- Tenant isolation, schema projection, tier gating\n\n### 2.2 Missing: ConnectorContext Middleware\n\n**What's needed:**\n- Extract MCP token from request headers\n- Validate HMAC signature\n- Populate SpineMcpActor from token\n- Middleware in CF Worker to inject actor into tool dispatch\n\n**Files to Create:**\n- `services/mcp-connector/src/middleware/auth.ts` (token validation)\n- `services/mcp-connector/src/middleware/actor.ts` (actor resolution)\n\n### 2.3 Missing: ChatGPT Connector Registration\n\n**Action:**\n1. Register IW MCP server in ChatGPT's connector UI\n   - URL: `https://mcp.integratewise.ai/mcp`\n   - Auth: Bearer token from session\n2. ChatGPT can now call IW tools (spine.query, memory.search, etc.)\n3. IW can call ChatGPT's tools (Salesforce, Gmail, etc.) via ChatGPT's connector API\n\n**No new code needed** — registration is manual/UI-driven\n\n---\n\n## 3. Spine Hydration from Connectors\n\n### 3.1 Architecture\n\n**Data Flow:**\n```\nConnector (Freshsales, Apollo, Gmail)\n  ↓ (emits raw events)\nIW Bridge MCP Emitter\n  ↓ (calls spine-emitter.ts)\nSpine Schema Provider\n  ├─ Resolve: functional_category + connector → capabilities\n  ├─ Map: raw fields → merge fields\n  ├─ Derive: business objects (Account, Contact, Deal)\n  └─ Hydrate: into Spine (D1 + encrypted vault)\n  ↓\nSpine Hydrated (canonical entities)\n```\n\n### 3.2 Current State\n\n✅ `packages/types/src/spine-schema-provider.ts` — formula works\n✅ `packages/connectors/src/spine-emitter.ts` — emitter pattern exists\n✅ `services/connector/src/connector.test.ts` — tests passing\n\n### 3.3 Missing: LiveConnector Implementations\n\n**For Customer-Zero (B2B SaaS GTM):**\n\n1. **Freshsales** (CRM)\n   - Adapter: map Lead, Contact, Deal → Opportunity, Account, Contact\n   - Merge fields: stage, value, probability, owner, fit_score, last_activity\n   - Events: new_contact, deal_moved, activity_logged\n\n2. **Apollo** (Enrichment)\n   - Adapter: map enriched Contact + Company → Contact + Account signals\n   - Merge fields: industry, revenue, employee_count, verified_email, phone\n   - Events: new_enrichment_available\n\n3. **Razorpay/Stripe** (Payments)\n   - Adapter: map Customer, Invoice, Subscription → Account, Invoice, Subscription\n   - Merge fields: MRR, ARR, payment_status, renewal_date, days_overdue, churn_risk\n   - Events: payment_success, subscription_created, invoice_due\n\n4. **Gmail** (Communications)\n   - Adapter: map Thread, Message → Communication (linked to Contact/Deal)\n   - Merge fields: last_email_date, email_sentiment, response_time\n   - Events: new_email, email_opened, reply_received\n\n5. **Zoom** (Meetings)\n   - Adapter: map Recording, Participant → Meeting (linked to Account/Contact)\n   - Merge fields: attendance_rate, duration, sentiment, action_items\n   - Events: meeting_recorded, participant_engaged\n\n### 3.4 Implementation Tasks\n\nFor each connector:\n\n1. **Create Connector Adapter** (`packages/connectors/src/adapters/{connector}.ts`)\n   ```typescript\n   export interface ConnectorAdapter {\n     connectorId: string\n     functionalCategory: 'crm' | 'enrichment' | 'accounting' | 'comms' | 'analytics'\n     mapToMergeFields(rawRecord: any): MergeFieldMap\n     mapToBusinessObjects(record: any): BusinessObject[]\n     getEventStream(): AsyncIterable<ConnectorEvent>\n   }\n   ```\n\n2. **Register in ConnectorRegistry** \n   ```typescript\n   const registry = ConnectorRegistry.getInstance()\n   registry.configure('crm', 'freshsales', { accessToken, instanceUrl })\n   ```\n\n3. **Stream events through Spine Emitter**\n   ```typescript\n   const emitter = getSpineEmitter(tenantId)\n   adapter.getEventStream().pipe((event) => {\n     emitter.emit('connector.event', event)\n   })\n   ```\n\n4. **Write to D1 Vault** (encrypted)\n   - Spine Emitter calls vault.upsert()\n   - Vault encrypts and stores\n   - Searchable via Vectorize index\n\n**Files to Create:**\n- `packages/connectors/src/adapters/freshsales.ts`\n- `packages/connectors/src/adapters/apollo.ts`\n- `packages/connectors/src/adapters/razorpay.ts`\n- `packages/connectors/src/adapters/gmail.ts`\n- `packages/connectors/src/adapters/zoom.ts`\n\n---\n\n## 4. Workspace Loader & Normalizer (\"Creamy Load\")\n\n### 4.1 Architecture\n\n**UI Load Flow:**\n```\nUser navigates to /workspace/{domain}/{view}\n  ↓\nWorkspace Loader (server component)\n  ├─ Read: user role, org, domain mapping\n  ├─ Call: L1Provider → generateProjectionForRole(role)\n  └─ Call: Normalizer → convert Spine data to L1 metrics/widgets\n  ↓\nNormalizer (data layer)\n  ├─ Query: Spine for entity_type + filters\n  ├─ Transform: canonical fields → L1Projection metrics\n  ├─ Bind: widgets to data sources\n  └─ Return: fully hydrated L1Projection\n  ↓\nDashboard renders\n  ├─ Metrics display values\n  ├─ Widgets show data\n  ├─ Quick actions enabled\n  └─ User sees their domain view\n\n```\n\n### 4.2 Current State\n✅ L1Provider generates projections (static data)\n✅ RoleAwareDashboard renders projections\n✅ Domain workspaces render views\n❌ Missing: hydration from actual Spine data\n\n### 4.3 Implementation Tasks\n\n1. **Create Workspace Loader** (`apps/web/components/workspace-loader.tsx`)\n   ```typescript\n   interface WorkspaceLoaderProps {\n     userRole: UserRole\n     domainId: DomainId\n     viewId: string\n   }\n   \n   export async function WorkspaceLoader({ userRole, domainId, viewId }: WorkspaceLoaderProps) {\n     // 1. Get projection from L1Provider\n     const projection = generateProjectionForRole(userRole)\n     \n     // 2. Load Spine data via normalizer\n     const normalizedData = await normalizeSpineForProjection(projection, domainId)\n     \n     // 3. Render view with hydrated data\n     return <WorkspaceView projection={projection} data={normalizedData} />\n   }\n   ```\n\n2. **Create Normalizer** (`apps/web/lib/spine-normalizer.ts`)\n   ```typescript\n   export async function normalizeSpineForProjection(\n     projection: L1Projection,\n     tenantId: string\n   ): Promise<NormalizedData> {\n     // For each metric in projection\n     for (const metric of projection.metrics) {\n       const rawData = await querySpine(metric.data_source, metric.filters)\n       metric.value = calculateValue(rawData)\n       metric.trend = calculateTrend(rawData)\n     }\n     \n     // For each widget\n     for (const widget of projection.widgets) {\n       const rawData = await querySpine(widget.data_source, widget.filters)\n       widget.data = normalizeWidgetData(rawData)\n     }\n     \n     return { metrics: projection.metrics, widgets: projection.widgets }\n   }\n   ```\n\n3. **Create Spine Query Helper** (`apps/web/lib/spine-query.ts`)\n   ```typescript\n   export async function querySpine(\n     dataSource: string,\n     filters: Record<string, any>\n   ): Promise<any[]> {\n     // Call MCP tool via server action\n     const result = await callSpineMcpTool('spine.query', {\n       entity_type: dataSource,\n       ...filters,\n     })\n     return result.entities\n   }\n   ```\n\n4. **Create Server Action for MCP Calls** (`apps/web/app/actions/spine.ts`)\n   ```typescript\n   'use server'\n   \n   export async function callSpineMcpTool(\n     toolName: string,\n     args: Record<string, any>\n   ) {\n     const token = await getMcpToken() // from session\n     const response = await fetch('https://mcp.integratewise.ai/call', {\n       method: 'POST',\n       headers: {\n         'Authorization': `Bearer ${token}`,\n         'Content-Type': 'application/json',\n       },\n       body: JSON.stringify({ tool: toolName, arguments: args }),\n     })\n     return response.json()\n   }\n   ```\n\n**Files to Create:**\n- `apps/web/components/workspace-loader.tsx`\n- `apps/web/lib/spine-normalizer.ts`\n- `apps/web/lib/spine-query.ts`\n- `apps/web/app/actions/spine.ts`\n\n---\n\n## 5. Indirect Data from ChatGPT → Spine\n\n### 5.1 Architecture\n\n**Conversation Capture Flow:**\n```\nUser talks to ChatGPT (via integratewise-connector)\n  ↓ (ChatGPT uses IW MCP tools: spine.query, spine.get, etc.)\nChatGPT generates response + calls spine.propose() or external tool\n  ↓\nIW Bridge captures:\n  ├─ Conversation metadata (user_id, tenant_id, timestamp)\n  ├─ Entity refs (which accounts/deals were mentioned)\n  ├─ Proposed memory (decisions, insights, actions)\n  └─ Executed actions (deals created, meetings scheduled)\n  ↓\nSpine Emitter writes:\n  ├─ Conversation entity\n  ├─ Memory proposals (to Triage Bot queue)\n  └─ Action audit log\n  ↓\nSpine Hydrated with conversation context\n```\n\n### 5.2 Implementation Tasks\n\n1. **Capture Conversation Metadata** (in MCP server)\n   ```typescript\n   // When memory.propose is called from ChatGPT\n   const proposal = {\n     id: generateId(),\n     source_type: 'ai_conversation',\n     source_id: req.headers['x-conversation-id'],\n     proposed_by: 'mcp_agent',\n     tenant_id: actor.tenant_id,\n     user_id: actor.user_id,\n     content_type: args.content_type,\n     title: args.title,\n     body: args.body,\n     entity_refs: args.entity_refs,\n     confidence: args.confidence,\n     evidence: args.evidence,\n     created_at: new Date().toISOString(),\n   }\n   \n   await db.prepare(\n     `INSERT INTO proposals (...) VALUES (...)`\n   ).bind(...).run()\n   ```\n\n2. **Triage Bot for Governance** (`services/governance/src/triage-bot.ts`)\n   - Scores proposals by confidence + entity refs\n   - Auto-approves ≥0.85\n   - Routes 0.60-0.84 to HITL (human)\n   - Rejects <0.60\n\n3. **Sync Actions Back to ChatGPT** (optional)\n   - When MCP tool modifies Spine, notify ChatGPT\n   - ChatGPT can track what it did\n\n**Files to Create:**\n- `services/governance/src/triage-bot.ts`\n- `services/mcp-connector/src/handlers/proposal.ts` (enhanced)\n- `services/governance/src/rules/auto-approve.ts`\n\n---\n\n## 6. Intelligent Overlay + AI Home (Gradual Activation)\n\n### 6.1 Phases\n\n#### Phase 0: Static Summaries (Week 1)\n- AI Home shows **morning brief**\n- Uses Spine data (no AI generation)\n- Hardcoded templates\n\n#### Phase 1: Recommendations (Week 2)\n- Overlays on each domain view\n- \"Account at risk\" badges\n- \"Deal stalled\" warnings\n- Powered by L2 twins (learned patterns)\n\n#### Phase 2: Workflows (Week 3)\n- \"Schedule EBR\" with context\n- \"Send renewal email\" with template\n- \"Escalate ticket\" with suggested priority\n\n#### Phase 3: Agent Loops (Week 4+)\n- Multi-step orchestrations\n- Governed by policies\n- E.g., \"check health, schedule EBR if <70, send playbook\"\n\n### 6.2 Implementation Tasks\n\n1. **Create AI Home View** (`apps/web/components/ai-home/index.tsx`)\n   ```typescript\n   export async function AIHome() {\n     const briefData = await generateMorningBrief()\n     return (\n       <div>\n         <MorningBrief data={briefData} />\n         <OperatingPulse data={briefData} />\n         <RoleBasedSuggestions data={briefData} />\n       </div>\n     )\n   }\n   ```\n\n2. **Create Intelligence Overlay** (`apps/web/components/intelligence-overlay/index.tsx`)\n   - Appears on domain views\n   - Fetches twin insights for visible entities\n   - Renders badges + recommendations\n\n3. **Create Twin Insights Service** (`apps/web/lib/twin-insights.ts`)\n   - Calls L2 context to get learned patterns\n   - Calculates risk scores\n   - Returns actionable recommendations\n\n**Files to Create:**\n- `apps/web/components/ai-home/index.tsx`\n- `apps/web/components/ai-home/morning-brief.tsx`\n- `apps/web/components/ai-home/operating-pulse.tsx`\n- `apps/web/components/intelligence-overlay/index.tsx`\n- `apps/web/lib/twin-insights.ts`\n\n---\n\n## 7. Memory: Conversational + Organizational\n\n### 7.1 Architecture\n\n**Memory Layers:**\n```\nConversational Memory (AI chats)\n  ↓ (patterns + insights extracted)\nOrganizational Memory (shared knowledge)\n  ├─ Doctrine (how we work)\n  ├─ Decisions (what we decided)\n  ├─ Workflows (processes that work)\n  ├─ Insights (patterns we learned)\n  └─ Commitments (what we promised)\n```\n\n### 7.2 Implementation Tasks\n\n1. **Store Conversations** (`services/knowledge/src/handlers/conversation.ts`)\n   - On ChatGPT tool calls, capture thread context\n   - Write to `conversations` table in D1\n   - Link to entities (contacts, deals, etc.)\n\n2. **Triage & Promote** (`services/governance/src/memory-promoter.ts`)\n   - Periodically scan conversations\n   - Find patterns (e.g., \"renewal deals close faster with X playbook\")\n   - Propose promotion to org memory\n\n3. **Memory UI** (`apps/web/components/org-memory/knowledge-base.tsx`)\n   - Browse org memory by category\n   - Search semantic\n   - Contribute new memories\n\n**Files to Create:**\n- `services/knowledge/src/handlers/conversation.ts`\n- `services/governance/src/memory-promoter.ts`\n- `apps/web/components/org-memory/knowledge-base.tsx`\n- `apps/web/components/org-memory/memory-search.tsx`\n\n---\n\n## 8. Integration with Existing System\n\n### How This Connects to What We've Built\n\n**Layer 1: Frontend (Already Built)**\n- ✅ OAuth + role-aware L1Provider\n- ✅ 16 role workbenches\n- ✅ 5 domain workspaces\n- ✅ 40+ action routes, 30+ widget routes\n- ❌ Missing: Loader & Normalizer to hydrate from Spine\n\n**Layer 2: Backend (Partially Built)**\n- ✅ MCP server (production-ready)\n- ✅ Connector Registry (pattern in place)\n- ✅ Spine Schema Provider (formula working)\n- ❌ Missing: Live connector implementations, hydration pipeline\n\n**Layer 3: Data (Partially Built)**\n- ✅ Merge fields + business objects\n- ✅ D1 + encrypted vault infrastructure\n- ❌ Missing: Population from connectors\n\n**Layer 4: AI (Not Started)**\n- ❌ Missing: Onboarding, AI Home, intelligence overlays, memory governance\n\n---\n\n## 9. Critical Path Implementation Order\n\nFor **Customer-Zero GTM (Sales + CS B2B SaaS)**:\n\n1. **Week 1: Auth + Onboarding**\n   - Onboarding wizard\n   - Connector selector UI\n   - MCP token generation\n\n2. **Week 1-2: Live Connectors (Freshsales + Razorpay)**\n   - Freshsales adapter (leads, deals, contacts)\n   - Razorpay adapter (invoices, subscriptions, MRR)\n   - Spine hydration working\n\n3. **Week 2: Workspace Loader + Normalizer**\n   - Dashboard fully hydrated with real data\n   - Metrics calculate from Spine\n   - Widgets display connector data\n\n4. **Week 3: ChatGPT Integration**\n   - Register MCP server as ChatGPT connector\n   - Test spine.query, memory.search from ChatGPT\n   - Capture conversation proposals\n\n5. **Week 3-4: AI Home + Intelligence Overlay**\n   - Morning brief template\n   - Risk badge overlays\n   - Gradual feature rollout\n\n6. **Week 4+: Memory Governance + Org Knowledge**\n   - Triage Bot for proposal scoring\n   - Memory promotion from conversations\n   - Org memory UI\n\n---\n\n## 10. Success Metrics\n\nTrack after launch:\n\n- **Adoption**: % users completing onboarding\n- **Data Freshness**: Last sync time for each connector\n- **Workbench Usage**: Metrics viewed, actions clicked per role/domain\n- **AI Engagement**: Memory searches, proposals generated\n- **Projection Accuracy**: Metrics match source-of-truth (Freshsales, Razorpay, etc.)\n\n---\n\n## 11. Files Summary\n\n### New Files to Create (35 total)\n\n**Onboarding:**\n- `services/gateway/src/onboarding.ts`\n- `apps/web/app/api/v1/onboarding/start/route.ts`\n- `apps/web/app/api/v1/onboarding/complete/route.ts`\n- `apps/web/components/onboarding/connector-selector.tsx`\n- `apps/web/components/onboarding/onboarding-wizard.tsx`\n\n**MCP + Bridge:**\n- `services/mcp-connector/src/middleware/auth.ts`\n- `services/mcp-connector/src/middleware/actor.ts`\n\n**Connectors:**\n- `packages/connectors/src/adapters/freshsales.ts`\n- `packages/connectors/src/adapters/apollo.ts`\n- `packages/connectors/src/adapters/razorpay.ts`\n- `packages/connectors/src/adapters/gmail.ts`\n- `packages/connectors/src/adapters/zoom.ts`\n\n**Workspace Loader:**\n- `apps/web/components/workspace-loader.tsx`\n- `apps/web/lib/spine-normalizer.ts`\n- `apps/web/lib/spine-query.ts`\n- `apps/web/app/actions/spine.ts`\n\n**ChatGPT Integrations:**\n- `services/mcp-connector/src/handlers/proposal.ts` (enhanced)\n- `services/governance/src/triage-bot.ts`\n- `services/governance/src/rules/auto-approve.ts`\n\n**AI Home + Overlay:**\n- `apps/web/components/ai-home/index.tsx`\n- `apps/web/components/ai-home/morning-brief.tsx`\n- `apps/web/components/ai-home/operating-pulse.tsx`\n- `apps/web/components/intelligence-overlay/index.tsx`\n- `apps/web/lib/twin-insights.ts`\n\n**Memory:**\n- `services/knowledge/src/handlers/conversation.ts`\n- `services/governance/src/memory-promoter.ts`\n- `apps/web/components/org-memory/knowledge-base.tsx`\n- `apps/web/components/org-memory/memory-search.tsx`\n\n### Files to Enhance (8 total)\n\n- `services/mcp-connector/src/spine-mcp-server.ts` (add conversation capture)\n- `apps/web/lib/l1-l2-context.ts` (add twin insights context)\n- `apps/web/app/(app)/dashboard/page.tsx` (route to workspace-loader)\n- `apps/web/components/domains/domain-views.tsx` (integrate intelligence overlay)\n- `services/gateway/src/auth.ts` (issue MCP token on callback)\n- `packages/types/src/projection-layers.ts` (add memory types)\n\n---\n\n## Next Actions\n\n1. **Start Onboarding** (Task 1 in TodoList)\n   - Create onboarding APIs\n   - Build connector selector UI\n   - Test flow end-to-end\n\n2. **Implement Live Connectors** (Task 2)\n   - Freshsales adapter (sales + contacts)\n   - Razorpay adapter (invoices + MRR)\n   - Test hydration\n\n3. **Build Workspace Loader** (Task 3)\n   - Connect L1Provider to actual Spine data\n   - Test metrics calculation\n   - Dashboard displays real data\n\n4. **Enable ChatGPT** (Task 4)\n   - Register MCP connector\n   - Test spine.query from ChatGPT\n   - Capture proposals\n\n5. **Add AI Features** (Task 5-6)\n   - AI Home + overlays\n   - Memory governance\n   - Gradual rollout\n\n---\n\n## Summary\n\nThis pipeline transforms IntegrateWise from a **projection engine** into a **living operational platform**:\n\n- Auth + Onboarding makes entry seamless\n- IW Bridge + MCP connects to ChatGPT and tools\n- Spine Hydration populates with real data\n- Workspace Loader renders workbenches with live metrics\n- ChatGPT conversations become learnable memories\n- Intelligence Overlay makes insights visible\n- Organizational Memory preserves knowledge\n\n**Timeline: 4-6 weeks to full activation**\n**Team: 2-3 engineers**\n**Customer-Zero: Sales + CS GTM team (8-10 users)**\n"}