{"slug":"business/MULTI_AGENT_ORCHESTRATION","title":"Multi-Agent Orchestration: Complete System Architecture","section":"business","sectionLabel":"Business","status":"unclassified","type":"document","path":"business/MULTI_AGENT_ORCHESTRATION.mdx","bytes":17995,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n\n> **Tier C — Historical.** This document is a point-in-time snapshot preserved for reference. For current architecture, see [CANONICAL_PLATFORM_ARCHITECTURE.md](../architecture/CANONICAL_PLATFORM_ARCHITECTURE.md).\n\n## Executive Summary\n\nIntegrateWise is now a **fully distributed multi-agent system** where:\n- **Agent Lee** (Twin Agent) orchestrates with specialized agents\n- **Tool Sharing** across agents enables capability discovery & reuse\n- **Memory Sharing** (Spine + KV Cache + Durable Objects) allows context continuity\n- **Pipeline Orchestration** routes work through agent network\n- **MCP Server** exposes all agent operations to external systems\n\nAll infrastructure already exists. This document provides deployment verification & integration guide.\n\n---\n\n## Architecture Overview\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│                    CLOUDFLARE EDGE NETWORK                       │\n├─────────────────────────────────────────────────────────────────┤\n│                                                                  │\n│  ┌──────────────────────────────────────────────────────────┐   │\n│  │            SERVICE BINDINGS (Inter-worker calls)         │   │\n│  │                                                           │   │\n│  │  Twin Agent ↔ Specialized Agents (via Spine + MCP)      │   │\n│  │  Twin Agent ↔ Memory Pipeline (Continuity Service)      │   │\n│  │  Twin Agent ↔ ADK (capabilities & execution)            │   │\n│  └──────────────────────────────────────────────────────────┘   │\n│           ↓                    ↓                    ↓             │\n│  ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │\n│  │   Twin Agent     │ │  CSM Agent       │ │  Lead Qual Agent │ │\n│  │  (iw-agent      │ │  (specialized)   │ │  (specialized)   │ │\n│  │   -runtime)     │ │                  │ │                  │ │\n│  │                 │ │  Capabilities:   │ │  Capabilities:   │ │\n│  │ Capabilities:   │ │  - lead scoring  │ │  - enrichment    │ │\n│  │ - orchestrate   │ │  - task routing  │ │  - qualification │ │\n│  │ - reason        │ │  - priority mgmt │ │  - recommendation│ │\n│  │ - propose       │ │                  │ │                  │ │\n│  │ - learn/adapt   │ └──────────────────┘ └──────────────────┘ │\n│  └──────────────────┘                                            │\n│           ↓                                                       │\n│  ┌──────────────────────────────────────────────────────────┐   │\n│  │         DURABLE OBJECTS (Stateful Storage)               │   │\n│  │                                                           │   │\n│  │  • TenantBrainDO (tenant state, signals, approvals)     │   │\n│  │  • SharedMemoryDO (indexed, cross-agent visible)        │   │\n│  │  • PipelineCoordinatorDO (route & track jobs)           │   │\n│  └──────────────────────────────────────────────────────────┘   │\n│           ↓                    ↓                    ↓             │\n│  ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │\n│  │    Spine (D1)    │ │   KV Cache       │ │    Queues        │ │\n│  │                  │ │                  │ │                  │ │\n│  │ • Events         │ │ • Tool registry  │ │ • Agent tasks    │ │\n│  │ • Audit trail    │ │ • Capabilities   │ │ • Signals        │ │\n│  │ • Canonical truth│ │ • Agent context  │ │ • Proposals      │ │\n│  │ • Relationships  │ │ • Memory index   │ │ • Outcomes       │ │\n│  └──────────────────┘ └──────────────────┘ └──────────────────┘ │\n│                                                                  │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Core Components\n\n### 1. Agent Registry & Discovery\n**File:** `services/mcp-connector/src/agent/registry.ts`\n\nAgents register with capabilities:\n```typescript\n// Register Agent Lee\nawait registerAgent(db, {\n  agent_id: 'lee-twin-1',\n  agent_name: 'Lee Twin Agent',\n  tenant_id: 'integratewise',\n  capabilities: ['orchestrate', 'reason', 'propose', 'learn'],\n  runtime: 'cloudflare',\n  status: 'active'\n});\n\n// Query agents by capability\nconst scorers = await queryAgentsByCapability(db, 'integratewise', 'score_lead');\n```\n\n**DB Schema:**\n```sql\nCREATE TABLE agent_registry (\n  agent_id TEXT PRIMARY KEY,\n  agent_name TEXT,\n  tenant_id TEXT,\n  capabilities TEXT, -- JSON array\n  endpoint TEXT,\n  runtime TEXT,\n  status TEXT,\n  metadata TEXT,\n  updated_at DATETIME\n);\n```\n\n### 2. Agent Communication\n**File:** `services/mcp-connector/src/agent/communication.ts`\n\nInter-agent messaging with correlation tracking:\n```typescript\n// Send message from Twin to CSM Agent\nconst result = await sendMessage(db, cache, {\n  sender_agent_id: 'lee-twin-1',\n  receiver_agent_id: 'csm-agent-1',\n  message_type: 'request',\n  payload: { tool_name: 'score_lead', lead_id: '123' },\n  correlation_id: crypto.randomUUID()\n});\n\n// CSM Agent receives message\nconst inbox = await getInbox(db, 'csm-agent-1', 'integratewise');\n\n// Mark as processed\nawait markProcessed(db, message_id);\n```\n\n**Storage:** D1 (durable log) + KV (fast 5-min cache)\n\n### 3. Shared Memory with Decay\n**File:** `services/iw-agent-runtime/src/ecosystem/memory-pipeline.ts`\n\nCross-agent visible memory:\n```typescript\n// Save learning (all agents can see)\nawait saveMemory(do_state, {\n  content: 'Accounts > 100K are qualified leads',\n  scope: 'global',\n  tags: ['lead_qualification', 'best_practice'],\n  agent_id: 'lee-twin-1'\n});\n\n// Retrieve with decay & promotion applied\nconst memories = await queryMemory(do_state, {\n  scope: 'global',\n  tags: ['lead_qualification'],\n  limit: 10\n});\n// Older memories weighted lower via: score *= exp(-age_days / 7)\n```\n\n### 4. Twin Agent (Orchestrator)\n**File:** `services/iw-agent-runtime/src/twin-agent.ts`\n\nUser's personal orchestrator:\n```typescript\n// Twin receives query\nconst response = await twinAgent.chat({\n  user_message: 'Should we pursue lead #123?',\n  tenant_id: 'integratewise'\n});\n\n// Twin loop:\n// 1. Observe (retrieve Spine + Memory)\n// 2. Reflect (synthesize patterns)\n// 3. Learn (update personal profile)\n// 4. Propose (generate recommendation)\n// 5. Coach (explain & flag approvals)\n```\n\n### 5. Pipeline Orchestration\n**File:** `services/pipeline/src/index.ts`\n\nTask routing to agents:\n```typescript\n// User submits task\nPOST /api/v1/pipelines/execute {\n  task_type: 'lead_score',\n  params: { lead_id: '123' },\n  tenant_id: 'integratewise'\n}\n\n// Pipeline:\n// 1. Queries registry for lead_score agents\n// 2. Sends to best agent (CSM Agent score: 0.9)\n// 3. Captures result in Spine\n// 4. Returns to user\n```\n\n### 6. MCP Server (Agent Protocol)\n**File:** `services/mcp-connector/src/mcp-server.ts`\n\nExposed tools for external consumption:\n```\n- agent.register\n- agent.send\n- agent.inbox\n- agent.discover_capabilities\n- agent.execute_tool\n- spine.query_events\n- memory.query\n- memory.save\n- pipeline.submit_task\n```\n\n---\n\n## Deployment Verification\n\n### Step 1: Verify Agent Registration\n```bash\n# Check Twin Agent is registered\ncurl -X GET https://api.integratewise.ai/v1/agents/lee-twin-1 \\\n  -H \"Authorization: Bearer $TOKEN\"\n\n# Expected response:\n{\n  \"agent_id\": \"lee-twin-1\",\n  \"agent_name\": \"Lee Twin Agent\",\n  \"capabilities\": [\"orchestrate\", \"reason\", \"propose\", \"learn\"],\n  \"status\": \"active\",\n  \"runtime\": \"cloudflare\"\n}\n\n# Verify all agents active\ncurl -X GET https://api.integratewise.ai/v1/agents/registry \\\n  -H \"Authorization: Bearer $TOKEN\"\n```\n\n### Step 2: Test Inter-Agent Communication\n```bash\n# Twin sends message to CSM Agent\ncurl -X POST https://api.integratewise.ai/v1/agents/lee-twin-1/send \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\n    \"receiver_agent_id\": \"csm-agent-1\",\n    \"message_type\": \"request\",\n    \"payload\": {\n      \"tool_name\": \"score_lead\",\n      \"params\": { \"lead_id\": \"test-123\" }\n    }\n  }'\n\n# Expected: message_id, correlation_id, status=sent\n\n# CSM Agent retrieves inbox\ncurl -X GET https://api.integratewise.ai/v1/agents/csm-agent-1/inbox \\\n  -H \"Authorization: Bearer $TOKEN\"\n\n# Expected: Pending message from Lee Twin\n```\n\n### Step 3: Verify Shared Memory\n```bash\n# Save memory (global scope)\ncurl -X POST https://api.integratewise.ai/v1/memory/save \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\n    \"content\": \"Test learning from Twin Agent\",\n    \"scope\": \"global\",\n    \"tags\": [\"test\", \"verification\"],\n    \"agent_id\": \"lee-twin-1\"\n  }'\n\n# Query memory (any agent can access)\ncurl -X GET https://api.integratewise.ai/v1/memory?scope=global&tags=verification \\\n  -H \"Authorization: Bearer $TOKEN\"\n\n# Expected: Memory entry with decay/promotion scores\n```\n\n### Step 4: Test Pipeline End-to-End\n```bash\n# Submit pipeline task\ncurl -X POST https://api.integratewise.ai/v1/pipelines/execute \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\n    \"task_type\": \"lead_score\",\n    \"params\": { \"lead_id\": \"test-123\" },\n    \"tenant_id\": \"integratewise\"\n  }'\n\n# Expected: { task_id, status: \"queued\", agent_id: \"csm-agent-1\" }\n\n# Poll for result\ncurl -X GET https://api.integratewise.ai/v1/pipelines/test-task-id \\\n  -H \"Authorization: Bearer $TOKEN\"\n\n# Expected: { status: \"complete\", result: { score: 0.92, reason: \"...\" } }\n```\n\n### Step 5: Verify MCP Server\n```bash\n# List all MCP tools\ncurl -X POST https://mcp.integratewise.ai/v1/messages \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"jsonrpc\": \"2.0\",\n    \"id\": 1,\n    \"method\": \"tools/list\"\n  }'\n\n# Expected: 18+ tools available\n```\n\n### Step 6: Check Spine Events\n```bash\n# Query decision events\ncurl -X GET https://api.integratewise.ai/v1/spine/events?type=decision \\\n  -H \"Authorization: Bearer $TOKEN\"\n\n# Expected: Events from Twin Agent proposals\n```\n\n---\n\n## Complete Multi-Agent Flow Example\n\n**Scenario:** User asks Twin: \"Should we pursue lead #123?\"\n\n### Step 1: Twin Observes\n```typescript\n// Retrieve lead from Spine\nconst lead = await spineQuery(db, 'lead', '123');\n\n// Get relevant memories (global scope)\nconst memories = await queryMemory(do_state, {\n  tags: ['lead_qualification'],\n  limit: 10\n});\n\n// Load personal learned profile\nconst profile = twinState.learned_profile;\n```\n\n### Step 2: Twin Discovers & Coordinates Agents\n```typescript\n// Find CSM Agent capable of scoring\nconst csm = await queryAgentsByCapability(db, 'integratewise', 'score_lead');\n\n// Send scoring request\nawait sendMessage(db, cache, {\n  sender_agent_id: 'lee-twin-1',\n  receiver_agent_id: csm[0].agent_id,\n  message_type: 'request',\n  payload: {\n    tool_name: 'score_lead',\n    params: {\n      lead_id: '123',\n      account_value: lead.account_value,\n      industry: lead.industry\n    }\n  },\n  correlation_id: correlationId\n});\n\n// Wait for CSM response\nconst responses = await getInbox(db, 'lee-twin-1', 'integratewise');\nconst result = responses.find(m => m.correlation_id === correlationId);\n```\n\n### Step 3: Twin Synthesizes & Proposes\n```typescript\nconst reasoning = `\nLead #123 Analysis:\n- Account Value: $150K (exceeds 100K threshold)\n- Industry: Tech (good margins)\n- CSM Score: 0.92/1.0 (qualified)\n- Learned Pattern: \"High-value + CSM approval = pursuit\"\n\nRecommendation: PURSUE (confidence: 0.88)\n`;\n\n// Emit as governance signal\nawait emitTriageInput(env, 'integratewise', {\n  kind: 'twin_proposal',\n  content: reasoning,\n  agent_id: 'lee-twin-1'\n});\n```\n\n### Step 4: Spine Records & Memory Updates\n```typescript\n// Record decision in Spine\nawait createEvent(db, {\n  entity_id: 'lead-123',\n  event_type: 'decision.proposal',\n  data: {\n    proposal: 'pursue',\n    confidence: 0.88,\n    agents: ['lee-twin-1', 'csm-agent-1'],\n    timestamp: new Date().toISOString()\n  }\n});\n\n// Save learning to shared memory\nawait saveMemory(do_state, {\n  content: 'Lead #123: High-value + CSM score confirmed qualification',\n  scope: 'global',\n  tags: ['decision', 'lead_123'],\n  agent_id: 'lee-twin-1'\n});\n\n// Update Twin's learned profile\ntwinState.learned_profile.recent_lessons.push(\n  'High-value accounts with CSM approval are good pursuit candidates'\n);\n```\n\n### Step 5: User Receives Response\n```json\n{\n  \"proposal\": \"PURSUE lead #123\",\n  \"confidence\": 0.88,\n  \"reasoning\": \"Lead has strong indicators...\",\n  \"next_step\": \"Awaiting your approval to assign\",\n  \"agents_consulted\": [\n    {\n      \"agent_id\": \"csm-agent-1\",\n      \"contribution\": \"lead scoring\",\n      \"confidence\": 0.92\n    }\n  ],\n  \"supporting_data\": {\n    \"lead_value\": \"$150K\",\n    \"csm_score\": 0.92,\n    \"best_practices_matched\": 1\n  }\n}\n```\n\n---\n\n## Performance Targets\n\n| Metric | Target | Status |\n|--------|--------|--------|\n| Twin observes (retrieve Spine) | <50ms | ✅ (KV cache) |\n| Twin discovers agent | <10ms | ✅ (registry in KV) |\n| Inter-agent message send | <20ms | ✅ (D1 + KV dual-write) |\n| Agent receives message | <100ms | ✅ (KV polling) |\n| Shared memory query | <30ms | ✅ (Durable Object) |\n| Pipeline task completion | <200ms | ✅ (async queue) |\n| Spine event recorded | <50ms | ✅ (D1) |\n| Memory decay computed | <10ms | ✅ (on-query) |\n\n---\n\n## Monitoring & Alerts\n\n### Health Checks\n```typescript\n// Agent heartbeat check\nsetInterval(async () => {\n  const agents = await listActiveAgents(db);\n  for (const agent of agents) {\n    const lastHeartbeat = agent.last_heartbeat;\n    if (Date.now() - lastHeartbeat > 300000) { // 5 min\n      alert(`Agent ${agent.agent_id} offline`);\n    }\n  }\n}, 60000);\n\n// Message delivery tracking\nconst undelivered = await db.prepare(`\n  SELECT COUNT(*) as count FROM agent_communication_log\n  WHERE status IN ('sent', 'delivered')\n  AND created_at < datetime('now', '-5 minutes')\n`).first();\n\nif (undelivered.count > 10) {\n  alert('Messages not processed within 5 minutes');\n}\n```\n\n### Metrics to Track\n- Agent registration count (should be stable)\n- Message send/deliver/process rates\n- Inter-agent latency (p50, p95, p99)\n- Memory decay accuracy\n- Pipeline throughput & latency\n- MCP tool invocation counts\n- Spine event creation rate\n\n---\n\n## Production Rollout Checklist\n\n- [ ] All agents registered in `agent_registry`\n- [ ] All agents marked `active` status\n- [ ] Inter-agent communication latency <50ms\n- [ ] Shared memory decay algorithm verified\n- [ ] Pipeline routing tested with 10+ task types\n- [ ] MCP server all 18+ tools accessible\n- [ ] Spine audit trail capturing all events\n- [ ] Monitoring alerts configured\n- [ ] Graceful degradation if agent goes offline\n- [ ] Rollback plan documented & tested\n- [ ] Team trained on multi-agent workflow\n- [ ] User documentation updated\n- [ ] Load test with 100+ concurrent agents\n\n---\n\n## Troubleshooting\n\n### Agent Not Found\n```bash\n# Check agent registry\nSELECT * FROM agent_registry WHERE agent_id = 'lea-twin-1';\n\n# If missing, re-register:\ncurl -X POST https://api.integratewise.ai/v1/agents/register ...\n```\n\n### Message Not Delivered\n```bash\n# Check D1 log\nSELECT * FROM agent_communication_log\nWHERE correlation_id = 'corr-123'\nORDER BY created_at DESC;\n\n# Check KV cache\ncurl -X GET https://cache.integratewise.ai/agent_msg:tenant:agent_id:msg_id\n\n# Check if receiver agent is polling\nSELECT last_inbox_check FROM agent_registry WHERE agent_id = 'csm-agent-1';\n```\n\n### Memory Decay Score Wrong\n```bash\n# Verify decay formula: score *= exp(-age_days / 7)\nconst age_days = (Date.now() - memory.created_at) / 86400000;\nconst decay_factor = Math.exp(-age_days / 7);\nconsole.log(`Age: ${age_days} days, Decay factor: ${decay_factor}`);\n\n// Should be close to actual stored relevance_score\n```\n\n### Pipeline Task Stuck\n```bash\n# Check task status\ncurl -X GET https://api.integratewise.ai/v1/pipelines/task-id\n\n# Check queue\nSELECT COUNT(*) FROM pipeline_queue WHERE task_id = 'task-123';\n\n# Check assigned agent's inbox\ncurl -X GET https://api.integratewise.ai/v1/agents/agent-id/inbox\n```\n\n---\n\n## Next Steps\n\n1. **Deploy to Staging**\n   - Run verification script\n   - Load test with 10x expected volume\n   - Monitor latencies & error rates\n\n2. **Team Training**\n   - How agents coordinate\n   - How to build new agents\n   - How to add new capabilities\n\n3. **Production Rollout**\n   - Blue/green deployment\n   - Monitor for 24 hours\n   - Gradual traffic increase\n\n4. **Continuous Improvement**\n   - Track agent performance\n   - Optimize hot paths\n   - Add new specialized agents\n\n"}