{"slug":"documentation/README_MULTIAGENT","title":"IntegrateWise Multi-Agent Orchestration System","section":"documentation","sectionLabel":"Documentation","status":"unclassified","type":"document","path":"documentation/README_MULTIAGENT.mdx","bytes":12499,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n\n> **Tier C — Historical.** This document is a point-in-time snapshot preserved for reference. For current architecture, see [CANONICAL_PLATFORM_ARCHITECTURE.md](../architecture/CANONICAL_PLATFORM_ARCHITECTURE.md).\n\n## 📋 Quick Navigation\n\n### 🚀 Get Started\n1. **Start Here:** [DEPLOYMENT_READY.md](./DEPLOYMENT_READY.md) - 5-minute overview\n2. **Full Architecture:** [MULTI_AGENT_ORCHESTRATION.md](./MULTI_AGENT_ORCHESTRATION.md) - Complete guide\n3. **Agent Lee Guide:** [AGENT_LEE_INTEGRATION.md](./AGENT_LEE_INTEGRATION.md) - Twin integration details\n4. **Verify Setup:** `./scripts/verify-multi-agent-deployment.sh` - Automated verification\n\n### 📚 Documentation\n\n| Document | Purpose | Length | Audience |\n|----------|---------|--------|----------|\n| [DEPLOYMENT_READY.md](./DEPLOYMENT_READY.md) | Quick status & checklist | 2 pages | Managers, Leads |\n| [MULTI_AGENT_ORCHESTRATION.md](./MULTI_AGENT_ORCHESTRATION.md) | Architecture & deployment | 80+ pages | Engineers, Architects |\n| [AGENT_LEE_INTEGRATION.md](./AGENT_LEE_INTEGRATION.md) | Agent Lee integration | 40+ pages | Twin developers |\n| README_MULTIAGENT.md | This file - index & quick ref | - | Everyone |\n\n---\n\n## 🎯 What Is This System?\n\n**Multi-Agent Orchestration** enables Agent Lee (Twin Agent) to:\n- **Coordinate** with specialized agents (CSM, Analytics, etc.)\n- **Share tools** - agents discover & invoke each other's capabilities\n- **Share memory** - cross-agent context with decay/promotion algorithm\n- **Route work** - pipeline coordinates agent execution\n- **Learn & evolve** - Twin's profile updates from outcomes\n\n---\n\n## ✅ What's Already Built\n\n### Agents\n- ✅ **Agent Lee (Twin)** - Orchestrator with learned profile\n- ✅ **CSM Agent** - Lead scoring & prioritization\n- ✅ **Lead Qualification Agent** - Enrichment & qualification\n- ✅ **Analytics Agent** - Forecasting & trends\n- ✅ **Triage Agent** - Task routing\n\n### Infrastructure\n- ✅ **Agent Registry** (D1 + KV) - agent discovery\n- ✅ **Communication Layer** (D1 + KV) - inter-agent messaging\n- ✅ **Shared Memory** (Durable Objects) - decay algorithm\n- ✅ **Pipeline** - task routing & execution\n- ✅ **Spine** - audit trail & canonical truth\n- ✅ **MCP Server** - 18+ tools exposed\n\n### Performance\n- ✅ Twin observes context: <50ms\n- ✅ Agent discovery: <10ms\n- ✅ Inter-agent message: <20ms\n- ✅ Shared memory query: <30ms\n- ✅ Full proposal: <200ms\n\n---\n\n## 🚀 Quick Start\n\n### 1. Verify Installation\n```bash\n./scripts/verify-multi-agent-deployment.sh --production --verbose\n```\n\n**Expected output:** 10/10 tests passing, \"READY FOR PRODUCTION\"\n\n### 2. Test Agent Communication\n```bash\n# Check Twin Agent\ncurl -X GET https://api.integratewise.ai/v1/agents/lee-twin-1 \\\n  -H \"Authorization: Bearer $TOKEN\"\n\n# Send test message\ncurl -X POST https://api.integratewise.ai/v1/agents/lee-twin-1/send \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\n    \"receiver_agent_id\": \"csm-agent-1\",\n    \"message_type\": \"request\",\n    \"payload\": { \"test\": \"connectivity\" }\n  }'\n```\n\n### 3. Try Agent Lee\n```bash\n# Get a proposal from Agent Lee\ncurl -X POST https://api.integratewise.ai/v1/agents/lee-twin-1/chat \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\n    \"user_message\": \"Should we pursue lead test-123?\",\n    \"tenant_id\": \"integratewise\"\n  }'\n```\n\n---\n\n## 📊 System Architecture\n\n```\nUser Query\n   ↓\nAgent Lee (OBSERVES)\n   ├─ Retrieves: Spine data\n   ├─ Gets: Shared memory\n   └─ Loads: Personal profile\n   ↓\nAgent Lee (DISCOVERS)\n   └─ Queries: \"Who has this capability?\"\n   ↓\nAgent Lee (INVOKES)\n   ├─ Sends message → Specialized Agent\n   ├─ Tracks correlation_id\n   └─ Waits for response\n   ↓\nSpecialized Agent (PROCESSES)\n   └─ Returns result with reasoning\n   ↓\nAgent Lee (SYNTHESIZES)\n   ├─ Combines: Data + Results + Memories\n   └─ Applies: Personal profile\n   ↓\nAgent Lee (PROPOSES)\n   ├─ Creates recommendation\n   ├─ Calculates confidence\n   └─ Flags approval needs\n   ↓\nSpine (RECORDS)\n   ├─ Events → D1 log\n   └─ Audit trail complete\n   ↓\nMemory (UPDATES)\n   ├─ Saves learning globally\n   └─ Available to all agents\n   ↓\nProfile (EVOLVES)\n   ├─ Recent lessons\n   ├─ Success patterns\n   └─ Risk signals\n   ↓\nUser (RECEIVES)\n   ├─ Proposal with confidence\n   ├─ Supporting data\n   └─ Agents consulted\n```\n\n---\n\n## 🔍 Component Locations\n\n### Source Code\n| Component | Location |\n|-----------|----------|\n| Agent Lee | `services/iw-agent-runtime/src/twin-agent.ts` |\n| Agent Registry | `services/mcp-connector/src/agent/registry.ts` |\n| Communication | `services/mcp-connector/src/agent/communication.ts` |\n| Tool Sharing | `services/mcp-connector/src/agent/mcp-tools.ts` |\n| Shared Memory | `services/iw-agent-runtime/src/ecosystem/memory-pipeline.ts` |\n| Pipeline | `services/pipeline/src/index.ts` |\n| Spine | `services/pipeline/src/spine/index.ts` |\n| MCP Server | `services/mcp-connector/src/mcp-server.ts` |\n\n### Storage\n| Storage | Purpose |\n|---------|---------|\n| D1 `agent_registry` | Agent metadata & capabilities |\n| D1 `agent_communication_log` | Message history |\n| KV `agent_msg:*` | Fast message cache (5-min TTL) |\n| KV `tool_registry:*` | Tool discovery cache |\n| Durable Objects | Shared memory, state, profiles |\n| Queues | Async task processing |\n\n---\n\n## 🛠️ Common Tasks\n\n### Add a New Specialized Agent\n```bash\n# 1. Register agent\ncurl -X POST https://api.integratewise.ai/v1/agents/register \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\n    \"agent_id\": \"analytics-2\",\n    \"agent_name\": \"Advanced Analytics\",\n    \"capabilities\": [\"forecast_revenue\", \"predict_churn\"],\n    \"runtime\": \"cloudflare\",\n    \"status\": \"active\"\n  }'\n\n# 2. Agent Lee automatically discovers it\n# (No code changes needed - Agent Lee queries registry)\n\n# 3. Any agent can invoke it\ncurl -X POST https://api.integratewise.api/v1/agents/lee-twin-1/send \\\n  -d '{\n    \"receiver_agent_id\": \"analytics-2\",\n    \"payload\": { \"tool_name\": \"forecast_revenue\", \"account_id\": \"123\" }\n  }'\n```\n\n### Add New Capability to Agent Lee\n```typescript\n// In services/iw-agent-runtime/src/twin-agent.ts\n// 1. Add to capabilities list in agent_registry\n// 2. Implement new method in TwinAgent class\n// 3. Update system prompt if needed\n// 4. Test via chat endpoint\n```\n\n### Query Shared Memory\n```bash\n# Save memory\ncurl -X POST https://api.integratewise.ai/v1/memory/save \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\n    \"content\": \"Best practice: High-value accounts convert at 2x rate\",\n    \"scope\": \"global\",\n    \"tags\": [\"best_practice\", \"lead_qualification\"],\n    \"agent_id\": \"lee-twin-1\"\n  }'\n\n# Query memory\ncurl -X GET \"https://api.integratewise.ai/v1/memory?scope=global&tags=lead_qualification\" \\\n  -H \"Authorization: Bearer $TOKEN\"\n\n# Memory entries include:\n# - relevance_score (with decay applied)\n# - access_count (promotion)\n# - created_at, expires_at\n```\n\n### Monitor Agent Health\n```bash\n# Get all agents\ncurl -X GET https://api.integratewise.api/v1/agents/registry \\\n  -H \"Authorization: Bearer $TOKEN\"\n\n# Check specific agent\ncurl -X GET https://api.integratewise.api/v1/agents/lee-twin-1 \\\n  -H \"Authorization: Bearer $TOKEN\"\n\n# Expected fields: agent_id, status, last_heartbeat, updated_at\n```\n\n---\n\n## 📈 Performance Characteristics\n\n### Latencies\n| Operation | Time | Bottleneck |\n|-----------|------|-----------|\n| Twin context retrieve | <50ms | KV cache |\n| Agent discovery | <10ms | Registry in KV |\n| Inter-agent send | <20ms | Dual-write (D1 + KV) |\n| Memory query | <30ms | Durable Object |\n| Decay calculation | <5ms | Exponential formula |\n| Full proposal | <200ms | Parallel execution |\n\n### Throughput\n- **Twin agents:** 1,000+ concurrent interactions\n- **Agent messages:** 100+ msg/sec per agent\n- **Memory queries:** 1,000+ queries/sec\n- **Pipeline tasks:** 500+ tasks/sec\n\n### Storage\n- **D1 database:** ~10MB/month (logs + events)\n- **KV cache:** ~5MB (with 5-min TTL)\n- **Durable Objects:** Variable (state + memory)\n\n---\n\n## ⚙️ Configuration\n\n### Environment Variables\n```bash\n# Required\nTENANT_ID=integratewise\nAPI_TOKEN=<your_token>\n\n# Optional\nLOG_LEVEL=debug\nMEMORY_DECAY_DAYS=7\nMESSAGE_CACHE_TTL=300\n```\n\n### Deployment Settings\n```toml\n# wrangler.toml\n[env.production]\nservice_bindings = [\n  { binding = \"CONTINUITY\", service = \"continuity-service\" },\n  { binding = \"MCP_CONNECTOR\", service = \"mcp-connector\" },\n  { binding = \"PIPELINE\", service = \"pipeline\" }\n]\n\nd1_databases = [\n  { binding = \"DB\", database_id = \"...\", database_name = \"spine\" }\n]\n\nkv_namespaces = [\n  { binding = \"CACHE\", id = \"...\", preview_id = \"...\" }\n]\n\ndurable_objects = [\n  { name = \"SHARED_MEMORY_DO\", class_name = \"SharedMemoryDO\" }\n]\n\nqueues = [\n  { binding = \"AGENT_QUEUE\", queue = \"agent-tasks\" }\n]\n```\n\n---\n\n## 🐛 Troubleshooting\n\n### Agent Lee Not Responding\n```bash\n# 1. Check registration\ncurl https://api.integratewise.api/v1/agents/lee-twin-1\n\n# 2. Check last heartbeat\n# Should be < 5 minutes ago\n\n# 3. Check service binding\n# Verify CONTINUITY in wrangler.toml\n\n# 4. Check logs\ntail -f logs/iw-agent-runtime.log\n```\n\n### Memory Not Shared\n```bash\n# 1. Verify global scope\nSELECT * FROM memory WHERE scope='global' LIMIT 5;\n\n# 2. Check decay calculation\nage_days = (NOW - created_at) / 86400\ndecay_factor = exp(-age_days / 7)\n\n# 3. Verify access_count incrementing\nSELECT id, access_count, relevance_score FROM memory ORDER BY updated_at DESC;\n```\n\n### Pipeline Tasks Stuck\n```bash\n# 1. Check queue\nSELECT * FROM pipeline_queue WHERE status='queued' LIMIT 10;\n\n# 2. Check agent inbox\ncurl https://api.integratewise.api/v1/agents/csm-agent-1/inbox\n\n# 3. Check error logs\nSELECT * FROM pipeline_errors WHERE created_at > NOW - INTERVAL 1 HOUR;\n```\n\n---\n\n## 📖 Deep Dives\n\n### Understanding Memory Decay\n```\nRelevance Score = Base Score × Decay Factor\n\nDecay Factor = exp(-age_days / 7)\n\nExamples:\n- 0 days old:    score = 1.0 (100%)\n- 7 days old:    score = 0.37 (37%)\n- 14 days old:   score = 0.13 (13%)\n- 21 days old:   score = 0.05 (5%)\n- 30 days old:   expires (TTL)\n\nPromotion:\n- Each access: access_count++\n- Query applies: score *= exp(access_count / 100)\n```\n\n### Agent Registration Flow\n```\n1. New Agent Starts\n2. Registers: POST /v1/agents/register\n3. Registry Stores: D1 agent_registry\n4. Cache Populates: KV agent_registry:*\n5. Other Agents Discover: queryAgentsByCapability()\n6. Cross-Agent Calls: Available immediately\n```\n\n### Message Correlation\n```\nTwin → CSM (Request):\n  {\n    id: \"msg-123\",\n    correlation_id: \"corr-456\",\n    message_type: \"request\",\n    payload: { tool_name: \"score_lead\", ... }\n  }\n\nCSM → Twin (Response):\n  {\n    id: \"msg-789\",\n    correlation_id: \"corr-456\", // ← Same ID\n    message_type: \"response\",\n    payload: { score: 0.92 }\n  }\n\nTwin matches by correlation_id to pair them\n```\n\n---\n\n## 📞 Support\n\n### Documentation\n- 📖 Full guide: [MULTI_AGENT_ORCHESTRATION.md](./MULTI_AGENT_ORCHESTRATION.md)\n- 🤖 Agent Lee: [AGENT_LEE_INTEGRATION.md](./AGENT_LEE_INTEGRATION.md)\n- ✅ Status: [DEPLOYMENT_READY.md](./DEPLOYMENT_READY.md)\n\n### Scripts\n- 🔧 Verify: `./scripts/verify-multi-agent-deployment.sh`\n\n### Code References\n- Twin Agent: `services/iw-agent-runtime/src/twin-agent.ts`\n- MCP Tools: `services/mcp-connector/src/agent/mcp-tools.ts`\n- Registry: `services/mcp-connector/src/agent/registry.ts`\n\n---\n\n## 🎓 Learning Path\n\n### Level 1: Understanding (30 min)\n1. Read this file (README_MULTIAGENT.md)\n2. Review DEPLOYMENT_READY.md\n3. Understand the 10-step flow diagram above\n\n### Level 2: Deployment (1 hour)\n1. Run verification script\n2. Try curl commands above\n3. Monitor agent health\n\n### Level 3: Integration (2 hours)\n1. Read MULTI_AGENT_ORCHESTRATION.md\n2. Understand each component\n3. Review source code\n\n### Level 4: Development (4+ hours)\n1. Read AGENT_LEE_INTEGRATION.md\n2. Study Twin Agent code\n3. Add new capabilities\n4. Build new specialized agents\n\n---\n\n## ✨ Key Takeaways\n\n- ✅ All infrastructure already exists\n- ✅ Zero breaking changes\n- ✅ Fully additive system\n- ✅ Production ready\n- ✅ < 200ms full proposal cycle\n- ✅ Agents discover each other automatically\n- ✅ Memory shared across entire network\n- ✅ Learning loop continuously improves\n- ✅ Complete audit trail\n- ✅ 18+ MCP tools exposed\n\n---\n\n**Status:** 🚀 READY FOR PRODUCTION\n\nNext step: `./scripts/verify-multi-agent-deployment.sh --production --verbose`\n\n"}