IntegrateWise Multi-Agent Orchestration System

Tier C — Historical. This document is a point-in-time snapshot preserved for reference. For current architecture, see CANONICAL_PLATFORM_ARCHITECTURE.md.

📋 Quick Navigation

🚀 Get Started

1. Start Here: DEPLOYMENT_READY.md - 5-minute overview
2. Full Architecture: MULTI_AGENT_ORCHESTRATION.md - Complete guide
3. Agent Lee Guide: AGENT_LEE_INTEGRATION.md - Twin integration details
4. Verify Setup: ./scripts/verify-multi-agent-deployment.sh - Automated verification

📚 Documentation

Document	Purpose	Length	Audience
DEPLOYMENT_READY.md	Quick status & checklist	2 pages	Managers, Leads
MULTI_AGENT_ORCHESTRATION.md	Architecture & deployment	80+ pages	Engineers, Architects
AGENT_LEE_INTEGRATION.md	Agent Lee integration	40+ pages	Twin developers
README_MULTIAGENT.md	This file - index & quick ref	-	Everyone

---

🎯 What Is This System?

Multi-Agent Orchestration enables Agent Lee (Twin Agent) to:

• Coordinate with specialized agents (CSM, Analytics, etc.)
• Share tools - agents discover & invoke each other's capabilities
• Share memory - cross-agent context with decay/promotion algorithm
• Route work - pipeline coordinates agent execution
• Learn & evolve - Twin's profile updates from outcomes

---

✅ What's Already Built

Agents

• ✅ Agent Lee (Twin) - Orchestrator with learned profile
• ✅ CSM Agent - Lead scoring & prioritization
• ✅ Lead Qualification Agent - Enrichment & qualification
• ✅ Analytics Agent - Forecasting & trends
• ✅ Triage Agent - Task routing

Infrastructure

• ✅ Agent Registry (D1 + KV) - agent discovery
• ✅ Communication Layer (D1 + KV) - inter-agent messaging
• ✅ Shared Memory (Durable Objects) - decay algorithm
• ✅ Pipeline - task routing & execution
• ✅ Spine - audit trail & canonical truth
• ✅ MCP Server - 18+ tools exposed

Performance

• ✅ Twin observes context: <50ms
• ✅ Agent discovery: <10ms
• ✅ Inter-agent message: <20ms
• ✅ Shared memory query: <30ms
• ✅ Full proposal: <200ms

---

🚀 Quick Start

1. Verify Installation

./scripts/verify-multi-agent-deployment.sh --production --verbose

Expected output: 10/10 tests passing, "READY FOR PRODUCTION"

2. Test Agent Communication

# Check Twin Agent
curl -X GET https://api.integratewise.ai/v1/agents/lee-twin-1 \
  -H "Authorization: Bearer $TOKEN"

# Send test message
curl -X POST https://api.integratewise.ai/v1/agents/lee-twin-1/send \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "receiver_agent_id": "csm-agent-1",
    "message_type": "request",
    "payload": { "test": "connectivity" }
  }'

3. Try Agent Lee

# Get a proposal from Agent Lee
curl -X POST https://api.integratewise.ai/v1/agents/lee-twin-1/chat \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "user_message": "Should we pursue lead test-123?",
    "tenant_id": "integratewise"
  }'

---

📊 System Architecture

User Query
   ↓
Agent Lee (OBSERVES)
   ├─ Retrieves: Spine data
   ├─ Gets: Shared memory
   └─ Loads: Personal profile
   ↓
Agent Lee (DISCOVERS)
   └─ Queries: "Who has this capability?"
   ↓
Agent Lee (INVOKES)
   ├─ Sends message → Specialized Agent
   ├─ Tracks correlation_id
   └─ Waits for response
   ↓
Specialized Agent (PROCESSES)
   └─ Returns result with reasoning
   ↓
Agent Lee (SYNTHESIZES)
   ├─ Combines: Data + Results + Memories
   └─ Applies: Personal profile
   ↓
Agent Lee (PROPOSES)
   ├─ Creates recommendation
   ├─ Calculates confidence
   └─ Flags approval needs
   ↓
Spine (RECORDS)
   ├─ Events → D1 log
   └─ Audit trail complete
   ↓
Memory (UPDATES)
   ├─ Saves learning globally
   └─ Available to all agents
   ↓
Profile (EVOLVES)
   ├─ Recent lessons
   ├─ Success patterns
   └─ Risk signals
   ↓
User (RECEIVES)
   ├─ Proposal with confidence
   ├─ Supporting data
   └─ Agents consulted

---

🔍 Component Locations

Source Code

Component	Location
Agent Lee	services/iw-agent-runtime/src/twin-agent.ts
Agent Registry	services/mcp-connector/src/agent/registry.ts
Communication	services/mcp-connector/src/agent/communication.ts
Tool Sharing	services/mcp-connector/src/agent/mcp-tools.ts
Shared Memory	services/iw-agent-runtime/src/ecosystem/memory-pipeline.ts
Pipeline	services/pipeline/src/index.ts
Spine	services/pipeline/src/spine/index.ts
MCP Server	services/mcp-connector/src/mcp-server.ts

Storage

Storage	Purpose
D1 agent_registry	Agent metadata & capabilities
D1 agent_communication_log	Message history
KV agent_msg:*	Fast message cache (5-min TTL)
KV tool_registry:*	Tool discovery cache
Durable Objects	Shared memory, state, profiles
Queues	Async task processing

---

🛠️ Common Tasks

Add a New Specialized Agent

# 1. Register agent
curl -X POST https://api.integratewise.ai/v1/agents/register \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "agent_id": "analytics-2",
    "agent_name": "Advanced Analytics",
    "capabilities": ["forecast_revenue", "predict_churn"],
    "runtime": "cloudflare",
    "status": "active"
  }'

# 2. Agent Lee automatically discovers it
# (No code changes needed - Agent Lee queries registry)

# 3. Any agent can invoke it
curl -X POST https://api.integratewise.api/v1/agents/lee-twin-1/send \
  -d '{
    "receiver_agent_id": "analytics-2",
    "payload": { "tool_name": "forecast_revenue", "account_id": "123" }
  }'

Add New Capability to Agent Lee

// In services/iw-agent-runtime/src/twin-agent.ts
// 1. Add to capabilities list in agent_registry
// 2. Implement new method in TwinAgent class
// 3. Update system prompt if needed
// 4. Test via chat endpoint

Query Shared Memory

# Save memory
curl -X POST https://api.integratewise.ai/v1/memory/save \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "content": "Best practice: High-value accounts convert at 2x rate",
    "scope": "global",
    "tags": ["best_practice", "lead_qualification"],
    "agent_id": "lee-twin-1"
  }'

# Query memory
curl -X GET "https://api.integratewise.ai/v1/memory?scope=global&tags=lead_qualification" \
  -H "Authorization: Bearer $TOKEN"

# Memory entries include:
# - relevance_score (with decay applied)
# - access_count (promotion)
# - created_at, expires_at

Monitor Agent Health

# Get all agents
curl -X GET https://api.integratewise.api/v1/agents/registry \
  -H "Authorization: Bearer $TOKEN"

# Check specific agent
curl -X GET https://api.integratewise.api/v1/agents/lee-twin-1 \
  -H "Authorization: Bearer $TOKEN"

# Expected fields: agent_id, status, last_heartbeat, updated_at

---

📈 Performance Characteristics

Latencies

Operation	Time	Bottleneck
Twin context retrieve	<50ms	KV cache
Agent discovery	<10ms	Registry in KV
Inter-agent send	<20ms	Dual-write (D1 + KV)
Memory query	<30ms	Durable Object
Decay calculation	<5ms	Exponential formula
Full proposal	<200ms	Parallel execution

Throughput

• Twin agents: 1,000+ concurrent interactions
• Agent messages: 100+ msg/sec per agent
• Memory queries: 1,000+ queries/sec
• Pipeline tasks: 500+ tasks/sec

Storage

• D1 database: ~10MB/month (logs + events)
• KV cache: ~5MB (with 5-min TTL)
• Durable Objects: Variable (state + memory)

---

⚙️ Configuration

Environment Variables

# Required
TENANT_ID=integratewise
API_TOKEN=<your_token>

# Optional
LOG_LEVEL=debug
MEMORY_DECAY_DAYS=7
MESSAGE_CACHE_TTL=300

Deployment Settings

# wrangler.toml
[env.production]
service_bindings = [
  { binding = "CONTINUITY", service = "continuity-service" },
  { binding = "MCP_CONNECTOR", service = "mcp-connector" },
  { binding = "PIPELINE", service = "pipeline" }
]

d1_databases = [
  { binding = "DB", database_id = "...", database_name = "spine" }
]

kv_namespaces = [
  { binding = "CACHE", id = "...", preview_id = "..." }
]

durable_objects = [
  { name = "SHARED_MEMORY_DO", class_name = "SharedMemoryDO" }
]

queues = [
  { binding = "AGENT_QUEUE", queue = "agent-tasks" }
]

---

🐛 Troubleshooting

Agent Lee Not Responding

# 1. Check registration
curl https://api.integratewise.api/v1/agents/lee-twin-1

# 2. Check last heartbeat
# Should be < 5 minutes ago

# 3. Check service binding
# Verify CONTINUITY in wrangler.toml

# 4. Check logs
tail -f logs/iw-agent-runtime.log

Memory Not Shared

# 1. Verify global scope
SELECT * FROM memory WHERE scope='global' LIMIT 5;

# 2. Check decay calculation
age_days = (NOW - created_at) / 86400
decay_factor = exp(-age_days / 7)

# 3. Verify access_count incrementing
SELECT id, access_count, relevance_score FROM memory ORDER BY updated_at DESC;

Pipeline Tasks Stuck

# 1. Check queue
SELECT * FROM pipeline_queue WHERE status='queued' LIMIT 10;

# 2. Check agent inbox
curl https://api.integratewise.api/v1/agents/csm-agent-1/inbox

# 3. Check error logs
SELECT * FROM pipeline_errors WHERE created_at > NOW - INTERVAL 1 HOUR;

---

📖 Deep Dives

Understanding Memory Decay

Relevance Score = Base Score × Decay Factor

Decay Factor = exp(-age_days / 7)

Examples:
- 0 days old:    score = 1.0 (100%)
- 7 days old:    score = 0.37 (37%)
- 14 days old:   score = 0.13 (13%)
- 21 days old:   score = 0.05 (5%)
- 30 days old:   expires (TTL)

Promotion:
- Each access: access_count++
- Query applies: score *= exp(access_count / 100)

Agent Registration Flow

1. New Agent Starts
2. Registers: POST /v1/agents/register
3. Registry Stores: D1 agent_registry
4. Cache Populates: KV agent_registry:*
5. Other Agents Discover: queryAgentsByCapability()
6. Cross-Agent Calls: Available immediately

Message Correlation

Twin → CSM (Request):
  {
    id: "msg-123",
    correlation_id: "corr-456",
    message_type: "request",
    payload: { tool_name: "score_lead", ... }
  }

CSM → Twin (Response):
  {
    id: "msg-789",
    correlation_id: "corr-456", // ← Same ID
    message_type: "response",
    payload: { score: 0.92 }
  }

Twin matches by correlation_id to pair them

---

📞 Support

Documentation

• 📖 Full guide: MULTI_AGENT_ORCHESTRATION.md
• 🤖 Agent Lee: AGENT_LEE_INTEGRATION.md
• ✅ Status: DEPLOYMENT_READY.md

Scripts

• 🔧 Verify: ./scripts/verify-multi-agent-deployment.sh

Code References

• Twin Agent: services/iw-agent-runtime/src/twin-agent.ts
• MCP Tools: services/mcp-connector/src/agent/mcp-tools.ts
• Registry: services/mcp-connector/src/agent/registry.ts

---

🎓 Learning Path

Level 1: Understanding (30 min)

1. Read this file (README_MULTIAGENT.md)
2. Review DEPLOYMENT_READY.md
3. Understand the 10-step flow diagram above

Level 2: Deployment (1 hour)

1. Run verification script
2. Try curl commands above
3. Monitor agent health

Level 3: Integration (2 hours)

1. Read MULTI_AGENT_ORCHESTRATION.md
2. Understand each component
3. Review source code

Level 4: Development (4+ hours)

1. Read AGENT_LEE_INTEGRATION.md
2. Study Twin Agent code
3. Add new capabilities
4. Build new specialized agents

---

✨ Key Takeaways

• ✅ All infrastructure already exists
• ✅ Zero breaking changes
• ✅ Fully additive system
• ✅ Production ready
• ✅ < 200ms full proposal cycle
• ✅ Agents discover each other automatically
• ✅ Memory shared across entire network
• ✅ Learning loop continuously improves
• ✅ Complete audit trail
• ✅ 18+ MCP tools exposed

---

Status: 🚀 READY FOR PRODUCTION

Next step: ./scripts/verify-multi-agent-deployment.sh --production --verbose