{"slug":"business/TWIN_VOICE_INTERFACE","title":"Twin Voice Interface Architecture","section":"business","sectionLabel":"Business","status":"current","type":"document","path":"business/TWIN_VOICE_INTERFACE.mdx","bytes":14888,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n**Date:** 2026-06-09\n**Status:** Active (via Open WebUI + x.ai)\n**Integration:** Voice as Interface to Twin Runtime\n\n---\n\n## Current Implementation\n\n### Voice Stack (Active)\n\n```\nUser (speaking)\n    ↓\nOpen WebUI (twin.integratewise.ai)\n    ├─ STT: x.ai Whisper (speech → text)\n    ├─ Chat: Grok-4.3 (text → reasoning)\n    └─ TTS: x.ai Kokoro \"eve\" (text → speech)\n    ↓\nUser (hearing)\n```\n\n**Location:** `integratewise-ops/vps-operations-stack/openwebui/`\n\n**Components:**\n\n- **Open WebUI:** Chat surface with voice UI\n- **x.ai Audio Proxy:** STT (Whisper) + TTS (Kokoro \"eve\")\n- **Grok-4.3:** Foundation model for reasoning\n- **Twin Profiles:** Hermes/Think/Fast (system prompts)\n\n**Status:** ✅ Working (voice-first interface)\n\n---\n\n## The Architecture Problem\n\n### Today (Voice → OWUI → LLM)\n\n```\nVoice Input\n    ↓ (x.ai STT)\nText\n    ↓ (OWUI)\nGrok-4.3 (with Twin profile)\n    ↓\nResponse Text\n    ↓ (x.ai TTS)\nVoice Output\n```\n\n**Limitation:** Voice is just STT + LLM + TTS. No persistence, no orchestration.\n\n### Canonical (Voice → Twin Runtime → Orchestration)\n\n```\nVoice Input\n    ↓ (x.ai STT)\nText\n    ↓ (Twin Runtime API)\nTwin Orchestrator\n    ├─ Observe (system state)\n    ├─ Reason (OpenRouter Agents)\n    ├─ Propose (governed actions)\n    ├─ Coordinate (execution)\n    └─ Monitor (outcomes)\n    ↓\nResponse Text\n    ↓ (x.ai TTS)\nVoice Output\n```\n\n**Benefit:** Voice interactions feed into the persistent Twin (memory, proposals, execution).\n\n---\n\n## Voice as Interface (Not Transport)\n\n### Key Insight\n\n**Voice is NOT a separate system.**\n**Voice is ONE INTERFACE to the Twin Runtime.**\n\n```\nTWIN RUNTIME\n    │\n    ├─→ OWUI (chat)\n    ├─→ Voice (speak)\n    ├─→ API (programmatic)\n    └─→ CLI (terminal)\n```\n\nAll interfaces reach the same Twin Runtime.\n\n### What This Means\n\n**When you speak to the Twin:**\n\n1. STT converts speech → text\n2. Text goes to Twin Runtime (not just LLM)\n3. Twin observes, reasons, proposes (if needed)\n4. Twin responds with text\n5. TTS converts text → speech\n\n**The Twin remembers the conversation.**\n**The Twin can propose actions based on voice input.**\n**The Twin coordinates execution (after approval).**\n\n---\n\n## Integration Architecture\n\n### Phase 1: Voice → Twin Runtime API (Chat Completions)\n\n**Change:** OWUI calls Twin Runtime instead of direct LLM\n\n**Before:**\n\n```\nOWUI → LiteLLM → Grok-4.3 → Response\n```\n\n**After:**\n\n```\nOWUI → Twin Runtime → (observe/reason/respond) → Response\n       ↓\n       (Twin also proposes actions, updates memory, monitors execution)\n```\n\n**Implementation:**\n\n```typescript\n// In Twin Runtime (services/twin-orchestrator/src/index.ts)\n\n/**\n * OpenAI-compatible chat completions endpoint\n * (so OWUI can call Twin Runtime instead of LiteLLM)\n */\napp.post(\"/v1/chat/completions\", async (c) => {\n  const { messages, model, stream } = await c.req.json();\n  const tenantId = c.req.header(\"x-tenant-id\") || \"iw-customer-zero\";\n\n  // Build context (Twin observes system state)\n  const context = await buildTwinContext(c.env, tenantId);\n\n  // Add chat messages to context\n  const chatContext = {\n    ...context,\n    conversation: messages,\n    last_message: messages[messages.length - 1].content,\n  };\n\n  // Twin reasons about context (via OpenRouter Agents)\n  const reasoning = await reasonAboutContext(c.env, chatContext, tenantId);\n\n  // If Twin identifies action needed, create proposal\n  if (reasoning.suggested_actions.length > 0) {\n    await generateProposals(c.env, reasoning, tenantId);\n  }\n\n  // Store conversation in memory\n  await storeConversationalMemory(c.env, {\n    tenant_id: tenantId,\n    session_id: c.req.header(\"x-session-id\") || crypto.randomUUID(),\n    role: \"user\",\n    content: messages[messages.length - 1].content,\n  });\n\n  // Return OpenAI-compatible response\n  const response = reasoning.response || reasoning.context_summary;\n\n  await storeConversationalMemory(c.env, {\n    tenant_id: tenantId,\n    session_id: c.req.header(\"x-session-id\") || crypto.randomUUID(),\n    role: \"assistant\",\n    content: response,\n  });\n\n  return c.json({\n    id: `chatcmpl-${crypto.randomUUID()}`,\n    object: \"chat.completion\",\n    created: Math.floor(Date.now() / 1000),\n    model: model || \"twin-runtime\",\n    choices: [\n      {\n        index: 0,\n        message: {\n          role: \"assistant\",\n          content: response,\n        },\n        finish_reason: \"stop\",\n      },\n    ],\n    usage: {\n      prompt_tokens: 0, // Placeholder\n      completion_tokens: 0,\n      total_tokens: 0,\n    },\n  });\n});\n```\n\n**OWUI Configuration:**\n\n```env\n# In integratewise-ops/vps-operations-stack/openwebui/.env\nOPENAI_API_BASE_URL=https://twin.dev.integratewise.ai/v1\nOPENAI_API_KEY=<twin-runtime-api-key>\n```\n\nNow when you speak to OWUI, it calls Twin Runtime (not LiteLLM).\n\n---\n\n### Phase 2: Voice-Initiated Actions\n\n**Use Case:** \"Twin, create a HubSpot contact for John Doe at Acme Corp\"\n\n**Flow:**\n\n```\nUser speaks\n    ↓ (x.ai STT)\n\"Twin, create a HubSpot contact for John Doe at Acme Corp\"\n    ↓ (Twin Runtime)\nTwin observes: No existing contact for John Doe at Acme Corp\nTwin reasons: User wants to create contact\nTwin proposes: Create HubSpot contact (governed action)\n    ↓ (Twin stores proposal)\nProposal ID: prop-123\n    ↓ (Twin responds)\n\"I've created a proposal to add John Doe at Acme Corp to HubSpot.\n You can approve it in the Awareness layer (L2).\"\n    ↓ (x.ai TTS)\nUser hears response\n```\n\n**Later (when user approves in L2):**\n\n```\nUser clicks \"Approve\" in L2 Awareness UI\n    ↓ (Governance)\nProposal prop-123 approved\n    ↓ (Act service)\nExecute: Create HubSpot contact\n    ↓ (HubSpot API)\nContact created: John Doe at Acme Corp\n    ↓ (Pipeline)\nUpdate Spine with new Person entity\n    ↓ (Outcome)\nSuccess logged, memory updated\n```\n\n**Key:** Voice input can trigger governed actions (just like chat, API, CLI).\n\n---\n\n### Phase 3: Voice-First Briefings\n\n**Use Case:** \"Twin, give me my morning brief\"\n\n**Flow:**\n\n```\nUser speaks\n    ↓ (x.ai STT)\n\"Twin, give me my morning brief\"\n    ↓ (Twin Runtime)\nTwin observes: High-priority signals (3), pending proposals (2), at-risk accounts (1)\nTwin reasons: User wants daily summary\nTwin responds: (structured brief)\n    ↓ (x.ai TTS)\n\"Good morning. You have 3 high-priority signals:\n 1. Account 'Acme Corp' health score dropped to 42 (renewal risk)\n 2. Deal 'Enterprise Plan' stalled for 7 days (no activity)\n 3. Jira ticket 'Login issue' escalated (customer blocked)\n\n You have 2 pending proposals awaiting approval:\n 1. Create follow-up task for Acme Corp account manager\n 2. Notify CSM team about Jira escalation\n\n Would you like to review these now?\"\n    ↓\nUser hears brief\n```\n\n**Key:** Voice becomes proactive briefing interface (not just reactive chat).\n\n---\n\n### Phase 4: Voice Commands (Shortcuts)\n\n**Use Cases:**\n\n- \"Twin, what's the status of Deal X?\"\n- \"Twin, who owns Account Y?\"\n- \"Twin, show me high-priority signals\"\n- \"Twin, approve proposal Z\"\n- \"Twin, create a task to follow up with John\"\n\n**Implementation:**\n\n```typescript\n// In Twin Runtime reasoning\nfunction detectVoiceCommand(input: string): VoiceCommand | null {\n  const patterns = {\n    status_check: /status of (deal|account|task|ticket) (.+)/i,\n    owner_query: /who owns (account|deal|project) (.+)/i,\n    signal_list: /show (me )?(high-priority |pending )?signals/i,\n    approve: /approve proposal (.+)/i,\n    create_task: /create (a )?task to (.+)/i,\n  };\n\n  for (const [type, pattern] of Object.entries(patterns)) {\n    const match = input.match(pattern);\n    if (match) {\n      return { type, params: match.slice(1) };\n    }\n  }\n\n  return null;\n}\n\n// In chat completions handler\nconst command = detectVoiceCommand(lastMessage);\nif (command) {\n  // Execute voice command directly\n  const result = await executeVoiceCommand(c.env, command, tenantId);\n  return formatVoiceResponse(result);\n}\n```\n\n**Key:** Voice becomes efficient command interface (not just conversation).\n\n---\n\n## Voice Modality Considerations\n\n### What Voice Changes\n\n**1. Brevity**\n\n- Voice responses should be concise (not wall of text)\n- Focus on key information (not exhaustive detail)\n\n**2. Clarity**\n\n- Avoid ambiguity (harder to clarify in voice than chat)\n- Confirm actions explicitly (\"I've created a proposal for...\")\n\n**3. Context**\n\n- Voice is often used hands-free (driving, walking)\n- Responses should be self-contained (no \"see link above\")\n\n**4. Memory**\n\n- Voice conversations feel more ephemeral\n- Important info should be repeated/confirmed\n\n### Voice-Optimized Responses\n\n**Bad (chat-style):**\n\n```\nBased on the current system state, I've identified 3 signals\nrequiring your attention:\n\n1. Account: Acme Corp\n   - Health Score: 42 (↓ from 78)\n   - Risk Level: High\n   - Reason: No activity for 14 days\n   - Recommendation: Schedule check-in call\n   - Confidence: 0.87\n\n[... 2 more signals with full detail ...]\n```\n\n**Good (voice-style):**\n\n```\nYou have 3 high-priority signals.\n\nFirst: Acme Corp health score dropped to 42. No activity for 2 weeks.\n\nSecond: Enterprise deal stalled for 7 days.\n\nThird: Customer blocked by login issue.\n\nShould I create follow-up tasks?\n```\n\n---\n\n## Voice + Browser Agent (Advanced)\n\n**Use Case:** \"Twin, check if the HubSpot contact was created\"\n\n**Flow:**\n\n```\nUser speaks\n    ↓ (x.ai STT)\n\"Twin, check if the HubSpot contact was created\"\n    ↓ (Twin Runtime)\nTwin reasons: User wants visual verification\nTwin proposes: Browser agent screenshot of HubSpot contact page\n    ↓ (Governance)\nApproved (low-risk read-only action)\n    ↓ (Act → Browser Agent)\nBrowser navigates to HubSpot, screenshots contact page\n    ↓ (Outcome)\nScreenshot stored: /screenshots/hubspot-contact-123.png\n    ↓ (Twin responds)\n\"Yes, John Doe at Acme Corp is in HubSpot. I've captured a screenshot for verification.\"\n    ↓ (x.ai TTS)\nUser hears confirmation\n```\n\n**Key:** Voice can trigger browser verification (visual audit trail).\n\n---\n\n## Implementation Checklist\n\n### Phase 1: Voice → Twin Runtime ✅ (Designed)\n\n- [ ] Twin Runtime exposes `/v1/chat/completions` (OpenAI-compatible)\n- [ ] Twin Runtime stores conversational memory\n- [ ] Twin Runtime reasons about chat context\n- [ ] Twin Runtime creates proposals from chat\n- [ ] OWUI configured to call Twin Runtime instead of LiteLLM\n- [ ] Test: Voice input → Twin response with memory\n\n---\n\n### Phase 2: Voice-Initiated Actions\n\n- [ ] Twin detects action requests in voice input\n- [ ] Twin creates governed proposals from voice\n- [ ] Twin responds with proposal confirmation\n- [ ] L2 UI shows proposals with \"Voice-initiated\" tag\n- [ ] Test: Voice → Proposal → Approval → Execution\n\n---\n\n### Phase 3: Voice Briefings\n\n- [ ] Twin detects briefing requests (\"morning brief\", \"status update\")\n- [ ] Twin formats structured briefs for TTS\n- [ ] Twin includes signals, proposals, at-risk items\n- [ ] Test: Voice brief → Concise summary → Action follow-up\n\n---\n\n### Phase 4: Voice Commands\n\n- [ ] Twin parses voice commands (shortcuts)\n- [ ] Twin executes read queries directly (no proposal)\n- [ ] Twin creates proposals for write actions\n- [ ] Test: Voice command → Immediate response\n\n---\n\n## Voice Stack (Current vs Canonical)\n\n### Current (OWUI-Centric)\n\n```\nVoice Input\n    ↓ STT (x.ai Whisper)\nText\n    ↓ OWUI\nTwin Profile (Hermes/Think/Fast)\n    ↓ LiteLLM\nGrok-4.3\n    ↓\nResponse\n    ↓ TTS (x.ai Kokoro \"eve\")\nVoice Output\n```\n\n**Status:** Works for conversation, no persistence/orchestration\n\n---\n\n### Canonical (Twin Runtime-Centric)\n\n```\nVoice Input\n    ↓ STT (x.ai Whisper)\nText\n    ↓ OWUI\nTwin Runtime API (/v1/chat/completions)\n    ├─ Observe (system state)\n    ├─ Reason (OpenRouter Agents)\n    ├─ Propose (governed actions)\n    ├─ Store (conversational memory)\n    └─ Respond (with context)\n    ↓\nResponse\n    ↓ TTS (x.ai Kokoro \"eve\")\nVoice Output\n```\n\n**Status:** Needs implementation (Phase 1)\n\n---\n\n## Voice vs Chat vs API\n\n| Dimension         | Voice                 | Chat                  | API                   |\n| ----------------- | --------------------- | --------------------- | --------------------- |\n| **Input**         | Speech (x.ai STT)     | Text (typing)         | JSON (programmatic)   |\n| **Output**        | Speech (x.ai TTS)     | Text (markdown)       | JSON (structured)     |\n| **Context**       | Hands-free, brief     | Detailed, searchable  | Automated, bulk       |\n| **Persistence**   | ✅ (via Twin Runtime) | ✅ (via Twin Runtime) | ✅ (via Twin Runtime) |\n| **Proposals**     | ✅ (voice-initiated)  | ✅ (chat-initiated)   | ✅ (API-initiated)    |\n| **Execution**     | ✅ (after approval)   | ✅ (after approval)   | ✅ (after approval)   |\n| **Orchestration** | ✅ (Twin coordinates) | ✅ (Twin coordinates) | ✅ (Twin coordinates) |\n\n**Key:** All interfaces reach the same Twin Runtime. Voice is not special, just another modality.\n\n---\n\n## Voice as Daily Interface\n\n### Morning Routine\n\n```\nUser: \"Twin, morning brief\"\nTwin: \"Good morning. 3 high-priority signals, 2 pending proposals, 1 at-risk account.\"\nUser: \"Details on the at-risk account\"\nTwin: \"Acme Corp health score dropped to 42. No activity for 14 days. Should I create a follow-up task?\"\nUser: \"Yes, assign to Sarah\"\nTwin: \"Proposal created. Sarah will be notified once you approve in L2.\"\n```\n\n### During Work\n\n```\nUser: \"Twin, status of Enterprise deal\"\nTwin: \"Stalled for 7 days. Last activity: pricing sent. Next step: follow-up call.\"\nUser: \"Create task to call tomorrow\"\nTwin: \"Proposal created. Would you like to review it now?\"\nUser: \"No, I'll approve later\"\n```\n\n### End of Day\n\n```\nUser: \"Twin, what did I accomplish today?\"\nTwin: \"You approved 3 proposals, closed 2 deals, resolved 4 signals. 1 proposal is still pending: follow-up with Acme Corp.\"\nUser: \"Remind me about that tomorrow\"\nTwin: \"Reminder set for 9 AM.\"\n```\n\n---\n\n## Summary\n\n**Voice is an interface to the Twin Runtime.**\n\n**Current Implementation:**\n\n- ✅ Voice works (x.ai STT + TTS via OWUI)\n- ✅ Conversation flows (Grok-4.3 + Twin profiles)\n- ❌ No persistence beyond chat session\n- ❌ No proposals from voice input\n- ❌ No orchestration (just LLM responses)\n\n**Canonical Architecture:**\n\n- OWUI calls Twin Runtime (not LiteLLM)\n- Voice input → Twin observes/reasons/proposes\n- Twin stores conversational memory\n- Twin creates proposals from voice\n- Twin coordinates execution (after approval)\n\n**Next Steps:**\n\n1. Twin Runtime exposes `/v1/chat/completions`\n2. OWUI configured to call Twin Runtime\n3. Twin stores conversational memory\n4. Twin detects action requests in voice input\n5. Twin creates proposals from voice\n\n**Voice becomes a first-class interface to the persistent Twin.**\n\n---\n\n**Ready to implement Phase 1 (Voice → Twin Runtime API)?**\n"}