{"slug":"architecture/TWIN_BROWSER_EXECUTION","title":"Twin Browser Execution Architecture","section":"architecture","sectionLabel":"Architecture","status":"unclassified","type":"document","path":"architecture/TWIN_BROWSER_EXECUTION.mdx","bytes":18410,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n**Date:** 2026-06-09\n**Status:** Design Proposal\n**Integration:** Twin Orchestrator + Playwright MCP + Browser Automation Worker\n\n---\n\n## Overview\n\nThe Twin cognitive orchestrator needs to execute actions via browser automation in addition to API-based execution. This enables the Twin to:\n\n1. **Interact with web interfaces without APIs** (legacy tools, admin panels, etc.)\n2. **Perform complex multi-step web workflows** (form filling, data extraction, verification)\n3. **Execute in environments where API access is restricted** (internal tools, authenticated portals)\n4. **Verify execution outcomes visually** (screenshots, DOM inspection)\n5. **Handle human-in-the-loop approvals** (approve in UI, Twin executes in browser)\n\n---\n\n## Architecture Layers\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│ Twin Orchestrator (Cognitive Layer)                         │\n│ - Observes system state                                     │\n│ - Reasons via OpenRouter Agents                             │\n│ - Proposes actions (API or Browser)                         │\n│ - Coordinates execution                                     │\n└──────────────┬──────────────────────────────────────────────┘\n               │\n         ┌─────┴─────┐\n         │           │\n         ▼           ▼\n┌────────────┐  ┌────────────────────┐\n│ Act Worker │  │ Browser Agent      │\n│ (API exec) │  │ (Browser exec)     │\n└────────────┘  └──────────┬─────────┘\n                           │\n                     ┌─────┴─────┐\n                     │           │\n                     ▼           ▼\n            ┌────────────┐  ┌──────────────┐\n            │ Playwright │  │ Puppeteer    │\n            │ MCP Server │  │ (CF Browser) │\n            └────────────┘  └──────────────┘\n```\n\n---\n\n## Execution Decision Matrix\n\n| Scenario                                 | Method      | Reason                        |\n| ---------------------------------------- | ----------- | ----------------------------- |\n| Create HubSpot contact                   | **API**     | Fast, reliable, structured    |\n| Update Jira ticket                       | **API**     | Fast, reliable, structured    |\n| Fill out complex web form                | **Browser** | No API, multi-step validation |\n| Extract data from legacy admin panel     | **Browser** | No API, auth session required |\n| Verify email was sent (in Gmail UI)      | **Browser** | Visual verification needed    |\n| Approve workflow in custom internal tool | **Browser** | No API available              |\n| Multi-step wizard with dynamic fields    | **Browser** | API too complex/unavailable   |\n| Screenshot for audit trail               | **Browser** | Visual evidence required      |\n\n**Decision Logic:**\n\n1. **Prefer API** if available and sufficient\n2. **Use Browser** if:\n   - No API exists\n   - API is too complex (multi-step wizard, dynamic fields)\n   - Visual verification required\n   - Authentication only works in browser session\n   - Human-in-the-loop approval flow requires browser context\n\n---\n\n## Implementation Options\n\n### Option 1: Playwright MCP (Local Development)\n\n**Status:** Already configured in `.vscode/mcp.json`\n\n**Pros:**\n\n- Already available via MCP\n- AI agents (Claude, Kiro) can call Playwright tools directly\n- Rich debugging (screenshots, traces, videos)\n- Good for development/testing\n\n**Cons:**\n\n- Runs on developer machine (not production)\n- Not accessible to Twin Orchestrator worker (edge runtime)\n- Session state management complex\n\n**Use Case:** Development, testing, manual Twin interactions via IDE\n\n---\n\n### Option 2: Cloudflare Browser Rendering API\n\n**Status:** Available (Cloudflare Workers can launch browsers)\n\n**Architecture:**\n\n```typescript\n// services/browser-agent/src/index.ts\n\nexport interface Env {\n  BROWSER: Fetcher; // Cloudflare Browser binding\n  D1: D1Database;\n  CACHE: KVNamespace;\n}\n\n  async fetch(request: Request, env: Env): Promise<Response> {\n    const { action, url, steps } = await request.json();\n\n    // Launch browser via CF\n    const browser = await puppeteer.launch(env.BROWSER);\n    const page = await browser.newPage();\n\n    try {\n      await page.goto(url);\n\n      // Execute steps\n      for (const step of steps) {\n        switch (step.type) {\n          case \"click\":\n            await page.click(step.selector);\n            break;\n          case \"type\":\n            await page.type(step.selector, step.value);\n            break;\n          case \"screenshot\":\n            const screenshot = await page.screenshot();\n            // Store in R2\n            break;\n          case \"extract\":\n            const data = await page.evaluate(step.extractor);\n            break;\n        }\n      }\n\n      return Response.json({ success: true, data });\n    } finally {\n      await browser.close();\n    }\n  },\n};\n```\n\n**Pros:**\n\n- Runs at edge (close to Twin Orchestrator)\n- Scalable (Cloudflare handles browser instances)\n- No local infrastructure needed\n- Session state manageable (KV/Durable Objects)\n\n**Cons:**\n\n- Limited to Puppeteer (no Playwright)\n- Resource limits (CPU time, memory)\n- Cost per execution\n\n**Use Case:** Production browser automation for Twin\n\n---\n\n### Option 3: VPS Browser Agent (Self-Hosted)\n\n**Status:** Can deploy alongside LiteLLM/Twin UI on VPS\n\n**Architecture:**\n\n```\nVPS (187.127.166.105)\n├─ Docker: Open WebUI (Twin UI)\n├─ Docker: LiteLLM\n├─ Docker: Playwright/Puppeteer Agent  ← NEW\n└─ Cloudflare Tunnel → browser-agent.operations.integratewise.ai\n```\n\n**Implementation:**\n\n```typescript\n// integratewise-ops/vps-operations-stack/browser-agent/\n\nconst app = express();\n\napp.post(\"/v1/browser/execute\", async (req, res) => {\n  const { action, url, steps, sessionId } = req.body;\n\n  // Reuse browser context for session\n  const context = await getOrCreateContext(sessionId);\n  const page = await context.newPage();\n\n  try {\n    await page.goto(url);\n    const results = await executeSteps(page, steps);\n\n    res.json({ success: true, results });\n  } finally {\n    if (!req.body.keepSession) {\n      await page.close();\n    }\n  }\n});\n```\n\n**Pros:**\n\n- Full Playwright support (better than Puppeteer)\n- No resource limits (VPS control)\n- Session persistence (cookies, auth state)\n- Can run long-running automation\n- Easy debugging (VNC, remote debug)\n\n**Cons:**\n\n- Single point of failure (1 VPS)\n- Manual scaling\n- Network latency (VPS → web targets)\n\n**Use Case:** Complex, long-running automations; session-heavy workflows\n\n---\n\n### Option 4: Hybrid (Recommended)\n\n**Architecture:**\n\n```\nTwin Orchestrator\n    │\n    ├─→ API Actions → Act Worker → APIs\n    │\n    ├─→ Quick Browser Actions → CF Browser Worker (Puppeteer)\n    │\n    └─→ Complex Browser Actions → VPS Browser Agent (Playwright)\n```\n\n**Decision Logic:**\n\n- **Act Worker (API):** Structured API calls (HubSpot, Jira, etc.)\n- **CF Browser Worker:** Quick browser actions (<10s, simple workflows)\n- **VPS Browser Agent:** Complex browser automation (sessions, multi-step, >10s)\n\n---\n\n## Proposal Structure (Browser Actions)\n\nWhen Twin generates a browser-based proposal:\n\n```typescript\ninterface BrowserActionProposal {\n  tenant_id: string;\n  action_type: \"browser_execute\";\n  target: {\n    url: string;\n    method: \"cf_browser\" | \"vps_playwright\" | \"mcp_local\"; // Routing\n    session_required: boolean;\n    estimated_duration_ms: number;\n  };\n  steps: Array<{\n    type: \"navigate\" | \"click\" | \"type\" | \"select\" | \"wait\" | \"extract\" | \"screenshot\";\n    selector?: string;\n    value?: string;\n    timeout_ms?: number;\n    description: string; // For human approval\n  }>;\n  verification: {\n    method: \"screenshot\" | \"dom_check\" | \"api_callback\";\n    expected?: any;\n  };\n  reasoning: string; // Why browser is needed vs API\n  confidence: number;\n  risk: \"low\" | \"medium\" | \"high\"; // Browser actions are higher risk\n}\n```\n\n**Example:**\n\n```json\n{\n  \"tenant_id\": \"iw-customer-zero\",\n  \"action_type\": \"browser_execute\",\n  \"target\": {\n    \"url\": \"https://internal-tool.company.com/admin/users\",\n    \"method\": \"vps_playwright\",\n    \"session_required\": true,\n    \"estimated_duration_ms\": 8000\n  },\n  \"steps\": [\n    {\n      \"type\": \"navigate\",\n      \"description\": \"Navigate to user admin panel\"\n    },\n    {\n      \"type\": \"click\",\n      \"selector\": \"#add-user-btn\",\n      \"description\": \"Click 'Add User' button\"\n    },\n    {\n      \"type\": \"type\",\n      \"selector\": \"#email-input\",\n      \"value\": \"newuser@example.com\",\n      \"description\": \"Enter user email\"\n    },\n    {\n      \"type\": \"select\",\n      \"selector\": \"#role-dropdown\",\n      \"value\": \"admin\",\n      \"description\": \"Select admin role\"\n    },\n    {\n      \"type\": \"click\",\n      \"selector\": \"#submit-btn\",\n      \"description\": \"Submit form\"\n    },\n    {\n      \"type\": \"wait\",\n      \"selector\": \".success-message\",\n      \"timeout_ms\": 3000,\n      \"description\": \"Wait for success confirmation\"\n    },\n    {\n      \"type\": \"screenshot\",\n      \"description\": \"Capture success state for audit\"\n    }\n  ],\n  \"verification\": {\n    \"method\": \"screenshot\",\n    \"expected\": { \"contains\": \"User created successfully\" }\n  },\n  \"reasoning\": \"Internal tool has no API; browser automation required for user provisioning\",\n  \"confidence\": 0.75,\n  \"risk\": \"medium\"\n}\n```\n\n---\n\n## Twin Orchestrator Integration\n\n### Step 1: Reasoning (Twin Observes → Decides)\n\n```typescript\n// In reasonAboutContext() — Twin determines execution method\nconst reasoning: Reasoning = {\n  suggested_actions: [\n    {\n      type: \"provision_user_in_internal_tool\",\n      target: \"internal-admin-panel\",\n      method: \"browser\", // ← Twin decides browser is needed\n      reasoning: \"Tool has no API; requires browser automation\",\n      impact: \"high\",\n      risk: \"medium\",\n      urgency: \"high\",\n    },\n  ],\n};\n```\n\n### Step 2: Proposal Generation\n\n```typescript\n// In generateProposals() — Twin creates browser action proposal\nconst proposal = {\n  tenant_id,\n  type: \"browser_execute\",\n  action_type: \"provision_user\",\n  target: {\n    url: \"https://internal-tool.company.com/admin/users\",\n    method: \"vps_playwright\" // Route to VPS for session support\n  },\n  steps: [...], // Browser steps\n  requires_approval: true, // Browser actions always require approval\n  risk: \"medium\"\n};\n\n// Submit to Governance\nawait env.MCP_CONNECTOR.fetch(\n  new Request(\"http://internal/v1/proposals/create\", {\n    method: \"POST\",\n    body: JSON.stringify(proposal)\n  })\n);\n```\n\n### Step 3: Approval (HITL)\n\nHuman reviews:\n\n- **What:** \"Provision user in internal admin panel\"\n- **Why:** \"Tool has no API; browser automation required\"\n- **Steps:** List of browser actions (click, type, etc.)\n- **Risk:** Medium (browser can have side effects)\n- **Screenshot preview:** (optional) Show what UI will be interacted with\n\n### Step 4: Execution Coordination\n\n```typescript\n// In coordinateApprovedActions() — Twin dispatches browser execution\nasync function coordinateApprovedActions(env: Env, tenantId: string) {\n  const approved = await fetchApprovedProposals(env, tenantId);\n\n  for (const proposal of approved) {\n    if (proposal.action_type === \"browser_execute\") {\n      // Route to browser agent based on method\n      const agentUrl =\n        proposal.target.method === \"vps_playwright\"\n          ? \"https://browser-agent.operations.integratewise.ai/v1/execute\"\n          : \"https://browser-agent.connect-a1b.workers.dev/v1/execute\";\n\n      const response = await fetch(agentUrl, {\n        method: \"POST\",\n        headers: {\n          \"Content-Type\": \"application/json\",\n          \"x-tenant-id\": tenantId,\n          \"x-proposal-id\": proposal.id,\n        },\n        body: JSON.stringify({\n          url: proposal.target.url,\n          steps: proposal.steps,\n          verification: proposal.verification,\n          sessionId: proposal.session_id, // For session reuse\n        }),\n      });\n\n      const result = await response.json();\n\n      // Store execution outcome\n      await storeExecutionOutcome(env, {\n        proposal_id: proposal.id,\n        status: result.success ? \"completed\" : \"failed\",\n        screenshots: result.screenshots, // Store in R2\n        extracted_data: result.data,\n        error: result.error,\n      });\n    }\n  }\n}\n```\n\n### Step 5: Outcome Monitoring\n\n```typescript\n// In monitorOutcomes() — Twin verifies execution\nasync function monitorOutcomes(env: Env, tenantId: string) {\n  const executions = await fetchRecentExecutions(env, tenantId);\n\n  for (const execution of executions) {\n    if (execution.status === \"failed\") {\n      // Twin reasons about failure\n      const retry = await reasonAboutFailure(env, execution);\n\n      if (retry.should_retry) {\n        // Create new proposal with adjusted steps\n        await generateRetryProposal(env, execution, retry.adjustments);\n      }\n    }\n  }\n}\n```\n\n---\n\n## Implementation Plan\n\n### Phase 1: VPS Browser Agent (Self-Hosted Playwright)\n\n**Goal:** Deploy Playwright agent on VPS for complex browser automation\n\n**Tasks:**\n\n1. Create `integratewise-ops/vps-operations-stack/browser-agent/`\n2. Dockerfile with Playwright + dependencies\n3. Express API: POST `/v1/execute` (steps, session management)\n4. Docker Compose: Add browser-agent service\n5. Cloudflare Tunnel: Expose as `browser-agent.operations.integratewise.ai`\n6. Session store: Redis/SQLite for browser contexts\n7. Screenshot storage: Local filesystem or R2\n\n**Deliverables:**\n\n- `/v1/execute` endpoint (execute browser steps)\n- `/v1/sessions` endpoint (create/restore browser sessions)\n- `/v1/health` endpoint\n- Authentication: X-IW-BROWSER-KEY (in ~/.iw/secrets.env)\n\n---\n\n### Phase 2: CF Browser Worker (Quick Actions)\n\n**Goal:** Deploy Puppeteer worker on CF for quick browser actions\n\n**Tasks:**\n\n1. Create `integratewise-live/services/browser-agent/`\n2. Wrangler config with Browser Rendering API binding\n3. Puppeteer launch + execute steps\n4. Screenshot storage → R2\n5. Service binding from Twin Orchestrator\n\n**Deliverables:**\n\n- `browser-agent.dev.integratewise.ai`\n- POST `/v1/execute` (simpler than VPS agent)\n- Max 10s execution time\n- No session persistence\n\n---\n\n### Phase 3: Twin Integration\n\n**Goal:** Wire Twin Orchestrator to browser agents\n\n**Tasks:**\n\n1. Update `reasonAboutContext()`: Decide API vs Browser\n2. Update `generateProposals()`: Create browser action proposals\n3. Update `coordinateApprovedActions()`: Route to browser agents\n4. Update `monitorOutcomes()`: Verify browser execution outcomes\n5. Add browser action templates (form fill, data extract, screenshot)\n\n**Deliverables:**\n\n- Twin can reason about browser vs API\n- Twin proposes browser actions with steps\n- Twin coordinates browser execution after approval\n- Twin monitors and retries failed browser actions\n\n---\n\n### Phase 4: MCP Playwright Integration (Optional)\n\n**Goal:** Enable local development/testing via Playwright MCP\n\n**Tasks:**\n\n1. Twin → MCP Playwright tools (via Kiro/Claude)\n2. Local browser session management\n3. Developer workflow: Test browser actions locally before deploying\n\n**Use Case:** Development only (not production)\n\n---\n\n## Security Considerations\n\n### 1. Approval Required\n\n**ALL browser actions must be approved.** No auto-execution.\n\nReason: Browser can have unintended side effects (form submissions, account changes, etc.)\n\n### 2. Session Isolation\n\n**Each tenant gets isolated browser context.**\n\nPrevent cross-tenant data leakage via cookies/storage.\n\n### 3. Screenshot Redaction\n\n**Redact sensitive data from screenshots before storage.**\n\nPII, credentials, internal IPs should be masked.\n\n### 4. Audit Trail\n\n**Every browser action logged with:**\n\n- Proposal ID\n- Approved by (user ID)\n- Executed at (timestamp)\n- Steps performed\n- Outcome (success/failure)\n- Screenshots (if applicable)\n\n### 5. Rate Limiting\n\n**Limit browser executions per tenant:**\n\n- Max 10 concurrent browser sessions\n- Max 100 executions per day\n- Max 5 retries per failed execution\n\n### 6. Authentication\n\n**Browser agent endpoints require:**\n\n- X-IW-BROWSER-KEY (for VPS agent)\n- Service binding auth (for CF worker)\n- Tenant ID validation\n\n---\n\n## Cost Estimate\n\n### VPS Browser Agent (Self-Hosted)\n\n- **VPS:** Already running (no additional cost)\n- **Playwright:** Free\n- **Storage:** Minimal (screenshots, sessions)\n- **Total:** ~$0/month (already paying for VPS)\n\n### CF Browser Worker\n\n- **Browser Rendering API:** $0.005 per request\n- **Estimated usage:** 1000 executions/month = $5/month\n- **Workers compute:** Included in paid plan\n- **R2 storage:** <$1/month (screenshots)\n- **Total:** ~$6/month\n\n### Combined\n\n- **Total:** ~$6/month for browser automation\n\n---\n\n## Next Steps\n\n1. **Phase 1 (VPS Browser Agent):** Deploy Playwright agent on VPS\n2. **Test:** Manual browser action via curl\n3. **Phase 2 (CF Browser Worker):** Deploy Puppeteer worker\n4. **Test:** Quick browser action via CF\n5. **Phase 3 (Twin Integration):** Wire Twin to browser agents\n6. **Test:** End-to-end Twin → Proposal → Approval → Browser Execution\n\n---\n\n## Summary\n\n**Twin Browser Execution enables:**\n\n- ✅ Actions in web interfaces without APIs\n- ✅ Complex multi-step workflows\n- ✅ Visual verification (screenshots)\n- ✅ Session-based automation\n- ✅ Hybrid API + Browser execution\n\n**Architecture:**\n\n- **VPS Playwright Agent:** Complex, session-heavy automations\n- **CF Puppeteer Worker:** Quick, simple browser actions\n- **Twin Orchestrator:** Reasons, proposes, coordinates, monitors\n\n**Security:**\n\n- All browser actions require approval\n- Session isolation per tenant\n- Screenshot redaction\n- Full audit trail\n\n**Cost:** ~$6/month (mostly CF Browser Rendering API)\n\n**Ready to implement Phase 1 (VPS Browser Agent)?**\n"}