{"slug":"platform/22-ai-model-runtime","title":"22 — AI Model Runtime","section":"platform","sectionLabel":"Platform","status":"unclassified","type":"document","path":"platform/22-ai-model-runtime.mdx","bytes":4637,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n> **Extracted from:** IntegrateWise Platform Specification Suite\n> **Region:** third_region\n> **Original line:** 3011\n> **Lines:** 74 | **Chars:** 3,538\n> **Status:** Raw extraction — requires review and canonicalization\n\n22 — AI Model Runtime\n22.1 Responsibilities\nOwn the AI Gateway at the model-routing layer (Cloudflare AI Gateway).\nModel selection, routing, cost optimization, latency routing, fallback hierarchy.\nPrompt versioning, evaluation, safety filters, caching, streaming.\n22.2 Pipeline (preserves user’s ASCII diagram)\nCopyUser Request\n↓\nIntent Detection\n↓\nModel Router\n↓\nModel Selection\n↓\nTool Calls (via 05 Capability Fabric)\n↓\nResponse Validation\n↓\nMemory Commit (via 07 Memory → L1)\n22.3 Inputs\nTwin chat (06).\nWorkflow steps (11) calling capability cap*llm_x.\nAd-hoc admin calls from /ai/run.\n22.4 Outputs\nStreamed (SSE or chunked) or buffered model output.\nMemory intake events (07 / L1).\nCost/latency metrics (16, 30).\n22.5 Events produced\nModelRouted, ModelInvoked, ModelStreamingStarted, ModelCompleted, ModelFailed, ModelFallbackEngaged, PromptVersionUsed, SafetyFilterTriggered, CacheHit, CacheMiss, EvalRunCompleted.\n22.6 Events consumed\nTwinChat, CapabilityInvoked (when capability is llm*\\*), WorkflowStepStarted.\n22.7 APIs\nPOST /ai/run, POST /ai/route, POST /ai/eval, GET /ai/prompts/{prompt_id}/versions, POST /ai/prompts/{prompt_id}/publish.\n22.8 State transitions\nPer invocation: received → intent → routed → invoked → streaming → validating → completed | failed | fallback | aborted. Prompt versions: draft → canary → stable → deprecated → removed. Safety filters: bypass → warn → block → quarantine.\n\n22.9 Failure handling\nPrimary model timeout → fallback to next in hierarchy (22.10).\nFallback exhaustion → degrade to “Last Good Answer” projection (06.10).\nSafety filter trigger: warn returns redacted content, block denies, quarantine sends to admin review.\nEval regression detected on stable prompt version → auto-rollback to last known-good version.\n22.10 Fallback hierarchy (default)\nPrimary (provider × model) per tenant\\*spine_config.model_profile.\nDefault primary: `moonshotai/kimi-k3` (Workers AI, 2.8T params, 1M context, always-on reasoning)\nCheap secondary (smaller/cheaper model same vendor).\nDefault secondary: `@cf/meta/llama-3.3-70b-instruct-fp8-fast` (Workers AI, 70B, fast)\nTertiary: `@cf/meta/llama-3.1-8b-instruct` (Workers AI, 8B, lightweight)\nCross-vendor secondary.\nOpenRouter free models (openrouter/auto)\nCached prior good response (cache TTL ≤ 7 d).\nPersona-template static fallback (\"Last Good Answer\").\nQuarantine + alert.\n\n> **Kimi K3** (`moonshotai/kimi-k3`): Moonshot's flagship 2.8T parameter model available on Workers AI. Key capabilities: 1M token context window (1,048,576 tokens), always-on reasoning via `reasoning_effort` parameter (replaces K2.x `thinking`), native visual understanding, Kimi Delta Attention with Attention Residuals. Available via `env.AI.run()`, REST API, and OpenAI-compatible endpoint. See [Cloudflare docs](https://developers.cloudflare.com/ai/models/moonshotai/kimi-k3/).\n\n22.11 Cost & latency routing\nPer-route cost_budget_per_turn_cents and p95_latency_ms.\nRouter picks provider/model by (capability_needed × cost × latency × traffic_class).\nOff-peak windows may choose smaller models automatically; near SLA breach escalates to premium model.\n22.12 Prompt versioning\nEvery prompt template is a mod_prompt*{id} with version (semver), status, owner, evalset*ref.\nBrowse + diffing in CLI (iw prompt diff v1.4 v1.5).\nCanary rollout through feature flags (15.3, 19.4).\n22.13 Evaluation\nevalset = golden prompts × expected behaviors (text, tool-calls, JSON schema, refusal). Runs on every prompt publish and every model swap.\nRun record: mod_evalrun\\*{id} with pass/fail per expectation; downstream alerting.\n22.14 Safety filters\nCategories: PII exfiltration, jailbreak, toxicity, secret-leak, prompt-injection, vendor-policy-violation.\nLevels: bypass (debug only), warn, block, quarantine.\nTenant overrides require governancePolicy=judicial (09).\n22.15 Caching\nPrompt-level cache keyed by prompt_id + version + params_hash.\nResponse cache keyed by prompt_id + version + params_hash + tenant.tier.\nTTL configurable; default 7 d.\n22.16 Streaming\nWorkspaces and Chat both consume via SSE; mobile via chunked JSON.\nLate chunk miss → fetch from response cache; surface degraded badge.\n22.17 Extension points\nCustom router strategies AI_ROUTER(name).\nCustom safety filters SAFETY_FILTER(name).\nNew model provider adapters via @integratewise/connector-sdk.\n"}