{"slug":"business/CORE_ENGINE_EXCAVATION","title":"Core Engine Excavation Report","section":"business","sectionLabel":"Business","status":"unclassified","type":"document","path":"business/CORE_ENGINE_EXCAVATION.mdx","bytes":12790,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n> **Tier C — Historical.** This document is a point-in-time snapshot preserved for reference. For current architecture, see [CANONICAL_PLATFORM_ARCHITECTURE.md](../architecture/CANONICAL_PLATFORM_ARCHITECTURE.md).\n\n**The Precious Gold: integratewise-core-engine (Jan 2026)**\n\n---\n\n## What We Found\n\nThe **integratewise-core-engine** is the missing backbone of IntegrateWise. It's the **event ingestion → Spine → AI routing → task generation** pipeline that makes the Human Workbench continuous and adaptive.\n\n---\n\n## Five Versions Across the Ecosystem\n\n| Version | Location | Status | What It Is |\n|---------|----------|--------|-----------|\n| **CURRENT** | NirmalPrinceJ/integratewise-core-engine (standalone) | Updated 20h ago (62676c0) | Live production version |\n| **MONOREPO** | apps/integratewise-core-engine in ai-workspace | Jan 2026 | Archived production code |\n| **v0-VARIANT** | NirmalPrinceJ/v0-integratewise-core-engine | Jan 25 | Abandoned |\n| **v0-VARIANT-2** | NirmalPrinceJ/v0-integratewise-core-engine-yu | Jan 25 | Abandoned |\n| **EMBEDDED** | integrationwise-os app/api route | v11 | Merged into monorepo |\n\n**The Precious Gold Lives In:** Monorepo version (v1.0.0, production-ready)\n\n---\n\n## Architecture: Event → Spine → AI → Tasks\n\n```\n┌─────────────────────────────────────┐\n│  Webhook Ingress (Stripe, Slack,    │\n│  HubSpot, Notion, GitHub, etc.)     │\n└──────────────┬──────────────────────┘\n               │\n               ▼\n┌─────────────────────────────────────┐\n│  CORE ENGINE (Hono Worker)          │\n│  • Event validation                 │\n│  • Normalization                    │\n│  • Spine storage                    │\n└──────────────┬──────────────────────┘\n               │\n               ▼\n┌─────────────────────────────────────┐\n│  SPINE EVENTS TABLE (Neon)          │\n│  • id: UUID                         │\n│  • source: string                   │\n│  • type: string                     │\n│  • timestamp: ISO string            │\n│  • payload: JSONB                   │\n│  • metadata: JSONB                  │\n│  • created_at, updated_at           │\n└──────────────┬──────────────────────┘\n               │\n               ▼\n┌─────────────────────────────────────┐\n│  AI ROUTER                          │\n│  • Model selection (GPT-4o, Haiku)  │\n│  • Task generation                  │\n│  • Insight extraction               │\n└──────────────┬──────────────────────┘\n               │\n               ▼\n┌─────────────────────────────────────┐\n│  AI_TASKS Table / Workbench         │\n│  • Generated tasks (priority, due)  │\n│  • Insights (patterns)              │\n│  • Actions (to Twin or Human)       │\n└─────────────────────────────────────┘\n```\n\n---\n\n## Core Engine: Code Structure\n\n### 1. **index.ts** (Hono Worker Entry Point)\n```typescript\n- GET /health              → Check database connectivity\n- GET /readiness           → K8s readiness probe\n- POST /events             → Receive normalized event\n                             (validate → save → analyze → route to AI async)\n- GET /events              → Get recent events (all sources)\n- GET /events/:source      → Get events by source (e.g., /events/stripe)\n- GET /events/:source/:id  → Get single event\n- POST /ai/route           → Explicit AI routing endpoint\n- Error handler            → Structured error responses\n```\n\n**Key Feature:** Event ingestion is **non-blocking**. Save to database, then route to AI async. Fast feedback to caller, background task generation.\n\n### 2. **ai-router.ts** (Multi-Model Orchestration)\n```typescript\nselectModel(event) {\n  - 'stripe'        → gpt-4o-mini (simple financial)\n  - 'slack/discord' → claude-3-haiku (fast conversation)\n  - 'notion/github' → gpt-4o (complex understanding)\n  - default         → gpt-4o-mini\n}\n\ngenerateTasks(event):\n  - Stripe payment → \"Process successful payment\" (high priority)\n  - Slack mention → \"Respond to Slack mention\" (medium priority)\n  - GitHub PR     → \"Review pull request\" (medium priority)\n\ngenerateInsights(event):\n  - Extract patterns, trends, anomalies\n  - Source-specific analysis\n  - Return as insight array\n```\n\n### 3. **database.ts** (Neon Postgres Operations)\n```typescript\nsaveEvent()        → INSERT + UPSERT (ON CONFLICT)\ngetEventsBySource()→ SELECT by source + limit\ngetEventById()     → SELECT by ID\ngetRecentEvents()  → SELECT recent across all sources\ncheckDatabaseHealth() → Health probe\n```\n\n**Database Client:** @neondatabase/serverless (works in CF Workers)\n\n### 4. **types/spine-event.ts** (Schema Definition)\n```typescript\nSpineEvent = {\n  id:        UUID\n  source:    string  (stripe, slack, hubspot, etc.)\n  type:      string  (payment_intent.succeeded, app_mention, etc.)\n  timestamp: ISO string\n  payload:   Record<string, unknown>  (source-specific data)\n  metadata?: Record<string, unknown>  (optional)\n}\n```\n\n---\n\n## What This Enables\n\n### 1. **Continuous Context Building**\n- Every event → saved to Spine\n- Spine becomes source-of-truth for \"what happened\"\n- No context lost between sessions\n\n### 2. **Adaptive Task Generation**\n- Event → AI analysis → task (not manual)\n- Different models per source (efficiency)\n- Async background processing (fast UX)\n\n### 3. **Multi-Source Event Ingestion**\n- Stripe: Payments, customers, invoices\n- Slack: Mentions, reactions, messages\n- GitHub: PRs, issues, commits\n- Notion: Updates, database entries\n- HubSpot: Deals, companies, contacts\n- Discord: Similar to Slack\n- 100+ more via webhook normalization\n\n### 4. **Memory Without Forgetting**\n- Event stored permanently\n- Queryable by source, type, timestamp\n- Accessible to Twin for daily routines\n- Accessible to Human in workbench\n\n---\n\n## Database: Two New Migrations\n\n```sql\n-- Migration 1: spine_events table\nCREATE TABLE spine_events (\n  id UUID PRIMARY KEY,\n  source VARCHAR NOT NULL,\n  type VARCHAR NOT NULL,\n  timestamp TIMESTAMP NOT NULL,\n  payload JSONB NOT NULL,\n  metadata JSONB,\n  created_at TIMESTAMP DEFAULT NOW(),\n  updated_at TIMESTAMP DEFAULT NOW(),\n  INDEX source_timestamp ON (source, timestamp DESC),\n  INDEX created_at ON (created_at DESC)\n);\n\n-- Migration 2: ai_tasks table (from AI Router)\nCREATE TABLE ai_tasks (\n  id UUID PRIMARY KEY,\n  event_id UUID REFERENCES spine_events(id),\n  title VARCHAR NOT NULL,\n  description TEXT,\n  priority VARCHAR NOT NULL,  -- low, medium, high, critical\n  due_date TIMESTAMP,\n  model VARCHAR NOT NULL,     -- which AI generated this\n  created_at TIMESTAMP DEFAULT NOW(),\n  completed_at TIMESTAMP\n);\n```\n\n---\n\n## Integration Points\n\n### 1. **With Webhook Workers**\n```\nStripe Webhook → normalizes to SpineEvent\n                    → POST /events (Core Engine)\n                       → Save + Analyze + Route\n```\n\n### 2. **With Twin Routines**\n```\nTwin morning routine:\n  1. Query recent events: GET /events?limit=100\n  2. For each event: analyze patterns\n  3. Generate daily briefing\n```\n\n### 3. **With Human Workbench**\n```\nSales user opens dashboard:\n  1. Fetch recent Stripe events (revenue)\n  2. Fetch recent Slack events (conversation)\n  3. Fetch AI tasks generated\n  4. Display as insights + actionable tasks\n```\n\n### 4. **With Spine Memory**\n```\nEvent → Spine → Memory\n  \"Renewal Risk\" detected from 5 Stripe events\n  → Save as Memory: { scope: 'org', content: {...} }\n  → Available to Twin + Human across all surfaces\n```\n\n---\n\n## Stack\n\n| Component | Package | Version | Purpose |\n|-----------|---------|---------|---------|\n| **Runtime** | Cloudflare Workers | CF | Serverless event processing |\n| **Framework** | Hono | 4.6.0 | Lightweight CF Worker HTTP server |\n| **Database** | @neondatabase/serverless | 1.0.0 | Serverless Postgres client |\n| **Schema** | Zod | 3.24.0 | Event validation + TypeScript types |\n| **DevTools** | Wrangler | 4.0.0 | CF Workers CLI |\n| **TypeScript** | ts | 5.0.0 | Full type safety |\n\n---\n\n## Deployment\n\n```bash\n# Local dev\nnpm run dev\n\n# Deploy to staging\nnpm run deploy:staging\n\n# Deploy to production\nnpm run deploy:prod\n\n# Health check\ncurl https://core-engine.integratewise.com/health\n→ { status: \"ok\", database: \"connected\", service: \"integratewise-core-engine\", version: \"1.0.0\" }\n\n# Send event\ncurl -X POST https://core-engine.integratewise.com/events \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"id\": \"550e8400-e29b-41d4-a716-446655440000\",\n    \"source\": \"stripe\",\n    \"type\": \"payment_intent.succeeded\",\n    \"timestamp\": \"2026-01-21T10:30:00Z\",\n    \"payload\": { \"amount\": 5000, \"customer\": \"cus_123\" }\n  }'\n→ { success: true, event_id: \"550e8400...\", analysis: { summary: \"...\", importance: \"high\", actionable: true } }\n```\n\n---\n\n## What Was Lost / What's Missing\n\n### Lost in Consolidation\n- Original v0-integratewise-core-engine variants (abandoned)\n- Desktop client code for event ingestion\n- Mobile event handler\n\n### Still Needed\n- Webhook workers (integratewise-webhooks) normalizes → Core Engine\n- Consumer code that calls /events, /ai/route\n- Twin daily routine service (consumes events)\n- Memory consolidation service (events → memories)\n\n---\n\n## Integration with Platform Activation Framework\n\nThe Core Engine is **Layer 4 (AI Infrastructure)** from Platform Activation:\n\n```\nLayer 1: Human Workbench ← displays tasks/insights from Core Engine\n         ↓\nLayer 2: AI Workbench ← reads events from Core Engine\n         ↓\nLayer 3: Continuity Engine (Spine) ← stores events here\n         ↓\nLayer 4: AI Infrastructure\n         ├─ Core Engine (Event → Spine → AI → Tasks)\n         ├─ MCP routers\n         ├─ ADK discovery\n         ├─ Twin routines\n         └─ Memory consolidation\n         ↓\nLayer 5: Connected Applications (100+)\n```\n\nThe Core Engine **powers the continuity** by:\n1. Saving every event to Spine (persistent context)\n2. Routing to AI for task generation (automated inference)\n3. Generating insights (patterns from raw data)\n4. Enabling Twin to operate 24/7 (not idle)\n\n---\n\n## Why This Is \"Precious Gold\"\n\n1. **Production-Ready**\n   - Tested with real Stripe events\n   - Handles 5+ webhook sources\n   - Async processing (no timeouts)\n   - Error handling + logging\n\n2. **Event-Driven Architecture**\n   - Not polling or cron jobs\n   - Real-time ingestion\n   - Scalable to 1000s of events/min\n\n3. **Multi-Model Awareness**\n   - Routes to right AI model per source\n   - Optimizes cost + latency\n   - Extensible selection logic\n\n4. **Spine Integration**\n   - Single source of truth for events\n   - Queryable by source/type/time\n   - JSONB payload (flexible)\n   - Audit trail (created_at, updated_at)\n\n5. **Async Task Generation**\n   - Events ingested fast (save + return)\n   - Tasks generated in background\n   - No UX blocking\n   - Twin can operate independently\n\n---\n\n## Next Steps: Integration Plan\n\n### Immediate (This Week)\n1. Copy Core Engine code into v0-project/services/core-engine\n2. Integrate with Platform Activation SDK (expose as Capability endpoints)\n3. Test event flow: Webhook → Core Engine → Spine → Workbench\n\n### Short Term (Week 2-3)\n4. Integrate with webhook workers (normalize incoming events)\n5. Wire up Twin daily routines (consume events)\n6. Wire up Memory consolidation (events → memories)\n\n### Medium Term (Weeks 4+)\n7. Add support for 50+ more webhook sources\n8. Implement event deduplication\n9. Add event replay (replay lost events)\n10. Optimize for scale (million+ events/day)\n\n---\n\n## Status\n\n✅ **Core Engine Production Ready** — Extracted from monorepo\n✅ **Schema Defined** — spine_events + ai_tasks tables\n✅ **Stack Validated** — Hono + Neon + Zod working\n✅ **API Documented** — POST /events, GET /events, POST /ai/route\n\n⏳ **Integration Pending** — Copy into v0-project + wire up\n\n---\n\n*Excavation Complete. Precious Gold Located and Documented.*\n*Next: Deploy and activate the core engine in the unified platform.*\n\n"}