{"slug":"business/FOLDER_MONITOR_SETUP","title":"macOS Folder Monitor Bridge — Setup Guide","section":"business","sectionLabel":"Business","status":"unclassified","type":"document","path":"business/FOLDER_MONITOR_SETUP.mdx","bytes":11362,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\n**Status:** Ready for deployment\n**Created:** 2026-06-09\n**Purpose:** Bridge local filesystem events to Cloudflare Folder Watcher for real-time session log processing\n\n---\n\n## Overview\n\nThe macOS Folder Monitor Bridge watches the `IntegrateWise - Memory/Conversation memory/` directory for file changes and forwards events to the Cloudflare Folder Watcher Durable Object. This enables:\n\n- Real-time session log processing\n- Spine audit log ingestion to D1 + AI Search\n- Triage Bot handoff document processing\n- Agent continuity protocol enforcement\n\n## Architecture\n\n```\nIntegrateWise - Memory/Conversation memory/\n    ↓ (file change detected)\nchokidar watcher (macOS process)\n    ↓ (HTTP POST)\nCloudflare Folder Watcher DO\n    ↓ (audit-handler.ts)\nD1 spine_audit_log + AI Search\n```\n\n---\n\n## Prerequisites\n\n- **Node.js:** v18+ installed\n- **IntegrateWise - Memory:** Cloned at `/Users/$USER/Github/IntegrateWise - Memory/`\n- **Cloudflare Folder Watcher:** Deployed with audit-handler.ts\n- **D1 Database:** `integratewise-spine-cache` with `spine_audit_log` table\n- **PM2 (optional):** For production daemon mode\n\n---\n\n## Installation\n\n### Step 1: Run Setup Script\n\n```bash\ncd /Users/nirmal/Github/integratewise-live\n./scripts/setup-folder-monitor.sh\n```\n\nThis will:\n\n- Check prerequisites\n- Install dependencies (chokidar, node-fetch@2)\n- Verify IntegrateWise - Memory location\n- Display next steps\n\n### Step 2: Set Environment Variable\n\n**Option A: Session-only (temporary)**\n\n```bash\nexport FOLDER_WATCHER_KEY=\"your-secret-key-here\"\n```\n\n**Option B: Persistent (recommended)**\n\n```bash\necho 'export FOLDER_WATCHER_KEY=\"your-secret-key\"' >> ~/.zshrc\nsource ~/.zshrc\n```\n\n**Generate a secure key:**\n\n```bash\nopenssl rand -hex 32\n```\n\n### Step 3: Test Run (Foreground)\n\n```bash\nnode scripts/folder-monitor-bridge.js\n```\n\n**Expected output:**\n\n```\n[2026-06-09T18:30:00.000Z] 🚀 Folder Monitor Bridge started\n[2026-06-09T18:30:00.000Z] 📂 Watching: /Users/nirmal/Github/IntegrateWise - Memory/Conversation memory\n[2026-06-09T18:30:00.000Z] 🎯 Target: https://folder-watcher.connect-a1b.workers.dev/api/ingest\n[2026-06-09T18:30:00.000Z] 🔐 Auth: 8a3f9e2c1b...\n[2026-06-09T18:30:00.000Z] 👤 Tenant: iw-internal\n```\n\n**Test by creating a file:**\n\n```bash\necho \"test\" > \"../IntegrateWise - Memory/Conversation memory/test-$(date +%s).txt\"\n```\n\n**Expected output:**\n\n```\n[2026-06-09T18:30:05.000Z] ➕ add     test-1781025005.txt\n```\n\n**Stop with:** `Ctrl+C`\n\n---\n\n## Production Deployment (PM2)\n\n### Install PM2\n\n```bash\nnpm install -g pm2\n```\n\n### Start as Daemon\n\n```bash\npm2 start scripts/folder-monitor-bridge.js --name iw-folder-monitor\n```\n\n**Expected output:**\n\n```\n┌────┬────────────────────────┬──────────┬──────┬───────────┐\n│ id │ name                   │ mode     │ ↺    │ status    │\n├────┼────────────────────────┼──────────┼──────┼───────────┤\n│ 0  │ iw-folder-monitor      │ fork     │ 0    │ online    │\n└────┴────────────────────────┴──────────┴──────┴───────────┘\n```\n\n### Save Process List\n\n```bash\npm2 save\n```\n\n### Enable Autostart on Boot\n\n```bash\npm2 startup\n# Follow the instructions printed (will require sudo)\n```\n\n---\n\n## Monitoring & Management\n\n### Check Status\n\n```bash\npm2 status\n```\n\n### View Logs\n\n```bash\n# Live logs (tail -f)\npm2 logs iw-folder-monitor\n\n# Last 100 lines\npm2 logs iw-folder-monitor --lines 100\n\n# Error logs only\npm2 logs iw-folder-monitor --err\n```\n\n### Restart\n\n```bash\npm2 restart iw-folder-monitor\n```\n\n### Stop\n\n```bash\npm2 stop iw-folder-monitor\n```\n\n### Delete (remove from PM2)\n\n```bash\npm2 delete iw-folder-monitor\n```\n\n### View Monitoring Dashboard\n\n```bash\npm2 monit\n```\n\n---\n\n## Configuration\n\n### Environment Variables\n\n| Variable             | Required | Default                                                     | Description                                 |\n| -------------------- | -------- | ----------------------------------------------------------- | ------------------------------------------- |\n| `FOLDER_WATCHER_KEY` | ✅       | -                                                           | Authentication token for Folder Watcher API |\n| `FOLDER_WATCHER_URL` | ❌       | `https://folder-watcher.connect-a1b.workers.dev/api/ingest` | Folder Watcher endpoint URL                 |\n\n### Watch Directory\n\nDefault: `/Users/nirmal/Github/IntegrateWise - Memory/Conversation memory/`\n\nTo change, edit `scripts/folder-monitor-bridge.js`:\n\n```javascript\nconst WATCH_DIR = path.resolve(__dirname, \"../../IntegrateWise - Memory/Conversation memory/\");\n```\n\n### Ignored Patterns\n\n- Dotfiles (`.DS_Store`, `.git/`, etc.)\n- `node_modules/`\n- Temp files (`~$`)\n\nTo modify, edit the `ignored` array in `scripts/folder-monitor-bridge.js`.\n\n---\n\n## Event Types\n\nThe bridge sends three event types:\n\n### 1. Add (new file created)\n\n```json\n{\n  \"tenant_id\": \"iw-internal\",\n  \"seq\": 1,\n  \"events\": [\n    {\n      \"action\": \"add\",\n      \"path\": \"2026-06-09/session-kiro-20260609-handoff.md\",\n      \"content\": \"# Agent Session Handoff...\",\n      \"timestamp\": \"2026-06-09T18:30:00.000Z\"\n    }\n  ]\n}\n```\n\n### 2. Change (file modified)\n\n```json\n{\n  \"action\": \"change\",\n  \"path\": \"2026-06-09/session-kiro-20260609-handoff.md\",\n  \"content\": \"# Updated content...\",\n  \"timestamp\": \"2026-06-09T18:35:00.000Z\"\n}\n```\n\n### 3. Unlink (file deleted)\n\n```json\n{\n  \"action\": \"unlink\",\n  \"path\": \"2026-06-09/test-file.txt\",\n  \"content\": null,\n  \"timestamp\": \"2026-06-09T18:40:00.000Z\"\n}\n```\n\n---\n\n## Folder Watcher API\n\n### Endpoint\n\n```\nPOST https://folder-watcher.connect-a1b.workers.dev/api/ingest\n```\n\n### Headers\n\n```\nAuthorization: Bearer {FOLDER_WATCHER_KEY}\nContent-Type: application/json\n```\n\n### Response (Success)\n\n```json\n{\n  \"success\": true,\n  \"processed\": 1,\n  \"audit_events\": 0\n}\n```\n\n### Response (Error)\n\n```json\n{\n  \"error\": \"Unauthorized: Invalid bearer token\"\n}\n```\n\n---\n\n## Troubleshooting\n\n### \"FOLDER_WATCHER_KEY environment variable not set\"\n\n**Solution:**\n\n```bash\nexport FOLDER_WATCHER_KEY=\"your-secret-key\"\n```\n\n### \"Watch directory does not exist\"\n\n**Solution:**\nEnsure IntegrateWise - Memory is cloned at:\n\n```bash\n/Users/$USER/Github/IntegrateWise - Memory/\n```\n\nOr update `WATCH_DIR` in `scripts/folder-monitor-bridge.js`.\n\n### \"Failed to send event: ECONNREFUSED\"\n\n**Causes:**\n\n- Folder Watcher not deployed\n- Incorrect `FOLDER_WATCHER_URL`\n- Network connectivity issue\n\n**Solution:**\n\n```bash\n# Verify Folder Watcher is deployed\nwrangler deployments list --name folder-watcher\n\n# Test endpoint manually\ncurl -X POST https://folder-watcher.connect-a1b.workers.dev/api/ingest \\\n  -H \"Authorization: Bearer $FOLDER_WATCHER_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"tenant_id\":\"test\",\"seq\":1,\"events\":[]}'\n```\n\n### \"Failed to send event: HTTP 401\"\n\n**Cause:** Invalid `FOLDER_WATCHER_KEY`\n\n**Solution:**\nVerify the key matches the one configured in Folder Watcher's environment variables.\n\n### PM2 process keeps restarting\n\n**Diagnosis:**\n\n```bash\npm2 logs iw-folder-monitor --err\n```\n\n**Common causes:**\n\n- Invalid environment variable\n- Permission issues on watch directory\n- Node.js version incompatibility\n\n---\n\n## Testing End-to-End Pipeline\n\n### 1. Create a test audit log\n\n```bash\ncat > \"../IntegrateWise - Memory/Conversation memory/2026-06-09/audit-spine-test-$(date +%s).jsonl\" << 'EOF'\n{\"timestamp\":\"2026-06-09T18:00:00Z\",\"session_id\":\"test-session-123\",\"event\":\"spine_read\",\"entity_type\":\"account\",\"entity_id\":\"acc_test123\",\"via\":\"mcp\",\"tool\":\"spine.entityget\",\"tenant_id\":\"iw-internal\",\"result\":\"success\",\"duration_ms\":45}\n{\"timestamp\":\"2026-06-09T18:00:05Z\",\"session_id\":\"test-session-123\",\"event\":\"spine_write\",\"entity_type\":\"contact\",\"entity_id\":\"cont_test456\",\"via\":\"pipeline\",\"operation\":\"update\",\"fields_changed\":[\"email\",\"phone\"],\"tenant_id\":\"iw-internal\",\"result\":\"success\",\"duration_ms\":120}\nEOF\n```\n\n### 2. Check folder monitor logs\n\n```bash\npm2 logs iw-folder-monitor --lines 20\n```\n\n**Expected:**\n\n```\n➕ add     2026-06-09/audit-spine-test-1781025123.jsonl\n```\n\n### 3. Verify D1 ingestion\n\n```bash\nwrangler d1 execute integratewise-spine-cache \\\n  --command=\"SELECT * FROM spine_audit_log WHERE session_id = 'test-session-123';\" \\\n  --remote\n```\n\n**Expected:** 2 rows returned\n\n### 4. Query AI Search (semantic)\n\nVia Twin or ops dashboard:\n\n```\n\"Show me all Spine operations in session test-session-123\"\n```\n\n**Expected:** Returns both audit events with semantic context\n\n---\n\n## Performance & Limits\n\n- **Watch latency:** ~1 second (awaitWriteFinish stabilityThreshold)\n- **Max file size:** Unlimited (entire file content sent)\n- **Event queue:** In-memory (not persisted across restarts)\n- **Concurrency:** Single-threaded (events sent sequentially)\n- **Network timeout:** 10 seconds per request\n- **Retry logic:** None (failed events logged, not retried)\n\n**Recommendations:**\n\n- For large files (>1MB), consider streaming or chunking\n- For high-volume scenarios, implement event batching\n- For reliability, implement persistent queue (e.g., SQLite)\n\n---\n\n## Security Considerations\n\n- **FOLDER_WATCHER_KEY:** Store securely, rotate periodically\n- **File content:** Entire file content sent over network (use HTTPS only)\n- **Audit logs:** May contain sensitive data (ensure CF Folder Watcher is properly isolated)\n- **PM2 logs:** May contain file content snippets (review log retention policies)\n\n---\n\n## Maintenance\n\n### Update Dependencies\n\n```bash\nnpm update chokidar node-fetch\npm2 restart iw-folder-monitor\n```\n\n### Rotate Authentication Key\n\n1. Generate new key: `openssl rand -hex 32`\n2. Update Folder Watcher environment variable\n3. Update local `FOLDER_WATCHER_KEY`\n4. Restart: `pm2 restart iw-folder-monitor`\n\n### Log Rotation (PM2)\n\nPM2 automatically rotates logs. To configure:\n\n```bash\npm2 install pm2-logrotate\npm2 set pm2-logrotate:max_size 10M\npm2 set pm2-logrotate:retain 7\n```\n\n---\n\n## Integration with Agent Continuity Protocol\n\nThe folder monitor is a critical component of the Agent Continuity Protocol (AGENTS.md Rule 8 + Rule 9):\n\n1. **Agent writes handoff document** → `Conversation memory/[YYYY-MM-DD]/session-*.md`\n2. **Folder monitor detects change** → POST to Folder Watcher\n3. **Folder Watcher processes** → Routes to audit-handler.ts or triage pipeline\n4. **Audit events** → D1 `spine_audit_log` + AI Search indexing\n5. **Triage Bot** → Classifies handoff → `conversational_memory` → HITL → `org_memory`\n\n**Result:** Zero manual intervention, full audit trail, semantic search over all agent sessions.\n\n---\n\n## References\n\n- Agent Continuity Protocol: `/Users/nirmal/Github/AGENTS.md`\n- Folder Watcher Service: `services/folder-watcher/`\n- Audit Handler: `services/folder-watcher/src/audit-handler.ts`\n- D1 Migration: `services/intelligence/migrations/d1_spine_audit_log.sql`\n- Pipeline Wiring Guide: `Conversation memory/2026-06-09/PIPELINE_WIRING_COMPLETE.md`\n\n---\n\n**Questions or issues?** See KANBAN.md TODO section or consult next agent handoff.\n"}