{"slug":"business/incident-response","title":"Incident Response Runbook","section":"business","sectionLabel":"Business","status":"unclassified","type":"document","path":"business/incident-response.mdx","bytes":2917,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\nUse this runbook for production incidents affecting availability, correctness, security, or high-value user flows.\n\n## When To Use\n\n- `app.integratewise.ai` is down or materially degraded\n- Gateway or a critical Worker is failing\n- Webhooks, connector sync, or approvals are failing at scale\n- Data is being written incorrectly\n- A deployment caused customer-facing impact\n\n## Severity Guide\n\n| Severity | Meaning                                                                      |\n| -------- | ---------------------------------------------------------------------------- |\n| `SEV1`   | Complete outage, widespread auth failure, or unsafe/corrupting writes        |\n| `SEV2`   | Major feature degradation or a critical workflow is blocked for many tenants |\n| `SEV3`   | Localized issue, limited tenant impact, or non-critical degradation          |\n\n## First 15 Minutes\n\n1. Name an incident owner.\n2. Record the start time, impacted surface, and first observed symptom.\n3. Freeze risky deploys and migrations until the issue is understood.\n4. Check the current release or config change that may have triggered the incident.\n5. Confirm blast radius: tenant-specific, connector-specific, or platform-wide.\n\n## Triage Checklist\n\n### Platform checks\n\n- Gateway health and logs\n- Impacted Worker health and logs\n- Spine DB availability and query failures\n- Cloudflare Queue backlog, retry rate, and DLQ depth\n- Recent Pages or Worker deploy history\n\n### Product checks\n\n- Auth and tenant resolution\n- Initial hydration and workspace load\n- Webhook delivery or connector sync path\n- Knowledge search or intelligence actions, if relevant\n\n## Containment Options\n\nChoose the least risky option that stops additional damage:\n\n- Pause connector sync for the affected tenant or provider\n- Disable or slow cron-driven sync\n- Hold approval-driven writes if actions may be unsafe\n- Roll back the last Worker or Pages deployment\n- Stop replay or redrive until the root cause is known\n\n## Recovery Steps\n\n1. Fix or roll back the triggering change.\n2. Verify health on the affected request path.\n3. Run tenant-scoped smoke tests for the broken flow.\n4. Re-enable paused traffic or sync in a controlled order.\n5. Monitor error rate, queue depth, and customer reports for at least one stability window.\n\n## Incident Notes To Capture\n\n- start and end time\n- affected tenants or providers\n- deploy/version identifiers\n- correlation IDs or request IDs\n- exact mitigation taken\n- follow-up actions and doc changes needed\n\n## Exit Criteria\n\n- Error rates are back to baseline\n- Health checks pass\n- Queue backlog and DLQ depth are stable\n- Customer-facing flows are verified\n- A short post-incident summary is recorded\n\n## Related Runbooks\n\n- [deployment-and-rollback.md](deployment-and-rollback.md)\n- [sync-failure-and-dlq-redrive.md](sync-failure-and-dlq-redrive.md)\n- [data-recovery.md](data-recovery.md)\n"}