{"slug":"operations/data-recovery","title":"Data Recovery Runbook","section":"operations","sectionLabel":"Operations","status":"unclassified","type":"document","path":"operations/data-recovery.md","bytes":2452,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\nUse this runbook when production data is missing, corrupted, or written incorrectly and you need a controlled recovery plan.\n\n## Typical Triggers\n\n- destructive or incorrect migration\n- bad normalization or mapping logic\n- accidental overwrite of tenant-scoped records\n- failed release that wrote inconsistent state\n- operator error during replay, redrive, or backfill\n\n## First Response\n\n1. Stop new writes that could worsen the damage.\n2. Define the affected data surface:\n   - Spine\n   - knowledge records\n   - tenant config\n   - billing or audit records\n3. Record timeframe, affected tenants, and suspected change.\n\n## Preserve Evidence\n\n- capture row counts and sample broken records\n- record release version, migration IDs, and timestamps\n- preserve failing payloads or replay inputs\n- confirm whether a backup, snapshot, or PITR window exists\n\n## Recovery Strategy Decision\n\nChoose the least risky option:\n\n| Strategy               | Use when                                                           |\n| ---------------------- | ------------------------------------------------------------------ |\n| Forward fix            | Logic bug can be corrected and data can be repaired in place       |\n| Controlled replay      | Source events or sync inputs still exist and writes are idempotent |\n| Restore and compare    | Blast radius is large or direct repair is unsafe                   |\n| Point-in-time recovery | Database state must be rewound before corruption spread            |\n\n## Safe Recovery Flow\n\n1. Restore to an isolated environment first when possible.\n2. Compare restored data with current production state.\n3. Decide exactly which records or tenants need recovery.\n4. Apply repair or replay in small batches.\n5. Validate correctness before reopening writes broadly.\n\n## Validation Checklist\n\n- affected tenants are restored\n- counts and spot checks match expectations\n- application reads succeed on repaired entities\n- no duplicate writes or replay loops were introduced\n- audit trail documents what was changed\n\n## Escalate Immediately If\n\n- corruption spans many tenants\n- billing, auth, or approval records are involved\n- the only recovery option is destructive rollback in production\n\n## Related Docs\n\n- [deployment-and-rollback.md](deployment-and-rollback.md)\n- [sync-failure-and-dlq-redrive.md](sync-failure-and-dlq-redrive.md)\n- [../NEON_TO_SUPABASE_MIGRATION.md](../NEON_TO_SUPABASE_MIGRATION.md)\n"}