{"slug":"operations/sync-failure-and-dlq-redrive","title":"Sync Failure And DLQ Redrive Runbook","section":"operations","sectionLabel":"Operations","status":"unclassified","type":"document","path":"operations/sync-failure-and-dlq-redrive.md","bytes":2450,"description":null,"updated":null,"source":"integratewise/Integratewise-docs-wiki","content":"\nUse this runbook for connector sync failures, pipeline processing failures, queue backlog spikes, or DLQ redrive operations.\n\n## Symptoms\n\n- tenant sync is stuck or never completes\n- queue backlog is growing\n- retries are climbing rapidly\n- `dead_letter_queue` or ops DLQ volume spikes\n- normalized entities are missing after a sync or webhook burst\n\n## Determine The Failing Stage\n\nCheck which path is failing:\n\n| Area                  | What to inspect                                               |\n| --------------------- | ------------------------------------------------------------- |\n| Webhook ingress       | signature verification, dedupe, provider payload shape        |\n| Connector sync        | cron trigger, job creation, provider API errors, cursor state |\n| Pipeline / Normalizer | validation errors, mapping errors, idempotency collisions     |\n| Intelligence DLQ      | downstream analysis failures or governance path failures      |\n\n## Containment\n\nBefore redriving anything:\n\n1. Pause the noisy source if failures are still being produced.\n2. Confirm whether the issue is transient or deterministic.\n3. Capture sample failing payloads, tenant IDs, providers, and correlation IDs.\n\n## Diagnose The Root Cause\n\n- transient infra issue: provider outage, network issue, secret drift\n- deterministic schema issue: payload shape, mapping bug, validation mismatch\n- write-path issue: DB constraint, missing tenant config, stale contract\n\nFix the root cause before replaying.\n\n## Redrive Rules\n\n- redrive one tenant or provider at a time\n- preserve correlation and audit context where possible\n- rely on idempotent writes rather than manual dedupe hacks\n- never bulk-replay the entire DLQ blindly\n\n## Suggested Redrive Flow\n\n1. Query the failing records or DLQ entries.\n2. Narrow to the smallest safe replay batch.\n3. Re-enqueue to the correct queue only after the fix is live.\n4. Monitor:\n   - retry count\n   - DLQ depth\n   - resulting Spine writes\n   - duplicate side effects\n\n## Verification\n\n- connector status returns healthy\n- backlog drains instead of growing\n- expected entities appear in Spine or downstream read surfaces\n- no new DLQ spike from the same root cause\n\n## Related Docs\n\n- [../EVENT_DRIVEN_QUEUE_CHECKLIST.md](../EVENT_DRIVEN_QUEUE_CHECKLIST.md)\n- [../BEST_PRACTICES_MASTER_CHECKLIST.md](../BEST_PRACTICES_MASTER_CHECKLIST.md)\n- [incident-response.md](incident-response.md)\n"}