Unclassified
Sync Failure And DLQ Redrive Runbook
Use this runbook for connector sync failures, pipeline processing failures, queue backlog spikes, or DLQ redrive operations.
Symptoms
- tenant sync is stuck or never completes
- queue backlog is growing
- retries are climbing rapidly
dead_letter_queueor ops DLQ volume spikes- normalized entities are missing after a sync or webhook burst
Determine The Failing Stage
Check which path is failing:
| Area | What to inspect |
|---|---|
| Webhook ingress | signature verification, dedupe, provider payload shape |
| Connector sync | cron trigger, job creation, provider API errors, cursor state |
| Pipeline / Normalizer | validation errors, mapping errors, idempotency collisions |
| Intelligence DLQ | downstream analysis failures or governance path failures |
Containment
Before redriving anything:
- Pause the noisy source if failures are still being produced.
- Confirm whether the issue is transient or deterministic.
- Capture sample failing payloads, tenant IDs, providers, and correlation IDs.
Diagnose The Root Cause
- transient infra issue: provider outage, network issue, secret drift
- deterministic schema issue: payload shape, mapping bug, validation mismatch
- write-path issue: DB constraint, missing tenant config, stale contract
Fix the root cause before replaying.
Redrive Rules
- redrive one tenant or provider at a time
- preserve correlation and audit context where possible
- rely on idempotent writes rather than manual dedupe hacks
- never bulk-replay the entire DLQ blindly
Suggested Redrive Flow
- Query the failing records or DLQ entries.
- Narrow to the smallest safe replay batch.
- Re-enqueue to the correct queue only after the fix is live.
- Monitor:
- retry count
- DLQ depth
- resulting Spine writes
- duplicate side effects
Verification
- connector status returns healthy
- backlog drains instead of growing
- expected entities appear in Spine or downstream read surfaces
- no new DLQ spike from the same root cause