Unclassified

15 — Events, Queues, Reliability, and Observability

Status: Canonical requirements; individual components require runtime proof

Event envelope

Every platform event includes:

stable event ID and version;
event type and occurred / received time;
tenant, actor, and source;
subject entity and relationship references;
correlation, causation, and trace IDs;
schema and payload classification;
provenance and evidence links;
idempotency and deduplication keys;
retry and retention policy.

Delivery

Asynchronous work uses versioned topics or queues with:

explicit producer and consumer ownership;
at-least-once-safe handlers;
idempotent writes;
bounded retries and exponential backoff;
dead-letter capture;
replay and reprocessing;
poison-message quarantine;
cursor or offset visibility;
reconciliation after ambiguous delivery.

Observability

Every production service provides:

structured logs;
metrics and service-level indicators;
distributed traces and correlation;
health, readiness, and version endpoints;
queue depth and age;
error, retry, stale-binding, and reconciliation rates;
tenant-safe diagnostics;
alerts with owner and runbook.

SLO domains

identity and tenant resolution;
Gateway and public MCP availability;
connector authentication and sync freshness;
Pipeline promotion latency and failure;
projection freshness;
capability resolution and governed execution;
queue delay and dead-letter volume;
reconciliation completion;
Customer Zero journey success.

Idempotency law

Retrying a load, webhook, proposal, approval, execution, promotion, or
reconciliation must not create duplicate canonical consequences.

Evidence

Deployment or code presence does not satisfy reliability. Each critical path
needs fault injection or negative evidence for timeout, duplicate delivery,
provider outage, partial execution, callback loss, queue retry, and recovery.

Repository anchors

services/webhook-ingress/
packages/webhooks/
services/connector-sync/
services/pipeline/
services/telemetry/
packages/lib/src/logging.ts