Webhook Delivery: reliable dispatch with retries and recovery

Webhook Delivery is an engine that accepts events, creates an independent delivery for every subscribed destination, and processes the resulting HTTP requests outside the original request path. It addresses a common integration problem: a receiving system can be slow, temporarily unavailable, or return an error that needs operator intervention.

The goal was not merely to run HTTP requests in the background. Delivery work had to survive restarts, retain every attempt, and remain available for investigation or recovery without losing its original context.

Separating acceptance from delivery#

The API accepts an event, finds active endpoints, and persists the event, its deliveries, and an outbox entry in a single PostgreSQL transaction. It then returns 202 Accepted. This confirms durable acceptance rather than successful receipt by every destination.

A publisher forwards durable work to Redis Streams. Workers claim deliveries and send signed HTTP requests. If Redis becomes unavailable, accepted events remain in PostgreSQL and the backlog can be forwarded after recovery.

This boundary keeps receiver latency away from the publishing application. It also allows one destination to fail without blocking the other participants in the fan-out.

An explicit failure policy#

Every result is classified before the next state transition:

• 2xx responses complete a delivery.
• Timeouts, network failures, 408, 425, 429, and 5xx schedule another attempt.
• Permanent failures stop automatic processing.
• Exhausted retries move the delivery to dead_letter.

Retries use exponential backoff with jitter and are scheduled in the database, so workers do not sleep until work becomes due. A manual recovery opens a new retry budget without deleting earlier attempts. Operators can inspect both the original failure and the later outcome.

Security and consistency#

Each endpoint receives a secret when it is created. Workers sign the exact outgoing bytes with HMAC-SHA256 and attach stable identifiers that let receivers authenticate and deduplicate repeated deliveries.

The system provides an at-least-once guarantee. Avoiding silent loss means a receiver must treat the webhook identifier as an idempotency key. HMAC authenticates the body but does not encrypt it.

Production destinations require HTTPS, an explicit hostname allowlist, and DNS validation that blocks private and reserved addresses. Redirects and environment proxies are disabled. The documentation also records a remaining limitation: the connected IP is not pinned after DNS prevalidation, so a real deployment needs additional outbound controls against DNS rebinding.

Observability and recovery#

The dashboard filters deliveries by state and event type, exposes individual attempts, and supports manual recovery. The API reports volume, state counts, retry activity, success rate, and latency percentiles. Correlation identifiers connect the initial request with events and worker logs.

Claims use tokens and leases so an interrupted worker cannot overwrite a newer result. The outbox and reconciliation process recover work after Redis failures or lost stream messages. PostgreSQL, rather than the stream, remains authoritative for delivery state.

Load evidence#

The repository includes an isolated runner that creates dedicated resources, injects failures, and cleans up afterward. One documented local run processed 10,000 events and 20,000 deliveries, finishing with 19,600 successful deliveries and 400 intentionally terminal failures. The observed backlog peaked at 12,420 deliveries and drained to zero.

The run, its parameters, queue chart, and checked invariants are published as a reproducible Webhook Delivery Lab experiment. The original code and artifacts remain available in the project repository.

Those figures describe a controlled local test, not a production SLA. Reported latency includes queue time and retries, and the scenario deliberately includes both permanent and transient receiver failures.

Result#

The project turns a fragile HTTP operation into a durable, inspectable workflow. FastAPI provides the publishing and management surface; PostgreSQL holds transactional truth; Redis Streams distributes work; and workers enforce delivery, security, and recovery policies.

The result demonstrates that reliable webhooks require more than retrying a request. A robust design must define what is persisted, who owns each state transition, how interrupted work is recovered, and which evidence remains when something goes wrong.