Subscribe

LAB_

Reproducible tests. Explicit limits.

Backend experiments with the code, environment, parameters, and results needed to review what happened and repeat the test.

02active experiments
02published runs
✓reproducible

What happens when the PostgreSQL LISTEN/NOTIFY queue fills up?

Two connections, an open transaction and a 512 KiB queue: reproducing notification saturation and recovery in PostgreSQL 17.11.

HYPOTHESIS

A listener with an open transaction can prevent queue cleanup; ending that transaction should allow delivery to recover.

SUCCESSFUL SENDS64
FAILED SENDS36
PEAK OCCUPANCY100%
AFTER COMMIT0%

SETUP AND PROCEDURE_

Docker Compose · PostgreSQL 17.11 · 64 pages of 8 KiB · 100 attempts with payloads of 7,000 characters plus the send number. LISTEN committed before BEGIN; sender in autocommit.

OBSERVED SIGNALS_

  • idle in transaction
  • too many notifications in the NOTIFY queue
  • pg_notification_queue_usage(): 0 → 1 → 0
CONCLUSION_ Occupancy reached 100%. After the listener committed, it dropped to zero and a new message was successfully sent and received without restarting PostgreSQL.

Limit: one local run with one database and a reduced queue. The 64 successful sends depend on payload size; latency, disk performance and an 8 GiB queue were not measured.

Read procedure and results →PostgreSQL connection pools →

Can a webhook engine drain its queue after losing a worker and pausing Redis?

A run of 10,000 events and 20,000 deliveries subjected the durable flow to deliberate failures while measuring final states, duplicates, signatures, and backlog growth.

HYPOTHESIS

If PostgreSQL holds authoritative state and the outbox can reconcile work, killing a worker and pausing Redis should leave no pending deliveries and should preserve the security invariants.

EVENTS10,000
DELIVERIES20,000
DELIVERED19,600
FAILED200
DEAD-LETTER200
PENDING0
Measured queue growth and drain during 20,000 deliveries
The observed backlog peaked at 12,420 deliveries and drained to zero. The dashed line marks the end of ingestion; the shaded interval marks the Redis pause.

INJECTED FAILURES_

  • One worker process was killed while holding a claim.
  • The isolated Redis instance was paused for three seconds.
  • The scenario included permanent and transient HTTP responses.

INVARIANTS_

  • 0 duplicate Event/Endpoint pairs
  • 0 invalid HMAC signatures
  • 0 pending deliveries at completion
CONCLUSION_ The run supported the hypothesis in this environment: the interrupted claim recovered, the queue drained again, and every final state matched the scenario.

Limit: this is a controlled local run, not an SLA or a production throughput estimate. Latencies include queue time and retries.

Read case study →Read full report ↗View repository →

NOTES_001

How these analyses are built

01

Hypothesis first

Every test starts with a question that can be proven wrong.

02

Explicit scope

Each piece states whether it is a guide, source analysis or measured experiment.

03

Verifiable evidence

Experiments will be published with reviewable environments, parameters and results.