Queues, Backpressure, and Idempotency: Designing for Failure

A queue is often introduced as a performance optimization: accept a request quickly, process it later, and smooth temporary bursts. That description is incomplete. A queue is a stored promise to do work. If the producer can create promises faster than consumers can fulfill them, the queue does not remove overload—it delays the moment when overload becomes visible.

Reliable asynchronous design therefore needs four properties together:

1. A bounded amount of admitted work.
2. Backpressure when capacity is exhausted.
3. Idempotent effects when delivery is repeated.
4. An explicit terminal path for work that cannot succeed.

A concrete order pipeline#

Consider a hypothetical API that accepts an order and asynchronously reserves inventory, charges payment, and schedules fulfillment.

sequenceDiagram
    participant C as Client
    participant A as Order API
    participant D as Orders DB
    participant Q as Durable Queue
    participant W as Worker
    participant P as Payment Provider

    C->>A: POST /orders + Idempotency-Key
    A->>D: Insert order and outbox record
    A-->>C: 202 Accepted + orderId
    D->>Q: Publish outbox event
    Q->>W: Deliver order.created
    W->>P: Charge with stable operation key
    P-->>W: Result or timeout
    W->>D: Record terminal or retryable state

The API returning 202 Accepted means the request was durably accepted, not that every downstream effect completed. That contract must be visible to callers through an order-status resource or event stream.

Delivery guarantees are application guarantees#

Queues are described as at-most-once, at-least-once, or exactly-once. Those labels can create false confidence.

• At-most-once delivery may lose work but does not redeliver it.
• At-least-once delivery preserves work through retries but can deliver duplicates.
• Exactly-once usually applies to a constrained boundary or protocol. It does not automatically make external side effects exactly once.

If a worker charges a card and loses connectivity before acknowledging the queue message, the queue cannot know whether the charge happened. Redelivery is correct from the queue’s perspective. The application must make the charge safe to repeat.

Idempotency starts at the boundary#

An operation is idempotent when repeating the same logical request produces the same intended result. GET is naturally idempotent; “charge this customer $50” is not unless the operation has a stable identity.

Require callers to provide an idempotency key or derive one from a stable business operation identifier:

type CreateOrderRequest = {
    idempotencyKey: string;
    customerId: string;
    items: Array<{ sku: string; quantity: number }>;
};

The server stores the key with a fingerprint of the normalized request and the resulting order ID. On repetition:

• Same key and same fingerprint → return the original result.
• Same key and different fingerprint → reject as a conflict.
• New key → create a new operation.

The data constraint matters more than the application check:

CREATE UNIQUE INDEX orders_idempotency_key_unique
ON orders (customer_id, idempotency_key);

Two concurrent requests can both pass a read-before-write check. A unique constraint serializes the decision at the source of truth.

Propagate identity downstream#

Each effect should have its own stable operation key. For order ord_123, payment attempt payment:ord_123 should be reused across worker retries and sent to a provider that supports idempotency.

Do not generate a fresh UUID inside every retry. That identifies the attempt, not the logical operation, and defeats deduplication.

Store both:

• operation_id: stable across attempts.
• attempt_id: unique for tracing one execution.

The outbox closes a dangerous gap#

Writing an order to a database and publishing an event to a queue are two separate operations. If the process crashes between them, the order exists but no worker is notified. Publishing first has the opposite problem: a worker may observe an order that never commits.

The transactional outbox records the order and a pending event in the same database transaction:

BEGIN;

INSERT INTO orders (id, customer_id, state)
VALUES ('ord_123', 'cus_9', 'accepted');

INSERT INTO outbox (event_id, aggregate_id, event_type, payload)
VALUES ('evt_456', 'ord_123', 'order.created', '{...}');

COMMIT;

A relay publishes pending outbox records and marks them delivered. The relay may publish twice after a crash, so consumers still need idempotency. The outbox solves atomic acceptance; it does not create exactly-once side effects.

Backpressure is a product behavior#

Assume producers submit 500 orders per second while workers can sustainably process 400. The queue grows by 100 messages every second. After one hour:

100 messages/s × 3,600 s = 360,000 queued orders

If each message and index overhead consumes 4 KB, the queue adds roughly 1.4 GB per hour. Storage is not the first failure. Customer latency is. A newly accepted order now waits behind hundreds of thousands of older orders.

A bounded system defines limits on:

• Queue depth.
• Oldest-message age.
• In-flight work per consumer and dependency.
• Producer rate per tenant or priority.
• Total retry traffic.

When a limit is reached, choose an explicit response:

1. Reject new low-priority work with a retryable status.
2. Degrade optional behavior.
3. Route selected work to a slower overflow path with a stated service level.
4. Shed load that no longer has value, such as expired refresh jobs.

Accepting everything is not kindness. It replaces a fast, observable failure with an unbounded and increasingly expensive one.

Bound concurrency at the dependency#

Worker count should not be the only concurrency control. One order may call inventory, payment, email, and fulfillment systems with different quotas and latency distributions.

Use dependency-scoped semaphores or worker pools:

payment concurrency:     80
inventory concurrency:  200
email concurrency:      500

If payment slows down, its work should queue behind the payment boundary without consuming every worker and starving unrelated inventory operations. Bulkheads make partial failure stay partial.

Retry only when the failure can change#

Classify outcomes before defining retry policy:

Retries need a maximum attempt count, a maximum elapsed time, exponential backoff, and jitter. They also need a system-wide budget. If every layer retries independently, one user request can create a combinatorial retry storm.

The layer immediately above the failing dependency should own the retry. Higher layers should receive a terminal or explicitly non-retryable result once that budget is exhausted.

Delay retries instead of blocking workers#

A worker should not sleep while holding a queue lease or connection. Record the next eligible attempt and release the worker:

function nextDelayMs(attempt: number, random: () => number): number {
    const cap = 60_000;
    const exponential = Math.min(cap, 1_000 * 2 ** attempt);
    return Math.floor(random() * exponential);
}

This is full jitter: choose a random delay between zero and the capped exponential value. Exact algorithms vary, but the objective is the same—avoid synchronized retry waves.

Poison messages need a terminal state#

A message that can never succeed should not cycle forever. After validation or retry exhaustion, move it to a dead-letter queue or terminal failure store containing:

• Original payload or a secure reference.
• Operation and attempt identifiers.
• Failure classification and sanitized error details.
• Attempt count and timestamps.
• Source queue and consumer version.
• Replay or remediation status.

The DLQ is not a trash can. It is an operational workflow. Alert on new terminal failures, group recurring causes, provide a controlled replay path, and make replay pass through the same idempotency checks as normal delivery.

Invalid payloads should usually skip normal retries. A schema error will not heal after eight seconds. Mark it non-retryable and route it directly to terminal handling.

Model the state machine#

State should distinguish attempts from outcomes:

stateDiagram-v2
    [*] --> Accepted
    Accepted --> Processing
    Processing --> Succeeded
    Processing --> RetryScheduled: transient or ambiguous
    RetryScheduled --> Processing: delay elapsed
    Processing --> Failed: permanent failure
    RetryScheduled --> Failed: budget exhausted
    Failed --> Processing: controlled replay
    Succeeded --> [*]

Transitions should be conditional and atomic. For example, only one worker should move an order from accepted to processing, and a stale attempt should not overwrite a later success.

Observe age, not only depth#

Queue depth is necessary but incomplete. Ten thousand messages may be normal for a high-throughput system or catastrophic for a low-volume one.

Track:

• Oldest ready-message age.
• Accepted-to-terminal latency by percentile.
• Arrival and completion rates.
• Retry rate by dependency and error class.
• In-flight work and concurrency saturation.
• DLQ arrivals and replay outcomes.
• Idempotency conflicts and deduplication hits.

The most actionable alert often combines age and rate: the oldest-message age is increasing while completion rate remains below arrival rate.

Failure drills#

Test the behavior that happy-path load tests omit:

1. Pause one consumer group and verify admission control activates before the queue becomes unsafe.
2. Make a provider return 503 and confirm retries stay within budget.
3. Return a success but drop the response; verify redelivery does not repeat the effect.
4. Inject an invalid message and confirm it reaches terminal handling without wasteful retries.
5. Replay a DLQ item twice and confirm the business result remains singular.
6. Restore a dependency and measure how quickly the backlog drains without overwhelming it again.

Make recovery an explicit operating procedure#

Queue designs are incomplete until replay has a documented owner and procedure. A DLQ is evidence that automated processing stopped; it is not a resolution. Replay should preserve the original idempotency key, validate that the failure is now actionable, and use the same admission limits as normal traffic. Reinjecting thousands of messages directly into the hot queue can reproduce the outage or starve new work.

Classify terminal messages first. A schema-invalid message may require correction or deletion. A message blocked by a temporary dependency can usually be retried unchanged. A message whose effect is uncertain requires reconciliation against the authoritative system before another attempt. Delivery metadata alone cannot prove whether an external effect committed before a timeout.

Recovery also needs a traffic policy. Reserve most capacity for current orders and a bounded fraction for replay, or pause replay when queue age or dependency latency crosses a threshold. Either makes the trade-off visible: reducing historical backlog must not destroy service for current users.

Retain enough evidence to reconstruct each transition: message identifier, idempotency key, attempt count, handler version, timestamps, terminal classification, and identifiers of committed effects. Avoid secrets or unrestricted payloads. The goal is an audit trail for correctness, not a second ungoverned data store.

The final recovery condition should be measurable: the DLQ is classified, accepted messages reached a terminal state, duplicates were reconciled, and temporary controls were removed. That prevents “queue depth returned to zero” from masking silent loss or duplicated effects.

Design checklist#

• Define what 202 Accepted promises.
• Give every business operation a stable identity.
• Enforce uniqueness in durable storage.
• Make external effects idempotent or reconcilable.
• Use an outbox when database state and publication must agree.
• Bound queue depth, age, concurrency, and retry traffic.
• Classify permanent, transient, ambiguous, and overload failures.
• Retry at one layer with backoff, jitter, and a budget.
• Treat the DLQ as an operated recovery path.
• Alert on lag and time-to-terminal, not merely message count.

A reliable queue does not hide failure. It gives the system a controlled vocabulary for accepting, delaying, rejecting, retrying, and terminating work.

Sources and further reading#

• Google SRE: Handling Overload
• Google SRE: Addressing Cascading Failures
• AWS Builders' Library: Making retries safe with idempotent APIs
• AWS Builders' Library: Timeouts, retries, and backoff with jitter
• Microsoft Azure Architecture Center: Transactional Outbox