
A timeout is not a failure. It is an absence of information. The request may have committed, rolled back, or never left the load balancer, and the only thing your client knows for certain is that it stopped waiting.
Most retry policies quietly assume the first attempt did nothing. In practice, the most common production duplicate is a request that succeeded in 40 milliseconds and whose response got lost on the way back, usually because a connection was reaped, a proxy timed out at 30 seconds, or a mobile client changed networks mid-response. The write landed. The client retries. Now there are two.
Idempotency keys are the standard fix, and they are less about HTTP semantics than about storage.
Consider a payment capture or an order insert with a 3 second client timeout. Four outcomes are possible:
Cases 3 and 4 are the ones that cost money, and both are invisible from the client side. At-least-once delivery is not a choice you make at the queue layer. It is the only delivery guarantee available over an unreliable network, so the receiver has to make duplicates harmless.
A client sends a key that identifies one intent, not one attempt:
POST /v1/payouts
Idempotency-Key: 8f2c1a90-6b2e-4a71-9d3f-2f1c9d8b7a10
The server guarantees: for a given account and key, the endpoint executes the underlying side effect at most once, and every replay returns the same response the first attempt produced, including the same status code. Retries of a request that returned 500 should return the stored 500, not re-execute. That last part is what most homegrown implementations miss, and it is what makes the guarantee real.
Scope the key to the account or tenant, not globally. Global keys are a cross-tenant denial of service waiting for a typo, and they leak the existence of another tenant's request through the conflict path.
Three fields carry the whole feature: the key, a request fingerprint, and a response snapshot.
create table idempotency_keys (
account_id text not null,
key text not null,
fingerprint text not null, -- sha256(method + path + body)
state text not null, -- 'in_progress' | 'completed'
status_code int,
response_body jsonb,
response_headers jsonb,
created_at timestamptz not null default now(),
completed_at timestamptz,
expires_at timestamptz not null,
primary key (account_id, key)
);
The primary key on (account_id, key) is doing the enforcing. The insert is the lock. Everything else is bookkeeping.
The fingerprint earns its place for one case: a client that reuses a key with a different body. That is a bug in the caller, and the correct answer is 422 Unprocessable Entity with a message saying the key was already used for a different payload. Without the fingerprint, you would return an unrelated response and let a caller believe a change was applied when it was not.
Ordered by frequency:
First request. INSERT ... ON CONFLICT DO NOTHING succeeds. Write in_progress, execute the handler, persist the response, flip to completed, return.
Replay of a completed key. Row exists with state = 'completed'. Return the stored status, body and headers. No handler execution, no database write beyond a read. This path should be in single-digit milliseconds; if it is not, you are loading the row with a query that also scans history.
Collision with an in-flight request. The row exists with state = 'in_progress', so another attempt is holding the work. Two acceptable answers: return 409 Conflict with Retry-After: 1 and let the client poll, or block on the row for a bounded window and return the stored response when it lands. The blocking version is friendlier to mobile clients; the 409 version is friendlier to your connection pool. Pick one deliberately, because the choice changes how a retry storm behaves.
Fingerprint mismatch. 422, as above.
The awkward case is a handler that crashed after the row was written but before completion. The in_progress row becomes a tombstone and every retry gets 409 forever. Two mitigations: stamp the row with the worker's lease and allow a takeover after the lease expires, or run a reaper that deletes in_progress rows older than a few minutes so the next attempt starts clean. The reaper is simpler and is correct as long as your handlers are shorter than the reaper window. If a handler legitimately takes 90 seconds, set the window well above it, or you will re-execute exactly the long-running requests you most wanted to protect.
Storing response bodies is what makes the feature expensive. A few practical constraints:
Pick the window from your client's retry horizon, not from a security instinct. A client that retries for 15 minutes needs 15 minutes; a queue that retries with exponential backoff over 24 hours needs a day. AWS and Stripe both settle around 24 hours, which is long enough for an overnight incident and short enough that the table stays small with a scheduled delete of expired rows.
Below that horizon you are lying to callers. Above it you are paying to store replays nobody will ever request.
Idempotency keys protect one request boundary. They do not protect the side effects the handler fires outward. A handler that sends a confirmation email, publishes to a queue and charges a card is only idempotent at the API edge; the email still goes twice if the handler fails after sending it and the client retries with a new key. Fix that by making the downstream systems idempotent too: pass a derived key ({request_key}:email) as the message deduplication ID on the queue, and use the payment provider's own idempotency header rather than inventing one.
The second recurring bug is the client generating a fresh key per attempt. That produces a request that looks correct in logs and duplicates perfectly. Keys must be minted when the user intent is created, persisted alongside the draft, and reused across every retry, including retries that happen three days later from a background job.
Third: keys reused across endpoints. A key is only unique within the scope you defined. If your middleware does not include the path in the fingerprint, a key from POST /v1/orders will happily replay against POST /v1/refunds.
Four counters tell you whether the feature is working:
Idempotency costs a write and a read on the hot path, roughly 1 to 2 milliseconds on a local Postgres, plus a table you have to prune. In exchange, retries stop being dangerous, and once retries are safe you can retry aggressively: on timeout, on 5xx, on connection reset, with backoff through a region failover.
That is the part teams miss when they treat idempotency as a compliance checkbox. It is not only duplicate protection. It is the thing that lets your client be brave.
Zero-X platform primitives include request idempotency at the API boundary and deduplication IDs on internal queues. Retry without inventing your own ledger.
We use cookies to improve your experience on this site. By continuing, you agree to our Privacy Notice.