Technology

Idempotency Keys: The One Header That Makes Retries Safe

Zero-X·2026-11-05
Idempotency Keys: The One Header That Makes Retries Safe

A timeout is not a failure. It is an absence of information. The request may have committed, rolled back, or never left the load balancer, and the only thing your client knows for certain is that it stopped waiting.

Most retry policies quietly assume the first attempt did nothing. In practice, the most common production duplicate is a request that succeeded in 40 milliseconds and whose response got lost on the way back, usually because a connection was reaped, a proxy timed out at 30 seconds, or a mobile client changed networks mid-response. The write landed. The client retries. Now there are two.

Idempotency keys are the standard fix, and they are less about HTTP semantics than about storage.

What a timeout actually tells you

Consider a payment capture or an order insert with a 3 second client timeout. Four outcomes are possible:

  1. The request never reached the server. Retry is safe.
  2. The server rejected it. Retry is safe.
  3. The server committed and responded in time, but the response was lost. Retry duplicates.
  4. The server is still working when the client gives up. Retry races the first attempt.

Cases 3 and 4 are the ones that cost money, and both are invisible from the client side. At-least-once delivery is not a choice you make at the queue layer. It is the only delivery guarantee available over an unreliable network, so the receiver has to make duplicates harmless.

The contract

A client sends a key that identifies one intent, not one attempt:

POST /v1/payouts
Idempotency-Key: 8f2c1a90-6b2e-4a71-9d3f-2f1c9d8b7a10

The server guarantees: for a given account and key, the endpoint executes the underlying side effect at most once, and every replay returns the same response the first attempt produced, including the same status code. Retries of a request that returned 500 should return the stored 500, not re-execute. That last part is what most homegrown implementations miss, and it is what makes the guarantee real.

Scope the key to the account or tenant, not globally. Global keys are a cross-tenant denial of service waiting for a typo, and they leak the existence of another tenant's request through the conflict path.

The storage record

Three fields carry the whole feature: the key, a request fingerprint, and a response snapshot.

create table idempotency_keys (
  account_id      text        not null,
  key             text        not null,
  fingerprint     text        not null,   -- sha256(method + path + body)
  state           text        not null,   -- 'in_progress' | 'completed'
  status_code     int,
  response_body   jsonb,
  response_headers jsonb,
  created_at      timestamptz not null default now(),
  completed_at    timestamptz,
  expires_at      timestamptz not null,
  primary key (account_id, key)
);

The primary key on (account_id, key) is doing the enforcing. The insert is the lock. Everything else is bookkeeping.

The fingerprint earns its place for one case: a client that reuses a key with a different body. That is a bug in the caller, and the correct answer is 422 Unprocessable Entity with a message saying the key was already used for a different payload. Without the fingerprint, you would return an unrelated response and let a caller believe a change was applied when it was not.

The four paths through the middleware

Ordered by frequency:

First request. INSERT ... ON CONFLICT DO NOTHING succeeds. Write in_progress, execute the handler, persist the response, flip to completed, return.

Replay of a completed key. Row exists with state = 'completed'. Return the stored status, body and headers. No handler execution, no database write beyond a read. This path should be in single-digit milliseconds; if it is not, you are loading the row with a query that also scans history.

Collision with an in-flight request. The row exists with state = 'in_progress', so another attempt is holding the work. Two acceptable answers: return 409 Conflict with Retry-After: 1 and let the client poll, or block on the row for a bounded window and return the stored response when it lands. The blocking version is friendlier to mobile clients; the 409 version is friendlier to your connection pool. Pick one deliberately, because the choice changes how a retry storm behaves.

Fingerprint mismatch. 422, as above.

The awkward case is a handler that crashed after the row was written but before completion. The in_progress row becomes a tombstone and every retry gets 409 forever. Two mitigations: stamp the row with the worker's lease and allow a takeover after the lease expires, or run a reaper that deletes in_progress rows older than a few minutes so the next attempt starts clean. The reaper is simpler and is correct as long as your handlers are shorter than the reaper window. If a handler legitimately takes 90 seconds, set the window well above it, or you will re-execute exactly the long-running requests you most wanted to protect.

Where the response snapshot lives

Storing response bodies is what makes the feature expensive. A few practical constraints:

  • Cap the snapshot size, 64 KB is generous for typical API payloads, and treat an oversized response as an error at write time rather than silently truncating it on replay.
  • Never store secrets that were single-use. If the handler returns a token or a signed URL, the replay is handing out a stale credential. Either regenerate those fields on replay with a marker header, or keep them out of the body entirely.
  • Compress. JSON response bodies compress 5 to 10 times, and this table grows at request rate.

Expiry is a storage decision

Pick the window from your client's retry horizon, not from a security instinct. A client that retries for 15 minutes needs 15 minutes; a queue that retries with exponential backoff over 24 hours needs a day. AWS and Stripe both settle around 24 hours, which is long enough for an overnight incident and short enough that the table stays small with a scheduled delete of expired rows.

Below that horizon you are lying to callers. Above it you are paying to store replays nobody will ever request.

The failures that survive a correct implementation

Idempotency keys protect one request boundary. They do not protect the side effects the handler fires outward. A handler that sends a confirmation email, publishes to a queue and charges a card is only idempotent at the API edge; the email still goes twice if the handler fails after sending it and the client retries with a new key. Fix that by making the downstream systems idempotent too: pass a derived key ({request_key}:email) as the message deduplication ID on the queue, and use the payment provider's own idempotency header rather than inventing one.

The second recurring bug is the client generating a fresh key per attempt. That produces a request that looks correct in logs and duplicates perfectly. Keys must be minted when the user intent is created, persisted alongside the draft, and reused across every retry, including retries that happen three days later from a background job.

Third: keys reused across endpoints. A key is only unique within the scope you defined. If your middleware does not include the path in the fingerprint, a key from POST /v1/orders will happily replay against POST /v1/refunds.

What to measure

Four counters tell you whether the feature is working:

  • Replay rate: replays divided by total requests. A healthy API sees well under 1 percent. A spike means client timeouts are misconfigured, not that idempotency is broken.
  • Mismatch rate: fingerprint conflicts. Should be near zero. Anything above that is a caller bug worth reporting back to the team that owns the client.
  • In-flight collisions: how often two attempts overlap. This is your real retry storm indicator, and it moves before p99 latency does.
  • Storage growth: rows and bytes per day, against your TTL.

The trade you are making

Idempotency costs a write and a read on the hot path, roughly 1 to 2 milliseconds on a local Postgres, plus a table you have to prune. In exchange, retries stop being dangerous, and once retries are safe you can retry aggressively: on timeout, on 5xx, on connection reset, with backoff through a region failover.

That is the part teams miss when they treat idempotency as a compliance checkbox. It is not only duplicate protection. It is the thing that lets your client be brave.


Zero-X platform primitives include request idempotency at the API boundary and deduplication IDs on internal queues. Retry without inventing your own ledger.

We use cookies to improve your experience on this site. By continuing, you agree to our Privacy Notice.