
A user taps checkout. The request crosses an edge proxy, an API gateway, an order service, a pricing service, an inventory service and a payment provider. Six hops, five of them internal. Every one of those services has a timeout configured by a different team, in a different quarter, for a different reason.
The p50 is 180ms. The p99 is 4.2 seconds. Nobody chose 4.2 seconds. It is simply the sum of the timeouts.
The usual pattern: a service owner measures their own latency, sees p99 at 800ms, sets the client timeout to 1.5 seconds for "headroom," and moves on. That is a defensible decision made in isolation. Applied six times in a chain, it produces a caller that waits 9 seconds for a request that could only ever succeed in 200ms.
Two compounding effects make it worse than simple addition.
Serial addition. A request that hits the slow tail at every hop accumulates. The probability is low, but you have millions of requests, so the low-probability corner is a permanent feature of your p99 and a daily feature of your p99.9.
Retry multiplication. If each hop retries twice on timeout, the leaf service can see up to 2^5 requests for one user action. The retry storm arrives exactly when the system is already slow, which is the only time it matters.
The deeper problem is not the numbers. It is that the number is a local fact applied to a global question. A timeout is a statement about how long the caller is willing to wait, and only the original caller knows that.
The fix is to pass the caller's remaining patience down the stack as a value that every hop reads and decrements. gRPC has carried this as grpc-timeout for years; HTTP has no equivalent primitive, so you build one.
X-Deadline-Ms: 900
Two rules make it work:
The middleware is small:
func withDeadline(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var budget time.Duration
if h := r.Header.Get("X-Deadline-Ms"); h != "" {
ms, err := strconv.Atoi(h)
if err != nil || ms <= 0 {
http.Error(w, "bad deadline", http.StatusBadRequest)
return
}
budget = time.Duration(ms) * time.Millisecond
} else {
budget = defaultBudget // 2000ms at the edge
}
// keep 15% (max 40ms) for our own serialization and response write
reserve := budget / 7
if reserve > 40*time.Millisecond {
reserve = 40 * time.Millisecond
}
child := budget - reserve
if child <= 0 {
http.Error(w, "deadline exceeded", http.StatusGatewayTimeout)
return
}
ctx, cancel := context.WithTimeout(r.Context(), child)
defer cancel()
r = r.WithContext(ctx)
r.Header.Set("X-Deadline-Ms", strconv.Itoa(int(child.Milliseconds())))
next.ServeHTTP(w, r)
})
}
The important line is the child <= 0 check. A hop that receives an expired budget should fail immediately and loudly rather than start work it will abandon. Failing fast at the fourth hop is worth more than succeeding slowly at the sixth, because the caller has already given up.
Linear chains are the easy case. Real services fan out: one order request calls pricing, inventory and fraud in parallel, then waits for all three. If each child gets the full remaining budget, you have not saved anything, you have just moved the deadline around.
Allocate the child budget from the parent's remaining time, and make the parallel calls share it:
child := childBudget(ctx) // parent remaining minus reserve
g, gctx := errgroup.WithContext(ctx)
g.SetLimit(3)
for _, svc := range []string{"pricing", "inventory", "fraud"} {
svc := svc
g.Go(func() error { return call(gctx, svc, child) })
}
err := g.Wait()
Two properties fall out of this and both matter:
gctx cancellation propagates to siblings when the first hard failure occurs, so you stop paying for work whose result you cannot use.Per-call retry counts multiply across hops. Retry budgets do not: track the fraction of recent requests that were retries, and stop retrying when that fraction crosses a threshold.
A workable default is a token bucket where retries draw tokens and successes replenish them, with a ceiling of about 10 percent of traffic. Under normal load, retries are free. During an incident, retries are capped and the system sheds load instead of amplifying it. Pair this with the deadline: a retry is only worth attempting if the remaining budget exceeds the cost of a fresh attempt by a healthy margin, typically two to three times the observed p50 for that call.
if remaining < 3 * p50 && retryBudget.Exhausted() {
return lastErr // do not retry, fail the request fast
}
The counterintuitive part: aggressive retries make p99 worse even when they make p50 better. If your retry policy is not deadline-aware, it is a latency amplifier wearing a reliability costume.
The hardest part of a timeout budget is that queueing time is invisible to the caller. A request can spend 700 of its 900 milliseconds waiting in a server's accept queue, a connection pool, or a thread pool, and the application code that receives it sees a fresh context with a nearly expired budget and no explanation.
Three things help:
Four signals tell you whether the budget is real:
Deadline propagation costs a header, a middleware, and the discipline to stop thinking of timeouts as per-service settings. In exchange, failure becomes predictable: a request either succeeds within the window the caller specified, or fails fast enough that the caller can do something else.
The second-order benefit is the one teams underestimate. Once every hop knows how much time is left, "slow" stops being a mystery and becomes a routing decision. Degrade the optional dependency, serve the cached price, return a partial result. That is not a fallback pattern you bolt on later. It is what a shared clock makes possible.
Zero-X platform primitives include deadline propagation across internal service calls, per-call retry budgets, and admission-time rejection for requests whose budget is already spent.
We use cookies to improve your experience on this site. By continuing, you agree to our Privacy Notice.