Retrying Failed Subgraph Fetches in Apollo Router

A retry is the cheapest way to turn a transient subgraph failure into a successful response, and the fastest way to turn a degraded subgraph into an unavailable one. The difference is entirely in the bounds. This page covers what the router will and will not retry, how a retry budget works, and which failures should never be retried at all. It sits under Error Handling and Partial Responses in Subgraphs.

When to use this pattern

  • Use it for subgraphs behind a load balancer or an autoscaler, where a single failed connection is genuinely likely to succeed on a different instance.
  • Use it for read-heavy subgraphs with idempotent fetches, which is what an _entities fetch always is.
  • Skip it for mutations, and for any subgraph whose failures are load-related. Retrying a service that is failing because it is overloaded adds exactly the load that is causing the failure.

Prerequisites

What a retry can and cannot fix

Retries help with exactly one class of failure: independent, transient faults. A connection reset, a pod terminating mid-request, a brief network blip — these are uncorrelated with your traffic, and a second attempt lands somewhere else with a good chance of succeeding.

They actively harm two other classes. Load-correlated failures — timeouts under pressure, connection-pool exhaustion, a saturated database — are caused by the volume of requests, and a retry adds volume. Deterministic failures — a malformed query, a missing key, an authorization denial — will fail identically every time, so a retry is pure waste plus latency.

The distinction matters because the router cannot always tell them apart from the outside. A connection error is almost certainly transient; a 500 might be either; a timeout is usually load. That uncertainty is why the budget exists: rather than deciding correctly per request, you cap the total additional load retries can generate as a fraction of normal traffic.

Retries help one failure class and hurt two Transient independent faults are fixed by a second attempt. Load-correlated failures are made worse because the retry adds load. Deterministic failures are unaffected and cost latency. Only the first row benefits transient and independent connection reset, pod terminating mid-request, a brief network blip. A second attempt lands elsewhere and usually succeeds. load-correlated — retries make it worse timeouts under pressure, pool exhaustion, a saturated database. The retry is additional load on the thing that is already failing from load. deterministic — retries are pure cost malformed selection, missing key, authorization denial. The second attempt fails identically, and the client waits twice as long to hear it.

Implementation walkthrough

The configuration is per subgraph, which is important: retry policy should follow the failure characteristics of each service rather than being applied uniformly.

# router.yaml
traffic_shaping:
  subgraphs:
    catalog:
      # Retries only make sense inside a timeout short enough to leave room
      # for a second attempt. 2s timeout + 1 retry fits a 5s client budget.
      timeout: 2s
      experimental_retry:
        # Floor: always allow this many retries per second, so a low-traffic
        # subgraph is not denied its first retry by a percentage calculation.
        min_per_sec: 10
        # Budget: retries may add at most 10% on top of normal request volume.
        # This is the line that stops a retry storm.
        retry_percent: 0.1
        retry_on_http_errors: true

    pricing:
      timeout: 3s
      experimental_retry:
        min_per_sec: 5
        retry_percent: 0.05    # this service is fragile; keep the ceiling low

    ledger:
      timeout: 5s
      # No retry block at all. This subgraph serves mutations, and a retried
      # write is a correctness problem rather than a resilience feature.

The budget is the mechanism worth understanding rather than copying. retry_percent: 0.1 means the router will not let retries exceed ten percent of the normal request rate to that subgraph. Under healthy conditions almost nothing retries and the budget is irrelevant. When a subgraph starts failing broadly, the budget caps the additional load at ten percent instead of doubling it, which is the difference between a degraded service recovering and a degraded service being held down.

min_per_sec exists because a percentage of a small number is zero. Without a floor, a subgraph receiving two requests per second would never get to retry at all, which is precisely the case where a single transient failure is most visible.

Verification steps

Confirm the retry happens, then confirm it is bounded — the second matters more.

# Fail a subgraph exactly once and confirm the operation still succeeds.
# The router should absorb it with no client-visible error.
curl -s localhost:4000/graphql -H 'content-type: application/json' \
  -d '{"query":"{ product(id:\"P-1\"){ name } }"}' | jq '.errors'
# expect: null, with the subgraph's request count showing 2

Then verify the budget engages under sustained failure, which is the test that actually protects you. Fail the subgraph continuously and watch its inbound request rate: it should rise by roughly the configured percentage, not double. If it doubles, the budget is not configured or the retries are happening somewhere else — usually in a client.

# Watch the subgraph's own request rate while it is failing.
# healthy: 1000/s  →  failing with a 10% budget: ~1100/s, never 2000/s

Finally, verify that no other layer is retrying. A client retrying on top of router retries produces a multiplicative effect that no single budget bounds, and it is easy to miss because each layer looks reasonable on its own.

What a budget prevents With unbounded retries, load on a failing subgraph doubles at the moment it is least able to cope. With a ten percent budget, load rises slightly and the service is able to recover. Requests reaching a failing subgraph unbounded retries — load doubles 10% budget — load barely moves subgraph starts failing t0 The pink line is why a struggling service often never recovers: the retries hold it down long after the original cause has passed, and the pattern is self-sustaining.

There is a useful ordering to remember when a subgraph starts failing intermittently and someone proposes turning retries up. Retries are the last of four things to reach for, not the first. Shorten the timeout so a failure is cheap. Check whether the failures are load-correlated, in which case capacity or rate limiting is the answer. Make the field nullable so a failure degrades rather than propagates. Only then consider whether a bounded retry converts enough failures into successes to justify the extra load. Teams that start at step four almost always end up with a service that fails more often, more expensively, and for longer.

Common mistakes & gotchas

Retrying inside a generous timeout. A five-second timeout with one retry means a failing fetch can consume ten seconds before the client hears anything. Shorten the timeout first; the retry has to fit inside the client’s patience, not extend it.

Retrying mutations. An _entities fetch is idempotent; a mutation generally is not. Configure retries per subgraph and leave them off wherever writes happen, unless every mutation carries an idempotency key.

Stacking retries across layers. Client, service mesh and router each retrying twice is eight attempts, and no single budget sees the total. Pick one layer — the router is usually the right one, because it knows which subgraph failed — and disable the others.

Retries can be configured at four layers and should be enabled at one. The table makes the reason concrete: only the router knows which subgraph failed while still holding the successful results of every other fetch, so a retry there costs exactly one request rather than repeating the whole operation.

Pick one layer, and make it the router A client retry repeats every subgraph. A mesh retry lacks context. A router retry re-issues exactly the failed fetch. A subgraph may still retry its own upstreams. Where a retry can be configured layer sees should it retry? client the whole operation no — repeats every subgraph service mesh one HTTP call only if the router does not router which subgraph failed yes — the right layer subgraph's own client its upstream yes, for its own dependencies Enabling two of these multiplies rather than adds, and no single budget bounds the product. The bottom row is independent — a subgraph retrying its database is a separate decision.

Frequently Asked Questions

Does a retry preserve the query plan?

Yes. The retry re-issues the same fetch to the same subgraph; nothing is replanned and no other fetch is affected. Sibling fetches that already succeeded keep their results, which is what makes a router-level retry so much cheaper than a client-level one — the client would repeat the entire operation, including every subgraph that was fine.

Should retries be enabled by default on every subgraph?

No, and a uniform policy is a sign nobody has looked at the failure characteristics. Read-heavy subgraphs behind an autoscaler benefit most. Subgraphs that fail under load benefit least and are harmed most. Subgraphs that serve writes should generally have retries off entirely.

How does this interact with rate limiting?

Directly, and in a way worth checking. A retry is a request and counts against a per-subgraph rate limit, so a retry storm can exhaust the limit and start rejecting first attempts. Set the budget well below the headroom in your rate limit, or the two controls will fight during the exact incident they were both configured for.

What should be measured?

Three numbers: retry rate as a fraction of requests, the success rate of retries, and the fraction of operations saved by a retry. If retries rarely succeed, the failures are deterministic or load-related and the policy is costing latency for nothing. If they usually succeed, the policy is earning its keep and the budget could be loosened slightly.

Does the router retry on a timeout?

It can, and it usually should not. A timeout most often signals that the subgraph is slow rather than momentarily unreachable, so retrying adds a request to a service already failing to keep up. Prefer retrying on connection errors and specific status codes, and treat a timeout as a signal to degrade rather than to try again.