Alerting on Federated Error Rates and Slow Subgraphs

The standard alert for an HTTP service — error rate above a threshold — is close to useless for a federated graph, because a partial response is a 200 and a degraded field is invisible in status codes. Getting alerting right means measuring the things federation actually breaks: field errors by path, subgraph latency against its own budget, and the plan-shape changes that precede both. This page covers what to measure, what to page on, and what to leave on a dashboard. It sits under Observability and Distributed Tracing in Federation.

When to use this pattern

  • Use it as soon as more than one team owns a subgraph, because that is when “the graph is slow” stops having an owner.
  • Use it before a public launch, since external clients will not tell you about a degraded field — they will simply see less of your product.
  • Skip paging alerts on a graph nobody is on call for. An alert with no responder is a dashboard with extra steps, and it trains people to ignore the channel.

Prerequisites

Four signals worth alerting on

Field error rate, by path. The count of errors[] entries divided by operations, grouped by the path they name. This is the metric that actually tracks user-visible degradation, and it is invisible to any status-code-based alert.

Subgraph latency against its own budget. Not a global p95, but each subgraph’s p95 compared with the number that subgraph agreed to. A shared threshold flags the naturally slow service constantly and never flags the fast one that just doubled.

Operation-level p95, by operation name. The number a user experiences. It moves for reasons no single subgraph metric explains — a plan-shape change, a cache-hit-rate drop — and it is the signal that says “something is worse” when no component looks unhealthy.

Composition and launch failures. Not a performance signal at all, but the one that predicts an inability to ship. A failed launch is silent by design, because routers keep serving, and silence is exactly why it needs an alert.

What to page on, and what belongs on a dashboard Field error rate and operation p95 are user-visible and page. Subgraph latency against budget pages its owner. Composition failures notify rather than page, because nothing is broken for users yet. Four signals, three different responses signal detects response field error rate by path degradation users can see page the field's owner operation p95 by name what a user experiences page the graph owner subgraph p95 vs its budget which service regressed page that subgraph's team composition / launch failure inability to ship notify — nothing is broken yet

Implementation walkthrough

Getting the attribution right in the router is what makes every alert below possible, and it is a few lines of configuration.

# router.yaml
telemetry:
  instrumentation:
    spans:
      subgraph:
        attributes:
          # Without this, a slow subgraph is indistinguishable from a slow graph.
          subgraph.name: true
      supergraph:
        attributes:
          # Operation name is the only client-facing dimension you get, because
          # every operation hits the same URL.
          graphql.operation.name: true
    instruments:
      supergraph:
        # Count errors by the path they name, not by HTTP status: a partial
        # response is a 200 and a status-based alert will never fire.
        graphql.error:
          attributes:
            graphql.error.path: true
            subgraph.name: true

The alert rules then follow directly. Two properties matter in each: a ratio rather than a count, so traffic changes do not create false alarms; and a duration, so a single spike does not page anyone.

# Alert 1 — a field is failing for users.
- alert: FederatedFieldErrorRate
  expr: |
    sum by (graphql_error_path) (rate(graphql_error_total[5m]))
      / sum by (graphql_error_path) (rate(graphql_operation_total[5m])) > 0.01
  for: 10m       # sustained: a brief blip is not worth a page
  labels: { severity: page }

# Alert 2 — a subgraph is slow relative to its OWN budget.
- alert: SubgraphLatencyBudget
  expr: |
    histogram_quantile(0.95, sum by (le, subgraph_name)
      (rate(subgraph_request_duration_seconds_bucket[5m])))
      > on (subgraph_name) group_left subgraph_latency_budget_seconds
  for: 15m
  labels: { severity: page }

# Alert 3 — the graph got slower without any subgraph looking unhealthy.
- alert: OperationLatencyRegression
  expr: |
    histogram_quantile(0.95, sum by (le, graphql_operation_name)
      (rate(supergraph_request_duration_seconds_bucket[10m]))) > 1.5
  for: 15m
  labels: { severity: page }

The budget in the second rule is a recorded metric rather than a constant, which is what lets each team set its own number and change it without editing the alerting rules.

Verification steps

Test alerts by causing the condition, not by reading the expression.

# 1. Force one field to fail and confirm the error-rate alert fires with the
#    right path label — the label is what routes the page to an owner.
# 2. Add latency to one subgraph and confirm only that subgraph's alert fires.
# 3. Break composition deliberately and confirm a notification, not a page.

The second test is the one that distinguishes useful alerting from noise. If adding two hundred milliseconds to one subgraph fires three alerts across two teams, your thresholds are coupled and every incident will involve more people than it needs.

Then check the reverse: cause a degradation that should be invisible — a field nulling out under a nullable parent, with no user impact — and confirm nothing pages. An alerting system that fires on graceful degradation punishes exactly the resilience work you want teams to do.

Finally, review the fired-alert log monthly. Any alert that has fired more than a handful of times without a corresponding action is either mistuned or measuring something nobody acts on, and both are worth fixing before the channel loses its meaning.

A threshold without a duration pages on noise A short spike above the threshold recovers within the window and raises nothing. A sustained regression stays above it for longer than the configured duration and pages. Operation p95 against a 1.5s threshold above threshold one spike recovers inside the window — no page sustained 15 minutes — page 0 30 min 60 min Set the duration to the shortest breach you would actually act on. Anything shorter trains the team to dismiss the alert, which is worse than not having it.

One structural point underlies all of the above. In a single service, alerting and ownership are the same question — the service is slow, the service’s team responds. In a federated graph they come apart: the signal a user experiences belongs to the graph, and the causes belong to individual subgraphs owned by different teams. An alerting design that ignores that split either pages everybody for everything or pages the graph owner for problems they cannot fix.

The arrangement that works is two tiers. Subgraph-level alerts, owned by subgraph teams, fire on their own budget and their own error rate, and are actionable by definition. Operation-level alerts, owned by whoever owns the graph, fire on what users experience and exist to catch the regressions no component-level signal explains — a plan gaining a level, a cache-hit rate collapsing, a slow path becoming common. That second tier is small, and it is the difference between noticing a graph-wide regression and waiting for a support ticket.

Write down which team receives which alert before configuring any of them. The routing is the part that decides whether the alerting is useful, and it is the part most often left until after the first page arrives at the wrong person.

A useful sanity check on any alerting setup is to ask which of the common federated failures it would actually catch. Four scenarios cover most of what goes wrong.

Four scenarios, one blind spot Field errors, subgraph latency, plan changes and composition failures are all invisible to status-code alerting. Would your alerts catch these? scenario status-code alert the alerts on this page one field errors on every request no — still a 200 yes, field error rate one subgraph doubles its latency no yes, budget alert a plan gains a sequential level no yes, operation p95 a composition failure blocks shipping no yes, launch notification The middle column is the whole argument: a federated graph reports success while degrading. If your current alerting is entirely in that column, none of these four would page anyone.

Common mistakes & gotchas

Alerting on HTTP status codes. A federated graph returns 200 with a partial response, so status-based error rates report perfect health while a field fails on every request. Count errors[] entries by path instead.

One latency threshold for every subgraph. It flags the naturally slow service permanently and misses the fast one that quietly doubled. Give each subgraph its own budget and alert on the ratio to it.

Paging on connection count or request rate. Both move for entirely legitimate reasons — a launch, a working day starting — and both produce alerts nobody can act on. Keep them on a dashboard.

Frequently Asked Questions

How do I set a subgraph’s latency budget?

Work backwards from the client-facing target. If an operation must complete in a second and its plan has two levels, each level has roughly four hundred milliseconds after overhead, and a subgraph appearing in the deeper level inherits that. Publishing the arithmetic matters as much as the number, because a team asked to hit two hundred milliseconds without knowing why will negotiate rather than optimise.

Should every subgraph team own its own alerts?

Yes for their own latency and error rate, no for operation-level alerts. The graph as a whole needs an owner, because operation p95 can regress with every subgraph inside budget — a plan-shape change or a cache-hit-rate drop belongs to nobody in particular, which is exactly why it needs somebody in particular.

What about alerting on query plan changes?

Worth tracking, rarely worth paging. A plan gaining a fetch is a strong leading indicator of a latency regression, but it is not itself user-visible, and paging on it converts a review conversation into an incident. Report it in the pull request that caused it instead — see query plan optimization strategies.

How do I alert on something that stopped happening?

With a floor rather than a ceiling. A subscription topic that normally delivers events, an operation that normally runs, a cache that normally hits — each of these fails silently, and only a lower-bound alert notices. Silence looks identical to health on every dashboard, which is why these are the alerts teams add after their first quiet outage.