Scaling Federated Subscriptions Across Router Instances

A subscription setup that works perfectly on one router replica and one subgraph replica can fail the moment either is scaled to two, and it fails intermittently, which makes it one of the harder federation problems to diagnose. This page covers what breaks when you add replicas, what has to be shared, and how to size the result — part of Subscriptions in Federated GraphQL.

When to use this pattern

  • Use it before the second replica of anything, because the failure it prevents is intermittent and load-dependent, which is the worst kind to debug in production.
  • Use it when subscription counts approach the connection limit of a single router process, so horizontal scaling is the next step.
  • Skip it only for genuinely single-instance deployments — a development environment or an internal tool with a handful of users.

Prerequisites

What actually breaks when you add a replica

There are exactly two independent failure modes, and conflating them wastes days.

The subgraph fan-out problem. A publisher and a subscriber must meet. If the subgraph publishes events through an in-process emitter, an event produced on replica A never reaches a subscriber connected to replica B. With two replicas roughly half of all events go missing; with ten, ninety percent do. The tell is that the failure rate tracks replica count exactly, and that a single-replica deployment works flawlessly.

The router affinity problem. A client’s subscription lives on the router process that accepted it. If the load balancer moves that client — during a reconnect, a scale-in, or a health-check failure — the client lands on a router with no knowledge of its stream. Unlike the first problem this is not silent: the client’s connection closes and it must resubscribe. It becomes a problem only when reconnection is not handled properly on the client, which is far more common than it should be.

The first problem is fixed with shared infrastructure. The second is fixed with client behaviour. No amount of broker configuration will make a client that does not resubscribe work correctly after a router replaces a pod.

Where an event goes missing with two subgraph replicas With an in-process emitter an event published on replica A is delivered only to subscribers held by replica A. A subscriber attached through replica B never sees it. A shared broker makes every replica a valid delivery path. In-process emitter: half your events vanish at two replicas broken — in-process emitter write on replica A emitter A subscriber on A ✓ subscriber on B ✗ no path exists working — shared broker write on replica A shared broker subscriber on A ✓ subscriber on B ✓

Implementation walkthrough

The subgraph change is small and mechanical: replace the in-process emitter with one backed by the broker, and make sure every place that mutates state publishes through it — including background workers and admin tooling, which are the usual sources of “some changes notify and others do not”.

import { RedisPubSub } from "graphql-redis-subscriptions";
import Redis from "ioredis";

// Two connections: Redis cannot publish on a connection in subscribe mode.
const options = { host: process.env.REDIS_HOST, retryStrategy: (t: number) => Math.min(t * 50, 2000) };
export const pubsub = new RedisPubSub({
  publisher: new Redis(options),
  subscriber: new Redis(options),
});

export const resolvers = {
  Subscription: {
    orderStatusChanged: {
      // Topic per entity keeps fan-out proportional to interest rather than
      // broadcasting every order change to every subscriber.
      subscribe: (_: unknown, { orderId }: { orderId: string }) =>
        pubsub.asyncIterator(`order:${orderId}`),
    },
  },
};

// Called from the HTTP path, from workers, and from admin tools alike.
export const publishOrderChange = (order: { id: string }) =>
  pubsub.publish(`order:${order.id}`, { orderStatusChanged: order });

The topic design in that snippet matters more than the broker choice. A topic per entity means a subscriber to one order is woken only by that order; a single global orders topic means every replica filters every event for every subscriber, and CPU grows with the product of event rate and subscriber count. Where subscribers legitimately need a broad stream — a dashboard watching all orders — give that its own coarse topic rather than making everyone pay for it.

On the router side, scaling out needs no shared state at all, because each router owns its clients’ connections and nothing else. What it does need is a load balancer configured for the traffic:

# Kubernetes Service / Ingress essentials for a scaled router
metadata:
  annotations:
    # Long idle periods are normal for a subscription; the default is not.
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
spec:
  # Give in-flight streams time to drain rather than cutting them at rollout.
  terminationGracePeriodSeconds: 120

Verification steps

The decisive test is deliberately adversarial: scale the subgraph to at least three replicas, open several subscriptions, and publish events from a process that is not serving any of them.

kubectl scale deploy/orders --replicas=3

# Publish from a one-off pod — no subscriber is attached to this process.
kubectl run publisher --rm -it --image=orders:latest -- \
  node -e 'require("./dist/publish").publishOrderChange({id:"O-1"})'

Every subscriber to O-1 must receive the event. If some do and some do not, the emitter is still process-local somewhere in the code path — often in one branch that was missed, such as an admin mutation or a retry handler.

Then verify router affinity handling by deleting a router pod while subscriptions are open. Clients should disconnect, reconnect, resubscribe, and refetch current state. A client that reconnects without refetching will silently display data from before the gap, which is the failure mode users report as “it stopped updating” long after the deploy that caused it.

Finally, confirm topic granularity is doing its job by watching broker delivery counts while one entity changes. A single order changing should not wake subscribers watching other orders.

Topic granularity decides whether fan-out is a problem With one global topic every subscriber is woken by every event, so deliveries grow with subscriber count. With a topic per entity only interested subscribers are woken, and delivery volume stays flat. Broker deliveries caused by one order changing one global topic — everyone is woken topic per entity — only interested subscribers 100 total subscribers 10,000 Filtering in the resolver instead of in the topic produces the pink line with extra CPU on top — every replica deserialises every event only to discard almost all of them.

Scaling failures in subscriptions are unusually easy to misattribute, because several quite different causes all present as “it stopped working for some people”. Mapping each symptom to the layer that owns it first avoids the common trap of tuning the broker when the problem is in the client, or rewriting client reconnect logic when the problem is a topic design that wakes everyone.

Four symptoms, four different layers Missing events point at the publish path, deploy-time drops point at client reconnect behaviour, CPU growth points at topic design, and silence without errors points at the broker. Symptom to layer when scaling out symptom likely cause where to look some subscribers miss events an in-process emitter the publish path clients drop on every deploy no resubscribe on reconnect the client CPU grows with subscribers one global topic topic design silence with no errors at all broker outage the delivery-rate metric None of these four is fixed in the same place, which is why guessing wastes so much time. Alert on a floor for delivery rate, or the last row stays invisible until a user notices.

Common mistakes & gotchas

Publishing from only one code path. The HTTP mutation publishes, the background job does not, and half the state changes silently produce no event. Centralise publication in one function that every writer calls.

One Redis connection for publish and subscribe. A connection in subscribe mode cannot publish, so a single-connection setup fails at the first publish. Use two clients, as in the example above.

Filtering in the resolver rather than in the topic. It produces correct results and terrible scaling, because every replica pays deserialisation cost for events none of its subscribers want.

Frequently Asked Questions

Does the router need sticky sessions?

No, and asking for them usually indicates a client that does not resubscribe on reconnect. Each router owns the connections it accepted; a new connection landing on a different replica is fine as long as the client re-establishes its subscriptions. Stickiness only papers over the real gap, and it makes scale-in and rolling deploys worse.

Which broker should we use?

Whichever you already run. Redis pub/sub is the common default because it is simple and usually already present, at the cost of no delivery guarantee — an event published while a subscriber is momentarily disconnected is gone. Kafka or NATS JetStream add durability and replay if you need them, but that durability rarely helps a GraphQL subscription, whose contract is current state rather than history.

Do the routers need to know about each other?

No. Routers hold client connections and nothing shared, so scaling them is ordinary stateless horizontal scaling. The only coordination in the system is the broker between subgraph replicas.

How do we size the router fleet?

From peak concurrent subscriptions divided by the per-process limit you set with max_opened_subscriptions, plus headroom for a replica being unavailable during a deploy. Memory per connection is small but not zero, and the enrichment work per event is usually the binding constraint before connection count is — so load-test with a realistic selection set rather than an empty one.

What happens during a broker outage?

Streams stay open and events stop flowing, which is the best available failure mode: clients see silence rather than errors. Because that silence is indistinguishable from “nothing is happening”, alert on a floor for events delivered per minute on your busiest topics, or the outage will be invisible until a user reports it.

Does the broker choice affect the router configuration at all?

No — the router never talks to your broker. It talks to subgraphs, and the broker is entirely a subgraph implementation detail. That separation is genuinely useful: you can migrate from one broker to another, or run two during a transition, without touching router configuration, republishing a schema, or reconnecting a single client.