System design practice

Multi-Region Active-Active

hard Based on a system published by Netflix availability replication real-world

Netflix's two US regions both serve, replicate asynchronously, and either can carry everyone.

Solve it in your browser Read the lesson first

Netflix runs the services on a member's call path in two AWS regions at once, US-East-1 (Virginia) and US-West-2 (Oregon). Geo-DNS sends members in each part of North America to one of them, about half to each. Every service is stateless; everything a member's request reads or writes lives in Cassandra, which replicates between the regions asynchronously, and no request crosses from one region to the other.

Regions do not fail often, but a bad change can break a critical service in one. When that happens, Netflix overrides geo-DNS and sends every member to the healthy region, which must then carry all of North America's traffic on its own. Design both regions so that either can.

Functional requirements

Use these use case and scenario names exactly: the traffic, requirements and tests in problem.proschi refer to them.

Scale

Constraints

What is given

problem.proschi declares viewer, the member's device, and dns, the geo-DNS layer (UltraDNS's directional records over Route53, switched with Netflix's Denominator). Add both regions' nodes, connect them, and write the three use cases.

What this model leaves out

Based on

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 13 min read

Learn it: Multi-Region Active-Active #

Multi-region active-active: two regions serving, either one enough #

What you'll learn #

The problem, explained #

In 2013 Netflix's internal availability goal was 99.99%: under an hour of downtime a year. Its services already ran on many instances across several Availability Zones (AZs), which protects against a machine or a data centre failing. It does not protect against what worried Netflix more: its own pace of change. A bad deploy, a broken configuration or an overloaded middle-tier service can take down a critical path in a whole region at once, every zone included.

So Netflix deployed every service on a member's call path in two AWS regions, US-East-1 in Virginia and US-West-2 in Oregon, both serving members all the time. On a normal day geo-DNS sends each member to the closer region, roughly half to each. When a region goes bad, Netflix overrides DNS and sends every member to the healthy one. Members see a short blip instead of an outage, and engineers can fix the broken region without the clock running.

In the problem, members Browse (8k requests a second) and Save progress (2k), half in each region. Each region has its own Zuul (Netflix's edge gateway), a stateless playback API, and EVCache (Netflix's Memcached-based cache) in front of Cassandra. The regions share nothing except Cassandra's background replication. The third use case, Browse in failover, adds to each region the members it takes over when the other is evacuated, so both regions are measured on their worst day at once.

The limits: p99 (the latency 99% of requests beat) under 100 ms for Browse and Browse in failover and under 130 ms for Save progress; every use case available 99.99% of the time; no latency limit broken by losing any single instance, even with the failover load on; and at most $5,000 a month for both regions.

Back-of-the-envelope #

Netflix's availability goal: about 53 minutes a year
99.99%
members per region on a normal day
50/50
one region's API load when the other is evacuated
9k / s
API instances per region: 64% busy, 75% after losing one
7
the API's load on a normal day: the cost of standing by
~36%
a month for both regions
$4,600

A normal day. Each region gets half: 4k Browse and 1k Save progress, 5k requests a second at its Zuul and its API.

The drill. When US-East is evacuated, US-West keeps its own 4k Browse and 1k writes and takes US-East's 4k Browse: 9k a second in the model (the real day also moves US-East's 1k writes, see the statement). At 2k per API instance that is 4.5 instances' worth of work. Seven run at 64%; losing one leaves six at 75%, and p99 stays under 100 ms. Six would run at 75%, but losing one leaves five at 90%, and the queues push p99 over the limit.

Zuul. 9k a second at 10k per instance: two would do on paper (45%), but losing one leaves one at 90%. Three run at 30%, 45% after losing one.

Cassandra and EVCache. Only cache misses read Cassandra: 10% of a region's own 4k Browse (400 a second) and, in the drill, 30% of the 4k it takes over (1.2k; the lower hit rate for members just moved is an assumption). Add 1k local writes and 1k replicated from the other region: about 2.4k operations a second on a normal day and 3.6k in the drill. Two Cassandra nodes (20k each) are almost idle; they are there for availability, as are two EVCache nodes.

Per regionNormal dayOther region evacuatedCapacity
Zuul5k / s9k / s3 × 10k
Playback API5k / s9k / s7 × 2k
Cassandra~2.4k / s~3.6k / s2 × 20k

Cost. Per region: 3 Zuul ($300), 7 API ($700), 2 EVCache ($300), 2 Cassandra ($1,000): $2,300, and $4,600 for both, inside the $5,000 budget. Active-active roughly doubles the compute you pay for: on a normal day each API runs at about a third of its capacity.

Concepts #

Active-active versus active-passive #

Active-passive keeps a second region on standby: deployed, maybe even sized, but serving no members. It is simple for data, since only one region writes. But the standby is a part of the system nobody uses: its configuration drifts, its caches are empty, and the first real test of the failover is the outage itself. Failing over then means cold caches, instances that need warming and procedures that have not run in months.

Active-active runs both regions for real. Every deploy, every config change and every cache is exercised by live members every day, so a failover only moves members to a region that is already working. The price: both regions take writes, so data must replicate both ways, and each region needs room to carry everyone.

Steering members with DNS #

A member's device finds the API by name, and DNS answers with an address in one region. Netflix used directional records (UltraDNS) to send each part of North America to a chosen region, rather than latency-based routing, which could shift traffic unpredictably. In front of the load balancers it kept a Route53 layer of CNAMEs, whose API changes records quickly and reliably, and drove both through its own library, Denominator. Failing over is then one change: point a Route53 CNAME at the other region.

DNS has limits. Answers are cached for their TTL, so some devices keep using the old region for a while; and DNS sees the resolver's location, not the member's. Netflix's Zuul gateway handled the stragglers: a request that arrives in the wrong region is mis-routed, and Zuul can either proxy it to the right region or answer with a redirect. In failover mode, a region stops sending mis-routed requests away and serves them itself.

Stateless services, local data #

For any region to serve any member, a service instance must hold nothing a member needs on the next request: no sessions in memory, no local files. All state lives in the data tier. And every resource a request touches must be in the same region: its own Cassandra, its own cache, its own queues and buckets. An application that published to one S3 bucket now publishes to one per region.

The rule is simple: no cross-region call on a member's request. A call to the other region adds a cross-country round trip, tens of milliseconds, and, worse, makes this region's requests fail when the other region does. A design whose services run in two regions on one region's database has one point of failure with extra latency.

Asynchronous replication between regions #

If each region has its own copy of the data, writes made in one must reach the other. Netflix chose Cassandra partly because it replicates between data centres natively, in every direction and asynchronously. A write is acknowledged when a quorum of replicas in the local region has it (LOCAL_QUORUM); the copy to the other region follows in the background.

Synchronous replication across regions would wait for the other region on every write, adding the round trip to every bookmark, and failing writes whenever the link between regions is cut. Netflix's Split-brain exercise cut exactly that link to show that each region kept serving while replication queued up.

Async has a price. A member who writes in one region and is moved to the other right away may not see the write yet. Netflix tested how big that window was: it wrote 1 million records in one region and read them in the other 500 ms later, under production load, and found all of them. And two writes to the same row in two regions can conflict; Cassandra keeps the one with the later timestamp.

Keeping each region's cache honest #

Netflix reads must be fast, generally single-digit milliseconds, so EVCache, its Memcached client, sits in front of Cassandra, and some data lives only there. Each region has its own EVCache, filled by its own reads. Replicating Memcached between regions would mean building multi-master replication for a cache.

Netflix did something smaller: after every write, the EVCache client sends the other region a message (through SQS) to invalidate that key. The next read there misses, falls through to the local Cassandra and fills the cache with the new value. The cache stays a cache, and the database stays the only thing that replicates.

Headroom: each region carries everyone #

This is the decision that sizes the system. On a normal day each region serves half the members. On the day one is evacuated, the other serves all of them. A region sized for its half saturates at the moment the second region exists for.

With two regions, each needs room for double its normal load; in practice, a normal utilisation under half its capacity, minus a margin for losing an instance during the failover. With three regions, a failed region's load splits over two, so each may run at up to two thirds: more regions, less headroom each.

Members moved to a region are also cold for it: their rows are not in its EVCache. Expect more misses, and more Cassandra reads, while the cache warms. Netflix shifted traffic gradually in its drills so services could scale up and caches warm, and gave Zuul a maximum traffic level above which it sheds requests to protect services still scaling and a persistence layer behind cold caches.

Failing over, and proving it works #

A failover plan that is never run is a hope. Netflix's Simian Army tested it at three sizes: Chaos Gorilla takes out a whole Availability Zone; Split-brain cuts the network between the regions; Chaos Kong evacuates a region. In its Chaos Kong exercise, Netflix moved most US traffic to US-West-2, enough to show that one region could carry all of Netflix's load there, kept it there for over 24 hours, then moved it back to 50/50.

It also worked for real before the drills were done: when a middle-tier system in one region stopped answering, Netflix moved members to the healthy region and service came back quickly.

Designing it step by step #

1. Scope. Ask what must survive (a whole region, not just a zone), the availability goal (99.99%), the traffic and how it splits between regions, and which data each request reads and writes. Agree that a write may reach the other region a moment later.

2. Two regions. Draw one region as a full stack: Zuul, the stateless playback API, EVCache, Cassandra. Draw the second as its mirror. In front of both, DNS, with the device asking it before it calls a gateway.

3. Normal traffic. In Browse, each region's scenarios touch only that region's nodes: gateway, API, cache, then Cassandra on a miss. In Save progress, write the local Cassandra before answering.

4. Replication. After the answer, Cassandra copies the write to the other region and the API sends the other region an invalidation for its cache. Both are ->> steps: nothing waits for the other region.

5. Failover. In Browse in failover, each evacuated region's members go to the other region and touch nothing in their own. Then size: count each region's load with the other one's members added, keep each tier comfortable, and keep it comfortable after losing one more instance. Check the bill.

Common mistakes #

Each region sized for its half (wrong/sized-for-half). Four API instances per region carry a normal day at 62%. In the failover drill each region gets 9k requests a second against 8k of capacity, and saturates. Caught by the p99 of Browse in failover and survive any node failure.

A standby region (wrong/standby-region). DNS sends everyone to US-East; US-West only serves when US-East is evacuated. Nothing exercises the standby until it is needed, and its caches are empty when it is. Caught by "Both regions serve live traffic".

Waiting for the other region on every write (wrong/synchronous-replication). Save progress answers only after the other region's Cassandra has the bookmark. Every write pays a cross-country round trip and fails when the link between regions does. Caught by "Writes replicate to the other region asynchronously".

One database for both regions (wrong/one-database). Services in two regions, Cassandra in one. Every western miss and write crosses the country, and evacuating US-East sends members to a region that still depends on it. Caught by "Both regions serve live traffic" and "Evacuated traffic stays out of the failed region".

Beyond the tests:

In the interview #

Likely follow-ups:

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems