System design practice

Prioritized Load Shedding

medium Based on a system published by Netflix load-shedding resilience api-gateway availability real-world

Netflix's Zuul gateway: rank every request and shed logs before playback when a spike hits.

Solve it in your browser Read the lesson first

Netflix's edge gateway, Zuul, sits in front of every backend and sees every request from phones, TVs and browsers. Not all of those requests matter the same: a failed play request is an error on the screen, a lost pause marker means the next episode starts a little off, and a dropped log upload goes unnoticed.

When a retry storm, a bad deploy or an under-scaled service leaves a backend with more traffic than it can serve, serving everything slowly fails everyone. Zuul instead gives each request a priority as it arrives and, while the backend is strained, rejects the lowest-priority requests first, so playback keeps working through the spike. Design that gateway.

Functional requirements

Name the gateway zuul, an [API Gateway]: the tests in problem.proschi refer to it, to the API service api and to the use case and scenario names above.

Scale

The traffic is the peak of a spike. Netflix does not publish request rates or how much of each class it sheds; every number here is an assumption for the exercise.

Constraints

What is given

problem.proschi declares the device (the client: the Netflix app on any device) and api, the API service with its 8 instances, which you cannot change. It sets the traffic of the three use cases with the shares of "Served" and "Shed". Add Zuul, the connections and the three use cases.

What this model leaves out

Based on

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 12 min read

Learn it: Prioritized Load Shedding #

Prioritized load shedding: keep playback up when the backend can't keep up #

What you'll learn #

The problem, explained #

Netflix's devices (phones, TVs, browsers) talk to Netflix through Zuul, its API gateway. Behind Zuul, backends answer very different kinds of requests. Some decide whether a member can watch at all: pressing play. Some make the experience a little worse if they fail: the marker that remembers where you paused. Some nobody notices: log uploads and background requests, which are also among the most numerous.

Overloads happen: a retry storm, a service that was scaled too small, a bad deploy. When a backend gets more requests than it can serve, its queues grow, every request slows down, many time out, and clients retry, which adds more load. Everything fails together, playback included.

Netflix's answer, described in its 2020 post, is prioritized, progressive load shedding at Zuul. Every request gets a priority score as it enters. When a backend (or Zuul itself) is under strain, Zuul raises a threshold and rejects the requests below it, so the lowest priorities go first, and more of them as the strain grows.

In this problem, a spike asks the API service for 20k requests a second while its 8 fixed instances serve 16k. Three use cases:

The traffic mix already holds those shares: the simulation cannot decide them for you. Your job is the shape: everything through Zuul, shed requests answered by Zuul with a 503 that never touches the API service, playback with no shed path, and enough Zuul for the full spike, within $1,600 a month.

Back-of-the-envelope #

asked of an API service that serves 16k
20k / s
its load with nothing shed
125%
reach it once Zuul sheds: 56%
9k / s
Zuul's load at 4 instances, 67% after losing one
50%
a month for the reference design
$1,200

Without shedding. 6k + 4k + 10k = 20k requests a second, against 8 × 2k = 16k: 125%. A saturated service misses every latency limit, playback's included.

With shedding. Playback 6k, half the progress markers 2k, a tenth of the logs 1k: 9k a second, 56% of 16k. Losing one instance leaves 14k: 64%, still comfortable. Zuul sheds 11k of the 20k requests (9k logs, 2k progress markers), and members barely notice: nobody misses a log, and a lost pause marker only means resuming a little off.

Zuul. Every request arrives at Zuul, shed or not: 20k a second. At 10k per instance, two are at 100%; four run at 50%, and 67% after losing one.

Cost. 4 Zuul instances at $100 and the API service's 8 at $100: $1,200 a month, under the $1,600 budget. Serving every progress marker and log at the peak instead would need a second pool for 14k a second: 10 more instances at 70%, $1,000 more, for a spike that lasts minutes.

ClassAt the peakShedReaches the API
Start playback (critical)6k / s0%6k / s
Save progress (degraded)4k / s50%2k / s
Send logs (non-critical)10k / s90%1k / s
Total20k / s9k / s

Concepts #

Load shedding #

Load shedding is rejecting some requests, early and cheaply, so the rest are served at normal speed. A service at 150% of its capacity that accepts everything serves nobody well: queues grow, latency climbs past every client's timeout, and the work done for requests that time out is wasted. If it rejects half of its requests at once, it runs at 75% and serves the other half as if nothing were wrong.

What it costs: the rejected requests fail. That is only acceptable if the client can live with it, which is why which requests to reject matters more than how many.

Priority buckets #

Netflix ranks requests into three buckets by what a member loses when they fail:

BucketExamples from Netflix's postWhen it fails
Non-criticalLogs, background requestsNobody notices
Degraded experienceStop and pause markers, viewing history, player language selectionThe experience is worse; playback still works
CriticalRequests that affect the ability to playThe member sees an error when pressing play

Zuul turns each request into a priority score from 1 to 100, computed from the request's attributes as soon as it arrives, so every later stage can use it. Normally the score changes nothing. Under strain, a threshold rises and everything below it is rejected, like a priority queue that drops from the bottom.

One limit for all traffic (a single concurrency cap, a random drop) is the tempting shortcut. It drops a play request as readily as a log upload, so members see errors while logs still get through.

Shed at the edge, not in the backend #

Where the rejection happens decides whether it helps. A backend that checks priority and answers 503 has already accepted the connection, read the request and parsed it; under overload, that work is the overload. In this problem, an API service that sheds for itself still receives all 20k requests a second: 125% of its capacity, saturated, every class slow.

The gateway sees every request first. It can score it, compare the score with the threshold and answer 503 without forwarding anything. The backend only receives what it will serve.

Netflix applies this in two places:

Progressive shedding #

Shedding is not on or off. The further past its limits a backend is, the higher the threshold goes. Netflix uses a cubic curve: with mild overload only the very lowest scores go, and the threshold climbs steeply as overload approaches its maximum. At the moment this problem models, all but a tenth of the logs and half of the progress markers are shed, and no play request.

Deep dive The curve in Netflix's post

The post's numbers read roughly like this: about 35% overload still keeps the threshold in the mid-90s, about 80% brings it to 50, and about 95% to 10. The number falls as overload grows, so on the post's scale the threshold seems to count down from 100 towards the most important requests; this lesson calls the same movement the bar rising. Either way the shape is what matters: each step gives up a slice of less important traffic before touching the next, gently at low overload and decisively near the top.

The shedder must not fall over #

Rejecting is cheaper than serving, but not free: Zuul still accepts the connection, reads the request and answers. Size the gateway for everything that arrives, not for what it forwards. In this problem two Zuul instances would sit at 100% on the 20k requests a second; a saturated gateway slows every class at once, the critical one first among them. Four instances run at 50%, and 67% after losing one.

This is also why Netflix has global throttling: protecting the backends is pointless if the gateway in front of them is the next thing to collapse.

Why not just add capacity? #

An obvious alternative is to buy enough capacity for the worst spike: a bigger API service, or a separate pool for low-priority traffic (a bulkhead). Both are useful tools, and bulkheads in particular stop one class from starving another. But:

The usual answer combines them: provision for normal peaks with headroom, isolate where it is cheap, and shed by priority for the rest.

Retries: shedding without a storm #

A rejected client retries. If every device retries at once, the shed traffic comes straight back, often several times over. Netflix's Zuul answers a shed request with a hint, how many retries the device may make and over what time window, and gives higher-priority requests more retries than lower ones. In the reference design a shed log upload gets no retries for two minutes and a shed progress marker one retry after 30 seconds (illustrative values; the post names the fields, not the numbers).

Designing it step by step #

1. Scope. Ask what the traffic classes are and what each costs the user when it fails. Confirm that the backend cannot grow during the spike and that playback must never fail. Ask how much each class is shed at this strain (given here).

2. High level. Devices → Zuul → API service, no other path. Three use cases, one per class.

3. Deep dive.

Priority. Zuul scores each request when it arrives. Draw it as a label on the first step: critical, degraded, non-critical.

Shed at Zuul. For progress markers and logs, two scenarios: "Served", where Zuul forwards to the API service, and "Shed", where Zuul itself answers 503 with a retry hint and the API service is never called. Playback has a single path that always reaches the API service.

Check the numbers. The API service now gets 9k of its 16k a second. Size Zuul for all 20k, with room to lose an instance.

Retries. Explain the retry hint and why lower priorities get fewer retries.

4. Wrap up. Mention how you would know it works: Netflix watches streams per second (SPS) and tests shedding with injected failures on a small share of real traffic.

Deep dive How Netflix tested it

Netflix used its failure-injection tool to shed chosen priority ranges for chosen devices or members, and ran 45-minute experiments on its chaos platform with a small treatment group whose low-priority requests were throttled, comparing playback per device type. The first run found a race condition on Android and iOS, where a low-priority request failing caused sporadic playback errors. Fixed, the experiments became a regular check. In a 2020 incident similar to one that stopped members from playing in 2019, shedding kept streams per second steady.

Common mistakes #

Shedding inside the backend (wrong/shed-at-the-backend). Zuul forwards everything, and the API service answers 503 to low-priority requests. It still receives all 20k a second, 125% of what it serves, so it saturates and every class is slow. Caught by "Shed requests never reach the API service" and the p99 limits.

Shedding without priority (wrong/shed-without-priority). One concurrency limit at Zuul for every request, so a play request can be shed like a log upload. Caught by "Playback is never shed".

Buying capacity for the spike (wrong/provision-for-the-spike). A second pool sized to serve every progress marker and log at the peak. Playback is safe, but the bill goes to $2,200 a month for minutes of need. Caught by the budget and "Shed requests never reach the API service".

A gateway sized for what it forwards (wrong/undersized-gateway). The right shedding on two Zuul instances: 20k requests a second fill them to 100%, and losing one makes it worse. Caught by playback's p99 and survive any node failure.

Classic mistakes beyond the tests:

In the interview #

Likely follow-ups:

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems