Netflix's edge gateway, Zuul, sits in front of every backend and sees every request from phones, TVs and browsers. Not all of those requests matter the same: a failed play request is an error on the screen, a lost pause marker means the next episode starts a little off, and a dropped log upload goes unnoticed.
When a retry storm, a bad deploy or an under-scaled service leaves a backend with more traffic than it can serve, serving everything slowly fails everyone. Zuul instead gives each request a priority as it arrives and, while the backend is strained, rejects the lowest-priority requests first, so playback keeps working through the spike. Design that gateway.
Functional requirements
Start playback: a member presses play. This request is critical: it always reaches the API service, whatever the load.
Save progress: the app saves a pause or stop marker. Losing it degrades the experience but does not stop playback. Two scenarios:
"Served": Zuul forwards the request to the API service.
"Shed": Zuul answers 503 itself, with a hint of when the device may retry, and the API service never sees the request.
Send logs: the app uploads logs and other background data. It is non-critical. Same two scenarios, "Served" and "Shed".
Name the gateway zuul, an [API Gateway]: the tests in problem.proschi refer to it, to the API service api and to the use case and scenario names above.
Scale
The traffic is the peak of a spike. Netflix does not publish request rates or how much of each class it sheds; every number here is an assumption for the exercise.
During the spike, devices send 6k play requests, 4k progress markers and 10k log uploads per second: 20k requests a second.
The API service runs 8 instances of 2k requests a second: 16k a second, so the spike asks for 125% of what it can serve.
At this level of strain, Zuul's priority threshold sheds half of the progress markers and 90% of the log uploads, and no play requests. The traffic mix in problem.proschi already holds these shares.
One Zuul instance handles 10k requests a second.
Constraints
Every request goes through Zuul: no path from the devices to the API service.
The API service is fixed at 8 instances: it cannot scale out within the minutes the spike lasts.
A shed request is answered by Zuul with a 503 and never reaches the API service; a served one is answered by the API service.
Playback is never shed.
p99 of Start playback under 100 ms, of the other two under 150 ms.
Start playback available 99.99% of the time.
Losing any single machine must not break a latency limit.
At most $1,600 / month, the API service included.
What is given
problem.proschi declares the device (the client: the Netflix app on any device) and api, the API service with its 8 instances, which you cannot change. It sets the traffic of the three use cases with the shares of "Served" and "Shed". Add Zuul, the connections and the three use cases.
What this model leaves out
The shedding decision itself. In Netflix's system the threshold moves with the backend's error rate and concurrency or Zuul's CPU, and sheds more of each class the worse it gets. The simulation does not react to load, so the problem fixes the shed shares for one moment of the spike in the traffic mix. The tests check where shedding happens and that playback is never shed; the design does not earn the percentages.
The priority score. Zuul scores each request from 1 to 100 from its attributes; here each class is a use case of its own, so the priority is the use case.
What a rejection costs. Rejecting a request is much cheaper for Zuul than proxying it, but the simulation charges both the same, so Zuul's load here is on the high side.
Retries. The 503 carries how many retries a device may make and after how long. The simulation does not model devices retrying, so a retry storm, the overload's usual cause, is described, not simulated.
Zuul's own overload (global throttling) and the per-device testing Netflix ran are in the lesson, not the tests.
Based on
Manuel Correa, Arthur Gonigberg and Daniel West, Keeping Netflix Reliable Using Prioritized Load Shedding, Netflix TechBlog, November 2020: the three priority buckets (non-critical such as logs and background requests, degraded experience such as stop and pause markers, viewing history and player language selection, and critical, which affects the ability to play), a priority score from 1 to 100 computed at Zuul from each request's attributes, shedding below a threshold that rises progressively (along a cubic curve) as service throttling (a backend's error rate or concurrent requests) or global throttling (Zuul's CPU, concurrent requests or connections) kicks in, the maxRetries and retryAfterSeconds hint sent to devices, the failure-injection and 45-minute ChAP experiments, and a 2020 incident where shedding kept streams per second steady where a similar 2019 one had stopped members from playing.
How your design is checked
You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.
Netflix's devices (phones, TVs, browsers) talk to Netflix through Zuul, its API gateway. Behind Zuul, backends answer very different kinds of requests. Some decide whether a member can watch at all: pressing play. Some make the experience a little worse if they fail: the marker that remembers where you paused. Some nobody notices: log uploads and background requests, which are also among the most numerous.
Overloads happen: a retry storm, a service that was scaled too small, a bad deploy. When a backend gets more requests than it can serve, its queues grow, every request slows down, many time out, and clients retry, which adds more load. Everything fails together, playback included.
Netflix's answer, described in its 2020 post, is prioritized, progressive load shedding at Zuul. Every request gets a priority score as it enters. When a backend (or Zuul itself) is under strain, Zuul raises a threshold and rejects the requests below it, so the lowest priorities go first, and more of them as the strain grows.
In this problem, a spike asks the API service for 20k requests a second while its 8 fixed instances serve 16k. Three use cases:
Start playback: critical, 6k a second, never shed.
Save progress: degraded experience, 4k a second; at this strain half are "Shed", half "Served".
Send logs: non-critical, 10k a second; 90% "Shed", 10% "Served".
The traffic mix already holds those shares: the simulation cannot decide them for you. Your job is the shape: everything through Zuul, shed requests answered by Zuul with a 503 that never touches the API service, playback with no shed path, and enough Zuul for the full spike, within $1,600 a month.
Without shedding. 6k + 4k + 10k = 20k requests a second, against 8 × 2k = 16k: 125%. A saturated service misses every latency limit, playback's included.
With shedding. Playback 6k, half the progress markers 2k, a tenth of the logs 1k: 9k a second, 56% of 16k. Losing one instance leaves 14k: 64%, still comfortable. Zuul sheds 11k of the 20k requests (9k logs, 2k progress markers), and members barely notice: nobody misses a log, and a lost pause marker only means resuming a little off.
Zuul. Every request arrives at Zuul, shed or not: 20k a second. At 10k per instance, two are at 100%; four run at 50%, and 67% after losing one.
Cost. 4 Zuul instances at $100 and the API service's 8 at $100: $1,200 a month, under the $1,600 budget. Serving every progress marker and log at the peak instead would need a second pool for 14k a second: 10 more instances at 70%, $1,000 more, for a spike that lasts minutes.
Load shedding is rejecting some requests, early and cheaply, so the rest are served at normal speed. A service at 150% of its capacity that accepts everything serves nobody well: queues grow, latency climbs past every client's timeout, and the work done for requests that time out is wasted. If it rejects half of its requests at once, it runs at 75% and serves the other half as if nothing were wrong.
What it costs: the rejected requests fail. That is only acceptable if the client can live with it, which is why which requests to reject matters more than how many.
Netflix ranks requests into three buckets by what a member loses when they fail:
Bucket
Examples from Netflix's post
When it fails
Non-critical
Logs, background requests
Nobody notices
Degraded experience
Stop and pause markers, viewing history, player language selection
The experience is worse; playback still works
Critical
Requests that affect the ability to play
The member sees an error when pressing play
Zuul turns each request into a priority score from 1 to 100, computed from the request's attributes as soon as it arrives, so every later stage can use it. Normally the score changes nothing. Under strain, a threshold rises and everything below it is rejected, like a priority queue that drops from the bottom.
One limit for all traffic (a single concurrency cap, a random drop) is the tempting shortcut. It drops a play request as readily as a log upload, so members see errors while logs still get through.
Where the rejection happens decides whether it helps. A backend that checks priority and answers 503 has already accepted the connection, read the request and parsed it; under overload, that work is the overload. In this problem, an API service that sheds for itself still receives all 20k requests a second: 125% of its capacity, saturated, every class slow.
The gateway sees every request first. It can score it, compare the score with the threshold and answer 503 without forwarding anything. The backend only receives what it will serve.
Netflix applies this in two places:
Service throttling: per backend. When a backend's error rate or concurrent requests cross a threshold, Zuul sheds the lowest-priority requests to that backend.
Global throttling: for Zuul itself. When Zuul's CPU, concurrent requests or connection count cross a threshold, it sheds across all backends, because if Zuul goes down, nothing gets through.
Shedding is not on or off. The further past its limits a backend is, the higher the threshold goes. Netflix uses a cubic curve: with mild overload only the very lowest scores go, and the threshold climbs steeply as overload approaches its maximum. At the moment this problem models, all but a tenth of the logs and half of the progress markers are shed, and no play request.
Deep dive The curve in Netflix's post
The post's numbers read roughly like this: about 35% overload still keeps the threshold in the mid-90s, about 80% brings it to 50, and about 95% to 10. The number falls as overload grows, so on the post's scale the threshold seems to count down from 100 towards the most important requests; this lesson calls the same movement the bar rising. Either way the shape is what matters: each step gives up a slice of less important traffic before touching the next, gently at low overload and decisively near the top.
Rejecting is cheaper than serving, but not free: Zuul still accepts the connection, reads the request and answers. Size the gateway for everything that arrives, not for what it forwards. In this problem two Zuul instances would sit at 100% on the 20k requests a second; a saturated gateway slows every class at once, the critical one first among them. Four instances run at 50%, and 67% after losing one.
This is also why Netflix has global throttling: protecting the backends is pointless if the gateway in front of them is the next thing to collapse.
An obvious alternative is to buy enough capacity for the worst spike: a bigger API service, or a separate pool for low-priority traffic (a bulkhead). Both are useful tools, and bulkheads in particular stop one class from starving another. But:
Spikes come faster than capacity. Autoscaling takes minutes; a retry storm takes seconds.
You pay all month for minutes of need. Here, a pool that serves every progress marker and log at the peak costs $1,000 a month more, nearly doubling the bill, for traffic nobody would miss for a few minutes.
Spikes have no ceiling. Whatever you provision, a bigger spike can come. Shedding works at any size.
The usual answer combines them: provision for normal peaks with headroom, isolate where it is cheap, and shed by priority for the rest.
A rejected client retries. If every device retries at once, the shed traffic comes straight back, often several times over. Netflix's Zuul answers a shed request with a hint, how many retries the device may make and over what time window, and gives higher-priority requests more retries than lower ones. In the reference design a shed log upload gets no retries for two minutes and a shed progress marker one retry after 30 seconds (illustrative values; the post names the fields, not the numbers).
1. Scope. Ask what the traffic classes are and what each costs the user when it fails. Confirm that the backend cannot grow during the spike and that playback must never fail. Ask how much each class is shed at this strain (given here).
2. High level. Devices → Zuul → API service, no other path. Three use cases, one per class.
3. Deep dive.
Priority. Zuul scores each request when it arrives. Draw it as a label on the first step: critical, degraded, non-critical.
Shed at Zuul. For progress markers and logs, two scenarios: "Served", where Zuul forwards to the API service, and "Shed", where Zuul itself answers 503 with a retry hint and the API service is never called. Playback has a single path that always reaches the API service.
Check the numbers. The API service now gets 9k of its 16k a second. Size Zuul for all 20k, with room to lose an instance.
Retries. Explain the retry hint and why lower priorities get fewer retries.
4. Wrap up. Mention how you would know it works: Netflix watches streams per second (SPS) and tests shedding with injected failures on a small share of real traffic.
Deep dive How Netflix tested it
Netflix used its failure-injection tool to shed chosen priority ranges for chosen devices or members, and ran 45-minute experiments on its chaos platform with a small treatment group whose low-priority requests were throttled, comparing playback per device type. The first run found a race condition on Android and iOS, where a low-priority request failing caused sporadic playback errors. Fixed, the experiments became a regular check. In a 2020 incident similar to one that stopped members from playing in 2019, shedding kept streams per second steady.
Shedding inside the backend (wrong/shed-at-the-backend). Zuul forwards everything, and the API service answers 503 to low-priority requests. It still receives all 20k a second, 125% of what it serves, so it saturates and every class is slow. Caught by "Shed requests never reach the API service" and the p99 limits.
Shedding without priority (wrong/shed-without-priority). One concurrency limit at Zuul for every request, so a play request can be shed like a log upload. Caught by "Playback is never shed".
Buying capacity for the spike (wrong/provision-for-the-spike). A second pool sized to serve every progress marker and log at the peak. Playback is safe, but the bill goes to $2,200 a month for minutes of need. Caught by the budget and "Shed requests never reach the API service".
A gateway sized for what it forwards (wrong/undersized-gateway). The right shedding on two Zuul instances: 20k requests a second fill them to 100%, and losing one makes it worse. Caught by playback's p99 and survive any node failure.
Classic mistakes beyond the tests:
Queueing excess requests instead of rejecting them: they wait past the client's timeout and the work is wasted.
Rejecting with no retry guidance, so devices retry immediately and the spike doubles.
Ranking by endpoint alone, so a prefetch and a user's own click on the same endpoint get the same priority.
Never testing shedding in production, so the first real overload is also the first test.
How do you assign priority? At the gateway, from the request's attributes (for example the endpoint, the device type, whether a user triggered it), computed once when it arrives so every later stage can use it.
What triggers shedding? Per backend: its error rate or concurrent requests. For the gateway itself: CPU, concurrent requests, connections.
Won't shed clients just retry? Answer with a retry hint, fewer retries for lower priorities, and clients that back off with jitter.
Why not autoscale? It takes minutes, a spike takes seconds, and paying for the worst spike all month is expensive. Shed now, scale later.
How do you know it works? Inject shedding on a small share of real traffic and watch the playback metric per device.