A CDN data center that does not have an asset asks the origin for it. With hundreds of data centers, each with its own cache, the origin sees the first request for a popular asset hundreds of times over, and its owner pays the cloud for every byte it sends out.
Cloudflare's Tiered Cache turns the flat network into a hierarchy. The data centers near visitors are the lower tier (the edge). When they miss, they ask an upper tier, a few data centers close to the origin, and only the upper tier asks the origin. Smart Tiered Cache picks the upper-tier data center for each origin and keeps a fallback in another location. Cloudflare says Tiered Cache cuts the cache miss rate by 60% or more.
Design the CDN side for one customer's site.
Functional requirements
Fetch asset: a visitor requests a cacheable asset (an image, a script) and gets 200. Three scenarios:
"Edge hit": the visitor's edge data center has it.
"Upper-tier hit": the edge misses, and the upper tier has it.
"Upper-tier miss": neither has it, and the upper tier fetches it from the origin.
Purge: the site owner changes an asset and purges it through an API. The old copy must be gone from both tiers when the owner hears 200, and no edge may refill the old copy afterwards.
Name the two tiers edge and upper: the tests in problem.proschi refer to them, and to the use case and scenario names above.
Scale
50k requests per second for the site at peak; 90% are edge hits.
Of the 10% that miss the edge, the upper tier answers 60%, the cut in misses Cloudflare reports: 6% of requests are upper-tier hits, 4% go to the origin.
About 50 purges per second, each reaching some 300 edge data centers.
Constraints
The origin has 4 servers that take 1k requests per second each and answer in 50 ms. You cannot add more.
Only the upper tier talks to the origin: no connection from the edge or the visitor to it, so the origin can allow just the upper tier's addresses.
p99 of Fetch asset under 150 ms, available 99.99% of the time.
Losing any single machine, an upper-tier data center included, must not break a latency limit or overload the origin.
At most $3,500 / month, the origin included.
What is given
problem.proschi declares the visitor, the owner and the origin, and holds the traffic, requirements and tests. Add the two tiers, the purge API, the connections and the two use cases.
The simulation is simpler than the real network: each tier is one node whose replicas stand for its data centers, the hit rates are the given mix rather than a result of the topology, and traffic between your own nodes is free, so the origin's egress bill shows up as load on the origin instead.
Based on
Argo, Cloudflare blog, May 2017: Argo Tiered Cache, which asks other Cloudflare data centers before the origin, and the 60% reduction in the cache miss rate.
Tiered Cache Smart Topology, Cloudflare blog, February 2021: lower and upper tiers, and the single best upper-tier data center chosen for each origin to raise the hit ratio and lower origin load.
Tiered Cache, Cloudflare docs: the topologies (Smart, Generic, Regional, Custom).
How your design is checked
You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.
A CDN is a network of data centers that cache your content near visitors; on a miss, a data center fetches the asset from the origin, your own web servers. With hundreds of data centers caching on their own, the origin sees the first request for a popular asset hundreds of times, and its owner pays the cloud for every byte it sends.
Cloudflare's Tiered Cache turns that flat network into a hierarchy. Data centers near visitors are the lower tier (the edge); on a miss they ask an upper tier, a few data centers close to the origin, and only the upper tier talks to the origin. You are designing the CDN side for one customer's site.
Two use cases:
Fetch asset: a visitor requests an image or script and gets 200. Three scenarios: "Edge hit" (the edge has it), "Upper-tier hit" (the edge misses, the upper tier has it) and "Upper-tier miss" (the upper tier fetches it from the origin).
Purge: the site owner changes an asset and calls a purge API. By the owner's 200, the old copy is gone from both tiers, and no edge may refill it.
Non-functional requirements: 50k requests per second, p99 under 150 ms, 99.99% availability, survival of any single machine (an upper-tier data center included), and $3,500 a month, the origin included.
What is given.given.proschi fixes the visitor, the owner and the origin: four servers that take 1k rps each, answer in 50 ms and cost $500 a month each. You cannot add origin servers: that is the point. A real origin is a customer's app in one cloud region, which the CDN must protect.
What the tests check: an edge miss asks the upper tier, and only an upper-tier miss reaches the origin; no path at all leads from the visitor or the edge to the origin; a purge sends PURGE to the upper tier before the edge and never touches the origin. The requirements add latency, availability, failure survival and cost.
The model's simplifications, from the statement: each tier is one node whose replicas stand for its data centers; hit rates are the given mix, not an outcome of the topology; and traffic between your nodes is free, so the origin's egress bill (what the cloud charges for data sent out) shows up as origin load instead.
Why does an upper tier cut misses so much? A flat CDN with N data centers fetches each asset up to N times, once per cold cache; with an upper tier, the upper tier fetches it once and the edges fill from it. Cloudflare's launch post promised customers a 60% drop in their cache miss rate from tiering, the number this problem uses.
Deep dive How the simulation sees it
At 50% utilisation the four origin servers queue a little: the 50 ms base becomes a bit over 54 ms. At 125% the node is saturated and every latency requirement of a use case that loads it fails. The upper-tier miss path is edge (5 ms) + upper tier (5 ms) + origin (about 54 ms), around 64 ms on average; because only 4% of requests take it, the use case's p99 lands well under 150 ms. The flat design's p99, with a saturated origin, is several hundred milliseconds.
Cost. The origin is $2,000 of the $3,500, leaving $1,500. A CDN replica and a service each cost $100 in the simulation's price table. CDN capacity (200k requests a second per replica) is far above what either tier sees, so replicas here are about failure, not load.
Failure.survive any node failure removes one replica of each node you run and re-runs the analysis. With one upper-tier data center, losing it breaks every edge miss, unless you add a fallback: that is where the classic mistake lives.
A cache hierarchy puts caches in layers: small caches near users, fewer bigger ones behind them, the source of truth at the end. CPUs (L1, L2, L3, RAM), browsers (memory, disk, network) and CDNs (lower and upper tiers) all do it. Commercial CDNs call the upper tier an origin shield: one designated location that every miss for an origin must pass through.
Why it works: miss rates multiply. If the edge misses 10% and the upper tier misses 40% of what reaches it, the origin sees 10% × 40% = 4%. The upper tier also hits far more often than any single edge, because it aggregates all edges' misses: an asset requested once in Tokyo and once in Paris is two edge misses but one upper-tier miss.
Trade-offs: an upper-tier hit adds a hop, often a long one, since the upper tier sits near the origin, not the visitor. Tiering slows edge misses a little to make the origin's life much easier. It also concentrates load on a few data centers, so the upper tier must be redundant. Tiering buys nothing with a single cache location or with uncacheable content (personalised pages). It buys less when the origin is a scalable store such as S3: load no longer matters there, though every fetch from it is still egress you pay for.
Deep dive The pattern in Proschi
title"Two cache tiers"user "User"[Actor]near "Near Cache"[CDN] x2shield "Shield"[CDN] x2source "Source"[Service] x2user -> near : HTTPSnear -> shield : fillshield -> source : fillusecase"Get" { user -> near : GET /a.png alt"Near hit" { near --> user : 200} alt"Shield hit" { near -> shield : GET /a.png shield --> near : 200 near --> user : 200} alt"Miss" { near -> shield : GET /a.png shield -> source : GET /a.png source --> shield : 200 shield --> near : 200 near --> user : 200 }}
When a cache layer fails, the tempting fallback is "go straight to the source": safe for a tiny cache, dangerous for a big one. The cache existed because the source cannot take the full load; the moment it disappears, the source receives everything it was shielded from, at once. This is a thundering herd (or cache stampede): a crowd of requests the cache used to absorb, all arriving at the source together.
The right failover keeps the shape of the hierarchy: a second member of the same tier (a fallback upper-tier data center in another location), not a path that skips the tier. Cloudflare's later work on Smart Tiered Cache describes exactly this: a primary and a fallback upper tier for each origin. The origin can then lock its firewall to the upper tier's addresses, which is why the tests forbid any path from the edge or the visitor to it.
Other tools in the same family: request collapsing (many concurrent misses for one key become one origin fetch), serving stale content while the origin is unreachable, and load shedding at the origin. Proschi models none of them directly, but the failure analysis makes the core lesson visible.
Purging means deleting cached copies after the source changes, and in a hierarchy the order matters. Purge the edge first, and an edge that gets a request between the two purges misses, asks the upper tier (which still has the old copy) and refills itself with it. The purge "succeeded" and the old asset is back.
The fan-out is large: one purge must reach every edge data center. In Proschi an x300 prefix on a step means it happens 300 times per request; load counts all 300 calls while latency counts the step once, as if they ran in parallel.
Alternatives to purging are versioned URLs (app.3f9a.js, never purged, just replaced) and short TTLs (time to live: how long a cached copy stays valid). Versioned URLs are the better default for static assets; purges are for content whose URL cannot change.
Clarify: is the content cacheable and the same for everyone? (Yes: images and scripts.) How many edge locations? (About 300.) The origin's capacity, and can it grow? (4k rps, fixed.) How fast must a purge take effect? (Before the owner hears 200.) Say that the fixed origin capacity drives everything.
The starter is a flat CDN: visitor → edge → origin. Do the arithmetic from the table and show the origin saturating at 125%. The obvious alternative, more origin servers, is ruled out by the constraints, and in reality costs servers and egress that a better cache hit makes unnecessary.
The winning design adds a node with id upper between edge and origin: visitor → edge → upper → origin. Write the three fetch scenarios as three branches of one alt: the edge hit touches nothing but the edge, the upper-tier hit adds one hop, and only the upper-tier miss reaches the origin.
Then add a purge service for the owner: owner → purge API → upper tier, then edge.
Redundancy at every tier. Every node you run must survive losing one replica: for the edge and the purge API, two replicas. For the upper tier, resist the fallback-to-origin idea and give the tier a second data center. Check the analysis: no single points of failure, and the origin still at 50% after an upper-tier data center is lost.
The purge. Clear the upper tier first, then fan out to every edge with an x300 step, then answer the owner; the flow test asks for upper before edge. The purge never calls the origin.
Latency. The miss path dominates p99 because of the origin's 50 ms (see the per-scenario numbers in the analysis), so keep the origin unsaturated and p99 follows. If p99 is high, check the origin's utilisation first.
Budget. Replicas times price, plus the $2,000 origin: the tiers are cheap next to the origin they protect.
Summarise: two tiers, so 96% of requests never reach the origin; a redundant upper tier instead of an origin fallback; purges from the upper tier outward. Then what the model leaves out: real hit rates depend on topology and asset popularity, a far upper tier adds latency to misses, and request collapsing at the upper tier would cut origin load further during a cold start.
Every edge fills from the origin (wrong/every-edge-fills-from-origin.proschi). The flat CDN: no upper tier, edges go straight to the origin. It is how a CDN behaves with tiering switched off, and it works while the origin is big enough for the number of edges; when it is not, cold caches after a deploy or a purge overload it. Here it fails three ways: "Edge misses go to the upper tier, not the origin", "Only the upper tier talks to the origin", and p99, because 5k rps saturates a 4k-rps origin.
One upper-tier data center (wrong/single-upper-tier.proschi). Everything flows correctly, but the upper tier has one replica and no fallback: when it fails, no edge miss can be filled. It fails survive any node failure. In production this is a single "shield POP" whose failure is a global outage for cache misses.
Upper tier down, so go to the origin (wrong/upper-down-goes-to-origin.proschi). A single upper tier with a -x failure scenario that falls back to the origin. It sounds resilient, but with the upper tier down the origin takes every edge miss, the very overload the tier prevents, and can no longer restrict who talks to it. It fails "Only the upper tier talks to the origin", because the design now has an edge-to-origin connection, and survive any node failure, because the lone upper tier is still a single point of failure for purges.
Purging the edge first (wrong/purge-edge-first.proschi). Same design, purge order reversed: between the two purges an edge can refill the stale copy from the upper tier, and it survives the purge. It fails "A purge clears the upper tier before the edge".
A load balancer as the edge. It distributes requests but caches nothing, so every request reaches whatever is behind it. Use a CDN technology such as [Cloudflare] for both tiers.
Then show the miss-rate arithmetic: 10% edge misses times 40% upper-tier misses equals 4% at the origin, 2k of 50k requests a second and half of its capacity.
Expected follow-ups:
How do you pick the upper-tier location? Close to the origin, by latency from candidate data centers; Cloudflare's Smart Topology picks it per origin. For an origin in several regions, one upper tier per region.
What happens on a cold start, such as after a deploy? Many edges miss on the same new asset at once. Request collapsing at the upper tier makes them one origin fetch; versioned asset URLs let you pre-warm.
Why not purge with a short TTL instead? A short TTL means constant revalidation traffic to the origin; purges keep long TTLs and still change content immediately. Versioned URLs avoid both for static assets.
How does the purge reach 300 data centers quickly? A fan-out from a central service, or a pub/sub channel each data center subscribes to; acknowledge the owner only after all confirm (or a bounded timeout, with retries).
Does tiering make latency worse? An upper-tier hit adds one hop, but is usually faster than a trip to a distant origin, and edge hits, the vast majority, are unchanged.
System Design Interview – An Insider's Guide, Vol. 1 (Alex Xu): the chapter "Scale From Zero To Millions Of Users", which introduces CDNs and cache considerations.