System design practice

Config Distribution

medium Based on a system published by Cloudflare replication fan-out read-heavy availability real-world

Cloudflare's Quicksilver: config written once, sent down a tree, read locally on each edge server.

Solve it in your browser Read the lesson first

Cloudflare's network answers millions of HTTP requests a second, and nearly every one depends on configuration: the site's DNS records, its firewall rules, its Workers, its certificates. A customer can change any of it at any moment, and the change has to reach every machine in some 200 cities within seconds. Requests cannot fetch configuration from a central place: each data center must keep serving even when cut off from the core.

Cloudflare's answer is Quicksilver, a replicated key-value store. Writes go to a single root, which numbers them in order. The changes are copied down a tree, from the root to main nodes and from them to every edge server, which keeps a full copy in a local LMDB file. A request reads that local copy and never leaves the machine.

Design the distribution of one kind of configuration, customer settings.

Functional requirements

Name the root root and the tier between it and the edge mains: the tests in problem.proschi refer to them, and to the use case names above.

Scale

Constraints

What is given

problem.proschi declares the visitor, the customer, the edge servers and the local stores, one replica of each per data center, and holds the traffic, requirements and tests. Add the config API, the root and the main nodes, connect them, and write the two use cases.

What this model leaves out

Based on

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 14 min read

Learn it: Config Distribution #

Config Distribution: write once, copy everywhere, read locally #

What you'll learn #

The problem, explained #

Every request Cloudflare serves needs configuration. Is this hostname a customer? Which firewall rules apply? Which Worker runs? What is the load-balancing weight of the second origin? The answers are written by customers, through the dashboard and the API, and they must be on every machine that serves traffic, in some 200 cities, a few seconds after the customer saves them.

Cloudflare's first system for this was Kyoto Tycoon (KT): central services wrote to root nodes, which fed a management node in each data center, which fed every server. It worked until scale broke it. Writes blocked reads, crashes corrupted databases, replication could silently skip entries, and failing over the root needed a human. Cloudflare replaced it with Quicksilver: the same shape of tree, rebuilt around LMDB, an embedded key-value store that every edge server keeps on its own disk.

Two use cases:

The rates are assumptions sized from the post: 1M requests a second at the edge (Cloudflare reports over 14M at peak), 1k changes a second at peak (it reports about 30 million writes a day), 2 KB per change. The limits are a p99 (the latency 99% of requests beat) under 20 ms for requests and 100 ms for changes, 99.99% availability, surviving the loss of any machine, the root included, and $5,000 a month.

What the tests check: requests read local in every scenario and never root or mains; a change is written to root before the answer and the customer never waits for mains or local; the root sends to the main nodes, the main nodes to the local stores, and nothing goes from the root straight to the edge.

Deep dive What the model leaves out

The simulation has no clock for work done after the answer, so it cannot say how long a change takes to reach the edge. Async copies show up as load and bytes, which is enough to decide the shape of the tree but not its speed. Batching, sequence numbers and catch-up are taught below, not graded.

Back-of-the-envelope #

The simulation's [Key-Value Store] node serves about 20k operations a second, carries 100 MB/s and costs $500 a month. The local store on an edge server takes 50k reads a second.

The table works through the two decisions in order. First, where reads come from: the local store or central nodes. Then, who copies each change to 200 data centers: the root (the one store that takes every write) on its own, or a tree in which the root sends to a few main nodes and they send on to the data centers.

configuration reads at the edge
3M/s
store writes after fan-out
200k/s
bytes reaching the edge
400 MB/s
what the root sends through a tree
12 MB/s
main nodes, one of them spare
6
monthly cost
$4,200
QuantityArithmeticResult
Configuration reads1M requests × 33M/s
Reads per data center3M ÷ 20015k/s, 30% of a local store
Central nodes to serve them3M ÷ 20k150 at 100%
Store writes after fan-out1k changes × 200200k/s, 1k per data center
Bytes to the edge200k × 2 KB400 MB/s
Root sending to every data center400 ÷ (2 × 100)200%: saturated
Root sending to 6 main nodes1k × 6 × 2 KB12 MB/s, 6%
Main tier traffic400 out + 12 in412 MB/s
Six main nodes, one down412 ÷ 50082%
Five main nodes, one down410 ÷ 400103%: saturated
Cost2 × $100 + 2 × $500 + 6 × $500$4,200

Two numbers decide the design. Reads outnumber writes 3,000 to 1, so the read path must be as cheap as possible: a lookup on the local disk. And every write is multiplied by the number of data centers, so the copying has to be spread over more machines than the one that took the write.

Concepts #

Read locally, never from the core #

Configuration is read on every request, which makes its read path part of every request's latency and availability. If an edge server asked a central store, a request in Sydney would wait for a round trip to wherever that store lives, and a cut fibre between Sydney and the core would take Sydney down. Cloudflare's post states the rule plainly: each data center must be able to serve requests even when cut off from any source of central configuration.

So each edge server holds a full copy of the configuration, on its own disk, and reads it there. Quicksilver uses LMDB, a memory-mapped B-tree that many processes on the machine can read at once and that answers in microseconds. The post reports about 2.5 trillion reads a day, at an average latency in microseconds. For the DNS service, moving from KT to Quicksilver cut the p99 of reads by two orders of magnitude.

The price is twofold. Every server stores everything, which works for configuration and not for user data. And a local copy is stale for as long as replication takes: a change saved now is served a few seconds later, fine for settings, not for a bank balance.

This is not a cache #

The Tiered CDN Cache problem also has edges and a tier above them, but its edges hold a subset of the data and fetch what they miss from the origin. Here there are no misses: every edge holds every key, and a read never leaves the machine. A cache needs a path to the source on the request path; a replicated store does not. That is what lets a data center run while cut off, and what makes the write side the hard part.

The root: single writer, replicated #

Writes are rare compared with reads: thousands a second against millions. That allows a simplification most databases cannot afford: one place takes every write. Cloudflare picks a single data center to aggregate writes. It gives each change the next number of a monotonically increasing sequence, so every replica applies changes in the same order, and a replica that sees entry 25 after 23 knows it missed 24 and can ask for it again.

Because every write passes through it, the root is the one machine whose loss stops every change. KT's root needed a person to promote the standby, so a root failure meant configuration stopped propagating for a while. Quicksilver runs a primary and a secondary. In the model, one root node fails survive failure of root, and its 99.99% availability multiplied by the API's drops Change config under the 99.99% limit.

Note what does not break when the root is down: requests. The edge keeps serving the last configuration it received. That separation, between the part that changes the system and the part that runs it, is why the read path never touches the root.

Write once, copy in the background #

A change is durable once the root has it. Copying it to 200 data centers takes longer and depends on all of them, so it must not hold the customer's request. The API writes to the root, answers 200, and the copy runs after the answer. In Proschi that is an async send, ->>, placed after the response.

If the API waited for every data center, the slowest of 200 would set every write's latency, and one unreachable data center would block every change. With many independent replicas, waiting for all is waiting for the worst.

What the customer gets in exchange is eventual delivery: the change is saved now and live everywhere a few seconds later. Cloudflare measures that gap by writing a heartbeat at the top of the tree and comparing its time on each server, because a lagging replica looks healthy until someone notices their change is not taking effect.

A fan-out tree #

Each change must be copied 200 times. If the root does all the copying, it sends 200 × 2 KB per change: 400 MB/s at 1k changes a second, twice what two root nodes carry. Adding root nodes would spread the bytes but not the job: every one of them would still talk to every data center over long links.

A tree spreads the work. The root sends each change to a handful of main nodes; each main node sends it on to its share of data centers. The root's traffic drops to 12 MB/s, and the 400 MB/s that must reach the edge is shared by the main tier. Quicksilver does exactly this: edge nodes replicate from main nodes, which replicate from the top-mains. Each node usually replicates from a close neighbour, so the long links carry one copy per region instead of one per server.

Size the middle tier like any pool, by its bottleneck, and here that is bytes, not requests. It moves about 412 MB/s (the 400 it sends, plus the 12 it receives). At 100 MB/s a node, five would run at 82% and pass 100% when one fails, so six it is: 69% normally, 82% with one down.

Deep dive How Quicksilver replicates

Replication is pull-based: a node asks its parent for every log entry after the last sequence number it has. Log entries live in the same LMDB file as the data, in a separate bucket, so an entry and the change it describes are committed together. Updates from a 500 ms window are batched into one disk write, which made durable writes affordable. Each instance has primary and secondary sources; the secondary main nodes keep a much longer history, so a machine that was offline for a week can still catch up. Each database has a unique id, required to start replicating, and a random process id stops a node from replicating from itself.

Designing it step by step #

1. Scope the problem #

Ask how often configuration is read and how often it changes, how fast a change must be live everywhere (seconds), and what happens to a data center that loses its link to the core (it must keep serving). Those three answers set the design: local reads, a single writer, async copies.

2. High-level design #

Use casePathWhy
Serve requestvisitor → edge → localReads never leave the server
Change configcustomer → API → root, answerDurable on one replicated writer
(after the answer)root → main nodes → localThe copy spreads over a tree

3. Deep dive #

The read path. Three lookups on the local store per request: 15k reads a second per data center, 30% of its read capacity, next to 1k replicated writes.

The root. Two nodes, a primary and a secondary. It takes 1k writes a second and sends 12 MB/s.

The main tier. Six nodes, sized for 412 MB/s with one down. Write the fan-out with prefixes, so the simulation counts every copy and every byte.

Freshness. Explain how a change reaches the edge in seconds and how you would know if it did not.

4. Wrap-up #

Summarise: read locally, write once to a replicated root, copy asynchronously down a tree. Name next steps: monitoring lag per server, sharding configuration into separate databases by kind (Cloudflare runs separate instances for DNS and Page Rules), and protecting against a bad change spreading as fast as a good one.

Common mistakes #

Reading the central store on every request (wrong/read-the-central-store.proschi). The edge asks the root for its settings. 3M reads a second land on two nodes that serve 40k, the root saturates, and requests and changes both slow down; even if it were big enough, every data center would depend on its link to the core. It fails the p99 of Serve request and "Edge requests read the local copy".

The root sending to every data center (wrong/root-sends-to-every-edge.proschi). No main nodes: the root sends 400 MB/s against 200 MB/s of capacity and saturates, which slows the writes that go through it. It fails the p99 of Change config and "Changes reach the edge through the main nodes".

Waiting for every data center (wrong/wait-for-every-edge.proschi). The copy runs inside the request. The model counts a fan-out's latency once, so the numbers barely move (in reality the slowest data center would decide): only with an API node down does the p99 of Change config cross 100 ms. It fails "A change is written once, then copied in the background", and survive any node failure.

A single root (wrong/single-root.proschi). The tree is right, but its top is one machine. Losing it stops every change, and Change config's availability drops under 99.99%. It fails survive failure of root and the availability limit.

In the interview #

Lead with the ratio and the outage rule: "reads are thousands of times more frequent than writes, and a data center must serve when cut off, so every server keeps a full local copy." The rest of the design follows from making that copy arrive.

Expected follow-ups:

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems