System design practice

Metrics Ingest

hard Based on a system published by Uber streaming aggregation replication sharding write-heavy real-world

Uber's M3: aggregate before storage, quorum writes to a replicated time series store.

Solve it in your browser Read the lesson first

Uber's metrics platform, M3, collects everything its services and hosts measure. In 2018 it took in 500 million metrics a second across Uber and kept 6.6 billion time series. It did not store all of those datapoints: an aggregation tier rolled them up first (dropping tags such as the host that nobody queries, and downsampling to the resolutions each namespace keeps), so only 20 million a second reached storage. Storage is M3DB, a distributed time series database that keeps each shard on three replicas and writes with a quorum. Queries in PromQL or M3QL go through a query service.

Design one region's slice of it. The M3DB cluster is given, and it is fixed: you cannot add replicas or shards to it.

Functional requirements

Use these use case and scenario names exactly: the traffic, requirements and tests in problem.proschi refer to them.

Scale

Constraints

What is given

problem.proschi declares the collectors and grafana (the clients) and m3db, the storage cluster, with its fixed capacity, and holds the traffic, requirements and tests. Add the aggregation tier, the query tier, the connections and the three use cases.

The simulation cannot express "two of three": write a majority write as x3 (every replica gets the write) and a replica that is down as x2 and a failed call (-x). It does not model the windows themselves either: an aggregator holding datapoints in memory is simply a call that touches nothing else.

Based on

The posts give global rates; the slice's rates, the batch size and the cluster's per-replica capacity are assumptions for this exercise.

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 10 min read

Learn it: Metrics Ingest #

Every service and host emits metrics (request counts, latencies, CPU, queue depths) tagged with host, endpoint, region and more. Uber's M3 took in about 500 million a second in 2018 and stored only about 20 million. This lesson is about that gap, and about storing the rest safely in a replicated time series database you may not grow.

What you'll learn #

The problem, explained #

You are designing one region's slice of a metrics platform:

Three use cases:

Limits: Emit p99 (the latency 99% of requests beat) under 40 ms and 99.99% available; Query p99 under 80 ms and 99.9% available; flushes durable; any single machine may fail; $7,500 a month with the M3DB cluster included.

The tests, in words:

Proschi cannot say "two of three", so model it as the statement says: a majority write is x3 (every replica gets the write), and a replica that is down is x2 plus a failed call (-x).

Back-of-the-envelope #

batches a second into the aggregators
20,000
Uber's ingest-to-stored ratio
25 : 1
replica writes a second after aggregation
2,400
replica writes a second the cluster takes
12k
of the $7,500 already spent on M3DB
$4,200
QuantityArithmeticResult
Ingest batches40,000,000 / 2,00020,000 batches/s
Datapoints after aggregation (25:1, Uber's 500M to 20M)40,000,000 / 251,600,000/s = 800 batch writes/s
Replica writes800 × 32,400/s
Cluster write capacity (partitioned: every replica takes writes)2 shards × 3 replicas × 2k12k/s
Cluster read capacity6 × 5k30k/s
DesignReplica writes/sShare of 12k
Aggregate first800 × 3 = 2,40020%
Store raw20,000 × 3 = 60,000500%

Aggregators. 20k batches a second at about 2k per service replica is 10 replicas at 100%. Keep them comfortably under 70%, and under 100% with one gone.

Budget. The cluster costs 6 × $700 = $4,200 of the $7,500. At $100 a service replica, roughly $3,300 is left: a sensible aggregation tier and a small query tier, not much more.

Deep dive Storage per day (illustrative)

At about 2 bytes per compressed datapoint, 1.6M × 86,400 × 2 bytes ≈ 276 GB a day per copy, about 830 GB with three replicas. Raw, multiply by 25: about 20 TB a day with three replicas. The 2 bytes is an assumption; time series databases compress far better than naive 16-byte timestamp-value pairs.

In the simulation. Above 100% utilisation M3DB saturates and fails the latency limit of every use case touching it, Query included. survive any node failure re-runs with one replica fewer, so the flush load must still fit. A use case is only as available as every tier on its synchronous path, so the short emit path (collector → aggregator, nothing else) is what makes 99.99% reachable. The "Replica down" scenario states that a replica outage does not fail the flush.

Concepts #

Pre-aggregation and downsampling #

Nobody reads most raw datapoints. A dashboard shows "p99 latency of the checkout service per minute", not "latency reported by host 4,812 at second 17". An aggregation tier applies rules as datapoints stream in:

The aggregator keeps open windows in memory, adds each datapoint to one, answers the collector at once, and writes each closed window once per resolution. Host latency stops depending on storage, and writes shrink by the reduction ratio.

Trade-offs. Detail you did not keep is gone (no raw series for one host once the host tag is dropped). An aggregator crash loses its open windows (acceptable here). The rules need maintenance.

When not to use it. Audit logs, billing events, or anything where each record matters; also low-volume systems, where storing raw data is cheap.

title "Aggregate in memory, flush later"

source "Sources"     [Actor]
agg    "Aggregators" [Go]              x4
tsdb   "TSDB"        [NoSQL Database]  x3

source -> agg  : batches
agg    -> tsdb : flush

usecase "Ingest" {
  source -> agg    : POST /write batch
  agg   --> source : 200
}

usecase "Flush" {
  agg   -> tsdb : WRITE rolled-up window
  tsdb --> agg  : ok
}

Quorum writes #

A store with N copies chooses how many acknowledgements a write needs (W) and how many replicas a read consults (R):

Write levelSurvives a replica down?Risk
W = 3 (all)No: one dead or slow replica fails or stalls every write to that shardStrongest, least available
W = 1YesA single disk loss can lose acknowledged data
W = 2 of 3 (majority)YesEvery acknowledged write is on two machines

With majority reads too (R = 2), W + R > N guarantees a read overlaps the latest write. M3DB's read levels include UnstrictMajority, which falls back to fewer replicas when no majority is available.

The client sends to all three and returns once two answer. The third catches up later (a bootstrapping M3DB node fetches missing blocks from its peers).

Trade-offs. A majority write is as slow as the second-fastest replica. Use an odd replica count: a fourth replica raises the majority to three and still tolerates only one failure. When not to use it: a single-leader relational database already serialises writes through one primary; quorums belong to leaderless or per-shard replicated stores.

title "Majority write"

writer "Writer"       [Go]             x2
store  "Replicated DB" [NoSQL Database] x3

writer -> store : write

usecase "Write" {
  alt "All up" {
    writer -> store  : x3 WRITE record
    store --> writer : majority acked
  } alt "One down" {
    writer -> store  : x2 WRITE record
    writer -x store  : WRITE to the third replica
    store --> writer : majority acked
  }
}

A query tier in front of storage #

Queries in PromQL or M3QL are programs: select series by tags, fetch blocks from their shards, compute rates, sums and percentiles. Without a tier in between, every Grafana instance needs the shard layout, a storage client and the query engine.

A query service does it in one place: find the shards, fan out the reads, compute the result. It can also cache results and limit expensive queries to protect storage. Uber's M3 query engine served about 2,500 queries a second at the end of 2018.

When not to add one. A small single-node setup where the database itself speaks the query language (a single Prometheus server) needs no extra tier.

title "Query tier"

ui    "Dashboards"   [Actor]
query "Query Engine" [Go]             x2
store "Series Store" [NoSQL Database] x3

ui    -> query : PromQL over HTTP
query -> store : fetch

usecase "Run query" {
  ui     -> query : GET /api/v1/query_range
  query  -> store : x2 FETCH blocks
  store --> query : datapoints
  query --> ui    : 200
}

Designing it step by step #

Step 1: scope. Confirm the input rate (40M datapoints a second), the batch size, the resolutions and retentions, whether short loss on aggregator failure is acceptable (yes), and that the storage cluster is fixed. That last fact makes the problem "reduce writes by at least 5× and protect them".

Step 2: high-level design. Two flows that share only the store:

Step 3: deep dive.

Routing to aggregators. Shard aggregators by series id so one series always lands on the same instance; otherwise two instances write partial aggregates that collide in storage. (The simulation spreads load evenly and does not model this, but say it.)

Flush semantics. Model the flush as aggregator → M3DB with three replica writes, plus a scenario where one fails and the flush still succeeds. Give "Replica down" a realistic share (a small fraction of a percent).

Sizing. Aggregators for 20k batches a second with headroom and a spare; a query tier of at least two replicas, more if its load needs it. Check M3DB utilisation stays low with one replica gone, and that the bill fits.

What the collectors see. The emit path is collector → aggregator → 200, with nothing else synchronous. Storage hiccups now only delay flushes.

Step 4: wrap-up. Discuss cardinality explosions (a user-id tag creates millions of series), late and out-of-order datapoints, aggregator failover (a standby per shard) and multi-region queries. Note what Proschi does not model: the windows, bytes on disk, compaction, peer streaming.

Common mistakes #

Store raw datapoints (wrong/store-raw-datapoints). Hosts write batches straight into M3DB and a downsampling job compacts later: 60k replica writes a second at a cluster that takes 12k. In Proschi M3DB saturates, so Emit p99 fails, then Query too; the flow tests "Ingest never waits for storage" and "Only aggregates reach storage" fail as well.

Write through on every batch (wrong/write-through-on-every-batch). The aggregator also writes each raw batch to M3DB before answering, "to be safe". Both problems return: hosts wait for a quorum write, storage takes the raw volume. It fails "Ingest never waits for storage" and p99 of Emit metrics < 40 ms.

Wait for all replicas (wrong/wait-for-all-replicas). This is consistency level all: when one replica is down, every flush to its shard fails. Real clusters always have a replica restarting, upgrading or being replaced, so routine maintenance becomes data loss or back pressure. Proschi catches it with "A quorum write survives a replica that is down", because the "Replica down" scenario never succeeds.

Also common: unbounded tag values (user ids, request ids) in metric labels, and treating metrics like logs (keeping every point forever at full resolution).

In the interview #

Likely follow-ups:

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems