System design practice

Discord Messages

hard Based on a system published by Discord hot-partitions request-coalescing sharding read-heavy real-world

Discord's message store: Rust data services that coalesce reads in front of a hot ScyllaDB partition.

Solve it in your browser Read the lesson first

Discord stores every message ever sent: trillions of them. Messages are partitioned by channel and a time bucket, so a channel's recent messages sit together in one partition, copied to three nodes. That makes the usual read (open a channel, load its latest 50 messages) one cheap query, until a huge server pings everyone. Then hundreds of thousands of people open the same channel within seconds, and every one of their reads lands on the same three replicas: a hot partition. Reads run at quorum, so a hot partition slows every query those nodes serve, not only the busy channel's.

Moving from Cassandra to ScyllaDB helped, but the fix that made hot partitions harmless was a layer in front of the database.

Functional requirements

Name the layer in front of the database data: the tests in problem.proschi refer to it, and to the use case and scenario names above.

Scale

Constraints

What is given

problem.proschi declares the api (the client of this problem), the messages cluster and hot, the busy channel's partition. Both are ScyllaDB, fixed by the problem: you cannot add replicas or shards to them. Add the data services, the connections and the three use cases.

What this model leaves out

Based on

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 12 min read

Learn it: Discord Messages #

Discord messages: taming a hot partition #

What you'll learn #

The problem, explained #

Discord stores every message ever sent, trillions of them, in a wide-column database (Cassandra, later ScyllaDB). Messages are partitioned by channel and a time bucket: one channel's messages from a 10-day window sit together in one partition, copied to three nodes. The common read, "open a channel and show the latest 50 messages", is one cheap query on one partition.

It works until someone in a huge server pings @everyone, and hundreds of thousands of people open the same channel within seconds. Every read targets the same partition, so the same three replicas: a hot partition, a small part of the cluster taking a huge share of the load while the other 69 nodes sit mostly idle. The overloaded nodes slow down for every query they serve, not only the busy channel's: reads run at quorum (each waits for a majority of the replicas, two of three), so other channels' queries often have to wait for one of the slow nodes.

The three use cases:

The given declares api (the Discord API monolith, treated as the client), messages (24 shards of 3 replicas, 72 nodes, fixed) and hot, the three replicas owning the busy channel's partition. hot is a separate node because the simulation spreads load evenly and cannot see one hot key on its own. It takes 10k reads a second per replica and costs nothing extra (its nodes are already paid for in the cluster).

Requirements: p99 of every use case under 60 ms; 99.99% availability; messages durable before the answer; survive any node failure; at most $42,000 a month, the 72 database nodes included. The tests check:

Back-of-the-envelope #

replica operations on a cluster built for ≈ 1.4M
45k / s
reads on hot without coalescing, ≈ 270%
80k / s
reads on hot with coalescing
800 / s
through the data services
60k rps
of the $42,000 is the 72 database nodes
$36,000

Quorum changes the arithmetic. With three replicas, QUORUM is a majority: a read asks two replicas, a write goes to all three and waits for two. So each logical read is two replica reads and each write three replica writes: in Proschi, the x2 and x3 fan-out on the step's label.

FlowLogical rateReplica operations
Send message5k rps15k writes on messages
Read messages15k rps30k reads on messages
Busy channel, no coalescing40k rps80k reads on hot
Busy channel, 1% start a query400 rps800 reads on hot

The cluster is fine on average. 72 nodes at 20k operations each is about 1.4M operations a second; 45k is a few percent. Discord sized it for trillions of messages on disk, not request rate.

The hot partition is not. Its three replicas take 10k reads a second each, 30k in total. Without coalescing they receive 80k, about 270% utilisation: saturated. "Survive any node failure" removes one of the three, leaving 20k. Adding shards or replicas to messages changes nothing: a partition lives on exactly its replication factor's nodes.

With coalescing, only about 1% of the busy reads start a query: 400 a second, 800 replica reads, a few percent of hot.

The data services carry everything: 5k + 15k + 40k = 60k requests a second. At about 2k rps per replica, 30 replicas would be at 100%; well under 70% even after losing one means somewhere in the 40s. At $100 each, with the 72 database nodes at $500 each already costing $36,000, the $42,000 budget leaves room for about 60 replicas at most. The squeeze is deliberate: the design has to be efficient, not just correct.

Latency. A coalesced read is one data-service hop returning a result already on its way: fast. A first read adds one quorum read on an idle hot partition. Sends and ordinary reads are a data-service hop plus a database hop. All fit in 60 ms at p99 if nothing is saturated.

Concepts #

Hot partitions #

Partitioned databases (Cassandra, ScyllaDB, DynamoDB, sharded SQL) spread data by hashing a partition key. That spreads keys evenly, not traffic. If one key is far more popular than the rest, the nodes that own it get far more requests: a hot partition or hot key.

Ways to deal with it, from cheapest to most invasive:

FixHowCost
Absorb the reads before the databaseRequest coalescing, or a cacheCoalescing needs a routing layer; a cache needs invalidation (both below)
Split the keyAdd a time bucket (Discord does, to keep partitions a manageable size) or salt the key into sub-keys, reading all and mergingHelps writes and size, but "latest 50" still targets the newest bucket
Replicate the hot data more widelyRead replicas of just that keyConsistency and complexity

Request coalescing #

Request coalescing: when identical requests arrive while one is already in progress, the later ones wait for the first one's result instead of redoing the work. Go's golang.org/x/sync module ships it as the singleflight package, which keeps only one call per key in flight. Discord's data services do the same for queries: the first request for a channel spawns a worker task that runs the query, later requests for that channel subscribe to it, and when the rows arrive everyone gets them.

Database reads then depend on how many distinct queries are in flight, not how many users asked. During a ping storm, tens of thousands of identical requests a second become a handful of queries.

Two details make it work:

When not to use it: requests whose answers differ per caller (permissions applied to results, personalised data), or writes. And it only helps when identical requests overlap in time; for steady, spread-out reads of popular data, a cache is the tool.

Deep dive Coalescing in Proschi
title "Coalesce identical reads"
client "Client" [Actor]
svc    "Service" [Go] x4 "Shares in-flight queries"
db     "Store"   [Cassandra] x3
client -> svc
svc    -> db : CQL

usecase "Get item" {
  client -> svc : GET /items/7
  alt "Joined" when "a query for item 7 is already running" {
    svc --> client : 200 shared result
  } alt "Leader" when "no query in flight" {
    svc  -> db     : SELECT item 7
    db  --> svc    : row
    svc --> client : 200
  }
}

Why not a cache? #

A cache in front of the database also absorbs repeated reads. Here it is the wrong trade:

Coalescing gives most of the benefit for the burst case (many identical reads at the same instant) without storing anything, which is why the problem forbids a message cache.

Designing it step by step #

1. Scope. Narrow it to the message store: sending, reading recent messages, and the hot-channel case. Confirm the data model (partition by channel and time bucket, replication factor three, quorum reads and writes) and that the cluster is fixed. Ask what happens today during a mass ping: that is the problem statement.

2. High level. The API talks to a new data-service layer, which alone talks to ScyllaDB. Each endpoint is one query, with no business logic. Draw sends and ordinary reads going straight through, and the busy-channel read with its two scenarios.

3. Deep dive.

Count the quorum load. Put x3 on writes and x2 on reads, and show the cluster has plenty of capacity while the hot partition's three replicas do not.

Coalesce. Explain the in-flight map: key = the query and its arguments, value = a task with a list of subscribers. The first request creates it, others subscribe, the result fans out, the entry is removed. Then explain why routing by consistent hash on channel_id makes it work across many instances.

Rule out the alternatives. More nodes do not move the partition. A cache needs invalidation on every edit, delete and reaction. An API monolith querying the database means thousands of processes each sending their own copy of the same read.

Durability. Sends go through the same layer, written at quorum; the sender hears back only after the write.

Sizing and budget. 60k requests a second across the data services: enough replicas to stay well under 70% after losing one, within what the budget leaves after the cluster.

4. Wrap up. Mention what else the layer enables: per-query concurrency limits that protect the database, metrics per query, and one place to change how queries run.

Deep dive What Discord measured

Discord made two changes: the data services, and a migration from Cassandra to ScyllaDB (written in C++, so no garbage-collection pauses). The post's numbers describe the migration: 177 Cassandra nodes became 72 ScyllaDB nodes, the p99 of reading history fell from 40–125 ms to 15 ms, and of inserting from 5–70 ms to a steady 5 ms. The data services are what keep a ping storm from undoing that.

Common mistakes #

The API queries ScyllaDB itself (wrong/api-queries-scylla). The simplest design, and fine for ordinary channels. During a ping storm every client's read goes to the same three replicas, saturating them, and losing one makes it worse. Caught by "Every query goes through the data services", plus the busy channel's p99 and the failure test.

Data services without coalescing (wrong/no-coalescing). The layer is a pass-through, so each request runs its own query and hot still sees 80k reads a second against 30k. A very common real-life half-step: a proxy that adds a hop and fixes nothing. Caught by "Concurrent reads of one channel share one query" (the joined scenario calls a database), and the busy channel's p99 and failure limits.

A cache in front (wrong/cache-in-front). A Redis cache of the latest messages absorbs the reads and passes the latency limits. But every edit, delete and reaction must now keep it in sync, and stale caches show deleted messages. Caught by "No message cache to keep in sync".

Acknowledge before writing (wrong/ack-before-write). The data service answers, then writes; a crash in between loses a message the sender saw as sent. Caught by "A message is stored before the sender hears back" (and the durability requirement).

Classic mistakes beyond the tests:

In the interview #

Likely follow-ups:

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems