System design practice

Durable Job Queue

medium Based on a system published by Slack queues durability availability migration real-world

Slack's job queue: Kafka in front of Redis, so a full Redis loses nothing.

Solve it in your browser Read the lesson first

Slack runs everything too slow for a web request as a background job: posting messages, push notifications, link unfurls, reminders, billing. The web app pushed jobs into Redis, and workers took them from there. Then workers fell behind during a database incident, Redis reached its memory limit and refused new jobs, and the web app could no longer enqueue anything.

The fix: keep Redis and the workers as they are, and put Kafka in front of Redis as a durable buffer.

Functional requirements

Use these use case and scenario names exactly: the traffic, requirements and tests in problem.proschi refer to them.

Scale

Constraints

What is given

problem.proschi declares the web app (four replicas) and the existing workers (eight replicas), and holds the traffic, requirements and tests. Add the gateway, Kafka, the relay, Redis, the connections and the three use cases.

Based on

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 14 min read

Learn it: Durable Job Queue #

Durable Job Queue: putting a log in front of a queue that can fill up #

Slack ran all its background work through one Redis-backed job queue. When workers slowed during a database incident, Redis filled up and refused new jobs. This lesson shows why, and how a durable log in front of Redis fixes it without rewriting the workers.

What you'll learn #

The problem, explained #

Who uses it. Slack's own PHP web app: whenever a request needs slow work done, it enqueues a job and moves on, and workers run it. Nobody outside the company calls this system, but every user feels it when it breaks.

What went wrong. The web app pushed jobs straight into Redis lists and workers popped them off. Redis keeps everything in memory: when workers slowed, jobs piled up, Redis hit its memory limit, and every enqueue failed. Worse, taking a job off the queue also needed a little free memory, so the queue stayed stuck even after the database recovered. A slowdown in one place (the workers' database) became an outage everywhere.

Functional requirements.

Non-functional requirements. Enqueue p99 under 50 ms (99% of enqueues are faster) and 99.99% available; the job durably stored before the web app hears back; any single machine can fail; at most $8,000 a month including the existing web app and workers.

What is given. given.proschi fixes the web app (four replicas) and the workers (eight replicas), which already exist and are not to be rewritten, and the traffic: 33k requests per second for each use case, with 0.1% of relays hitting a full Redis.

What the tests check:

Back-of-the-envelope #

peak jobs, for each use case
33k/s
average on a 1.4B-job day
≈ 16k/s
Redis load (pushes + pops)
66k rps
relays that find Redis full
33/s
Kafka disk per day
≈ 1.4 TB

Start from the published numbers: 1.4 billion jobs on the busiest days, with a peak of 33,000 a second.

QuantityArithmeticResult
Average rate on a busy day1.4B ÷ 86,400 sabout 16k jobs/s
Peak to average33k ÷ 16kabout 2×
Gateway loadevery enqueue is one HTTP call33k rps
Kafka writesone produce per job33k rps
Relay loadone job delivered per relay step33k rps
Redis load33k pushes from the relay + 33k pops from workers66k rps
Relays that find Redis full0.1% × 33k33 per second
Kafka disk per day (assume ~1 KB per job)1.4B × 1 KBabout 1.4 TB/day

The job size is an assumption, but it makes the point: a day of backlog is more than a terabyte (a few once Kafka replicates it). Easy for disks, impossible for the RAM of a single Redis: that is the whole argument for a log in front.

How many replicas?

A service replica ([Go] is a service) handles 2k requests a second, so 33k ÷ 2k = 16.5 replicas would just keep up, saturated. Queueing delay explodes near the limit, and the model flags a node as "hot" above 70%. So divide by 0.7 and round up: 33k ÷ (2k × 0.7) = 23.6, so 24 replicas (69% busy). Then check survive any node failure: with one replica fewer, 23 still run at 72%, well under 100%. The gateway and the relay both carry 33k rps, so each needs 24.

What the simulation does with it. Kafka is a broker node, simulated like a queue with 50k writes per replica, so three brokers are lightly loaded. Redis is a cache with 100k operations per replica and, unlike a relational database in the model, takes writes on every replica. The services are the bottleneck: with the M/M/c queueing formula, a big pool at 70% still answers in about its base 10 ms, a single replica at 70% in more than three times that. Enqueue's p99 (gateway hop plus Kafka hop, with their exponential tails) is about 43 ms when nothing is hot: inside the 50 ms limit, with little room for a hot tier.

Cost. A service replica is $100 a month, a Kafka broker $200, a Redis replica $150. The web app and workers already take $1,200 of the $8,000. Two tiers of 24 add $4,800, three brokers $600 and two Redis replicas $300: $6,900 in all. The budget deliberately forces you to size, not guess.

Availability. The Enqueue path is the gateway, then Kafka. A service replica is up 99.5% of the time, but a pool is up while any replica is, so in the model a large pool is effectively always up. 99.99% is easy once nothing on the path is a single replica.

Concepts #

A durable log as a buffer #

A queue in the loose sense is anything that holds work between a producer and a consumer. Two very different things go by that name:

In-memory queue (a Redis list)Durable log (Kafka, Kinesis, Pulsar)
Where messages liveRAMAppended to disk on several machines
When consumers fall behindIt fills, then must refuse or evict workConsumers read at their own pace; a backlog of hours is normal
StrengthFast and simpleSurvives a slow consumer side

The key property is decoupling under failure: a slow consumer side leaves the producer side working, and the backlog waits on disk.

The trade-offs: a log adds a hop and one more system to operate, and each consumer tracks its own position in the log (the offset: the number of the next message to read). A log is also a poor fit for per-message delays, priorities or retries. Slack's scheduling, execution and de-duplication logic was all built on Redis, which is why it kept Redis behind Kafka rather than pointing workers at the log.

When not to use it: cheap, synchronous work, or volumes small enough for a managed queue with built-in retries (SQS, say).

At-least-once delivery: commit after the side effect #

A log consumer does two things for each message: it performs its side effect (here, pushing into Redis) and it records that it is done (committing the offset, so a restart resumes after that message). The order matters:

Losing jobs is worse than running one twice, so job systems pick at-least-once and make the jobs (or the consuming step) idempotent: running twice has the same effect as once. Kafka's "Message Delivery Semantics" documentation describes exactly this choice.

In Proschi the two outcomes are scenarios. The failed side effect uses -x (a call that never gets an answer), and the scenario still completes without committing:

title "Buffered hand-off"

app    "App"        [REST API] x2
log    "Job Log"    [Kafka]    x3
mover  "Mover"      [Worker]   x2
target "Downstream" [Database] x2

app   -> log    : produce
log   -> mover  : consume
mover -> target : write

usecase "Move" {
  log -> mover : JobCreated offset 42

  alt "Done" {
    mover   -> target : UPSERT job 42
    target --> mover  : ok
    mover  --> log    : commit offset 42
  } alt "Downstream busy" {
    mover  -x target : UPSERT job 42
    mover --> log    : keep offset, retry with backoff
  }
}

Back pressure belongs in the right place #

Back pressure is how a system tells producers to slow down when it is full. The System Design Primer's classic version: bound the queue and answer "busy, try later" when it is full. That is what Redis did, correctly. The mistake was that the web app received the "busy".

The fix moves the pressure boundary: only the relay ever hits a full Redis, and it simply waits, because the job is safe in Kafka.

A stateless gateway in front of a broker #

Kafka clients keep long-lived broker connections, learn which broker leads which partition, and batch messages. A PHP process lives for one request: it would reconnect every time and need to know the broker topology. A small stateless service (Slack's Kafkagate) holds the connections and exposes one HTTP endpoint. Any replica can serve any request, so you scale by adding replicas and lose nothing when one dies.

The cost is one more hop and one more service. It pays off when clients are many, short-lived or lack a good broker client, not when a few long-running services can embed one.

Deep dive In Proschi: produce over HTTP
title "Produce over HTTP"

cli "Short-lived Client" [REST API] x2
gw  "Produce Gateway"    [Go]       x2
bus "Event Log"          [Kafka]    x3

cli -> gw  : HTTP
gw  -> bus : produce

usecase "Publish event" {
  cli  -> gw  : POST /events json {"topic": "audit"}
  gw   -> bus : PRODUCE audit
  bus --> gw  : ack
  gw  --> cli : 202
}

Designing it step by step #

1. Scope. Confirm that "enqueue succeeded" means the job is on replicated disk, not in a process's memory; that workers and Redis stay (the migration constraint); the peak rate (33k/s); and that ordering across jobs is not required. Ask what happens today when Redis is full: enqueues fail. That is the failure you are designing away.

2. High-level design. The minimal chain: web app → something stateless → durable log → something that moves jobs → Redis → workers. Three use cases fall out:

Mention and reject one alternative: workers reading Kafka, with no Redis. It removes a component but rewrites every worker at once and loses Redis queue features they rely on; Only the relay moves jobs from Kafka to Redis forbids it.

3. Deep dive. Three places deserve time.

Deep dive Leader-only acknowledgements: a durability trade-off

Waiting only for the leader, not for all replicas, means a job can be lost if the leader dies before the job is replicated. Say this out loud: it is a real durability trade-off. Proschi does not model replication acknowledgements, so it does not change the numbers here.

4. Wrap-up. Check every requirement: Enqueue p99 is two short hops, no single replica on the path, cost fits, every test's flow holds. Then name what comes next: alert on consumer lag (how far the relay is behind Kafka), cap the relay's push rate so a recovering Redis is not flooded, and a dead-letter topic for jobs that fail repeatedly.

Common mistakes #

Keep enqueueing into Redis (wrong/enqueue-into-redis). The web app still pushes straight into Redis, even with Kafka next to it: the original outage, waiting for the next slowdown. It fails Enqueuing never touches Redis (the web app is connected to a cache) and The web app enqueues through a stateless gateway.

Write to Kafka, then also to Redis before answering (wrong/enqueue-waits-for-redis). The job is in both places, but the answer waits for Redis, so a full Redis fails the enqueue as before, and the relay pushes a duplicate later. It fails Enqueuing never touches Redis.

A relay that commits even when Redis refused (wrong/relay-ignores-full-redis). Redis answers out-of-memory and the relay commits anyway: the job never reached Redis and is gone from Kafka's point of view. Silent data loss, the worst kind. It fails A full Redis leaves the job in Kafka, because no scenario shows a failed call to the cache that the use case survives.

The web app talks to Kafka directly (wrong/web-produces-to-kafka). In production every short-lived PHP request opens broker connections and must know the cluster layout. It fails The web app enqueues through a stateless gateway.

Rewrite the workers to read Kafka (wrong/workers-read-kafka). It deletes a component, but it is a big-bang migration of every job type, and the workers must stay as they are. It fails Only the relay moves jobs from Kafka to Redis.

Other classic mistakes.

In the interview #

Lead with the failure story: it justifies the whole design. Then draw the chain, one sentence per box.

Likely questions, with short answers:

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems