System design practice

Distributed Cron

medium Based on a system published by Slack scheduling queues availability leader-election real-world

Slack's cron: a leader-elected scheduler that only enqueues, a worker pool and a table of runs.

Solve it in your browser Read the lesson first

Slack ran its cron scripts the classic way: one big machine held every script and a single crontab with all their schedules. As the scripts and the data they processed grew, the machine was moved to bigger and bigger instances, and it stayed a single point of failure: a problem with provisioning, rotation or configuration on that one box stopped every script.

The replacement splits the job in three. A small scheduler service decides what is due and only enqueues it; other copies stand by, and leader election decides which one fires. Slack's existing job queue and its workers run the scripts. A table records every run and its state, so a script whose last run is still going is not started twice, and engineers can see what ran and what failed.

Functional requirements

Name the scheduler scheduler and the worker pool workers: the tests and the workers' capacity in problem.proschi refer to them, and to the use case and scenario names above.

Scale

Slack has not published how many cron scripts it runs or how often; these numbers are assumptions for the exercise.

Constraints

What is given

problem.proschi declares the clock (the minute tick that starts every Fire) and backend (the databases and APIs the scripts work on, already running), the workers' capacity, and holds the traffic, requirements and tests. Add the scheduler, the job queue, the worker pool, the run-state database, the connections and the two use cases.

What this model leaves out

Based on

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 13 min read

Learn it: Distributed Cron #

Distributed cron: no single box to lose #

What you'll learn #

The problem, explained #

Cron is the oldest scheduler there is: a crontab line such as */5 * * * * says "run this every five minutes", and the cron daemon on that machine starts the script when the minute matches. For a handful of scripts on one server it is perfect.

Slack ran its scheduled scripts this way: one node held every script and one crontab with every schedule. As the scripts and their data grew, the node moved to ever larger machines, and any problem with that one box (provisioning, rotation, configuration) stopped every script at once. Its replacement, described in 2023, has three parts: a scheduler, the company's existing job queue, and a database table of runs.

You build the same shape, in two use cases:

The tests check that the scheduler never reaches backend, that a run is inserted (INSERT) into a database before it is enqueued and updated (UPDATE) before and after the script runs, and that the scheduler has a standby.

The requirements add numbers: p99 of Fire under 100 ms and of Run script under 110 ms (p99 is the latency 99% of requests beat), Fire available 99.9% of the time, no single machine whose loss stops scheduling or running, and at most $5,500 a month. Run script's limit covers the worker's handling of a run, not the script's own run time: it climbs when the pool is too busy, so it stands for "a due script starts without waiting for a free worker".

Back-of-the-envelope #

The scale numbers are assumptions: Slack did not publish how many scripts it runs.

runs at the top of the minute
200 / s
averaged over the minute
50 / s
runs per worker replica
10 / s
workers at 100% busy
20
workers with headroom and a spare
30

Bursts, not averages. Cron strings have minute resolution, so every script due in a minute is due at its first second, and round times like 0 * * * * (every hour, on the hour) pile up. Here: 200 runs a second at peak against 50 over the whole minute, a 4× burst.

The pool. A worker replica completes about 10 runs a second (most scripts are short; how long each holds a worker is folded into that rate). At the peak:

WorkersBusy at 200 runs/sOne lost
8 (sized for the average)250%286%
20100%105%
2291%95%
3067%69%

Twenty is the bare minimum and saturates the moment one machine goes. Queueing delay climbs steeply as a pool nears 100%, so aim for under 70% busy: 200 ÷ (10 × 0.7) = 28.6, so 29 workers, and 30 stays under 70% even with one lost.

The run table. Every Fire reads the script's active runs and, when it enqueues, inserts a row: 200 reads and about 200 writes a second. Every run updates its row twice: 400 more writes. Six hundred writes a second is nothing for a primary that takes about 5,000; two replicas are for availability, not load.

The budget. Two scheduler pods ($200), a three-broker queue ($600), two database replicas ($800) and 30 workers ($3,000) come to about $4,600 of $5,500. Forty workers would not fit.

Concepts #

The cron box as a single point of failure #

A single cron machine is both the clock that knows what is due and the engine that runs it, so losing it loses both, with every run in progress. A larger box fixes running out of memory or CPU, not that: it is still one machine, and slower to replace.

A second, identical box is no answer either: two daemons reading one crontab fire every script twice. Something has to decide which copy fires.

Separate scheduling from running #

The two halves need opposite things. Scheduling is tiny: parse a few thousand cron strings, compare them with the clock once a minute, and send out messages. It must be always up and on time. Running scripts is heavy, varied and slow: some take a second, some an hour, some need lots of memory. It needs many machines and does not care which one a script lands on.

So the scheduler does nothing but decide and hand off. It writes each due run to a queue, and a pool of workers reads the queue and runs the scripts (the competing-consumers pattern: each message goes to one worker, not to all of them). Slack did not even build the second half: its job queue (Kafka for durability, Redis for the jobs being worked on, then job workers) already ran billions of jobs a day, and each cron run became one more job.

The scheduler stays small, so it fails over in seconds and a heavy script cannot delay the next minute. Losing a worker loses only the runs on it. The pool scales with the scripts without touching the scheduler. The cost: a run now waits in a queue, and its state is spread over three machines, which is why the next two concepts exist.

Leader election and a standby #

The scheduler must not be a new single point of failure, yet two schedulers firing at once would enqueue every script twice. The usual answer is leader election: run two or more copies, and let them agree that exactly one, the leader, fires. The others are standbys that watch the leader and take over if it stops renewing its claim.

Slack runs several scheduler pods on Kubernetes and uses Kubernetes leader election, which is built on a lease: the leader holds a record that expires unless renewed every few seconds. If the leader dies, the lease runs out and a standby takes it. This is active-passive: the standby does no scheduling until it leads. Slack decided against keeping several schedulers in sync: a new leader takes over quickly enough that downtime is unlikely, and with the scripts moved off, a pod has only scheduling to do.

The trade-off: a failover takes a few seconds, and runs due in that window fire late (or, if the scheduler does not catch up, are skipped). Timing matters: losing the leader just before the top of the minute is the worst moment, so Slack keeps a scheduler pod from going down then when it can.

Deep dive Why leases and not a lock held forever

A lock held until released never frees itself when the leader crashes or is cut off; someone would have to prove the leader dead. A lease expires on its own. Its weakness is a leader that pauses (a long garbage collection, a frozen VM) and wakes after its lease ran out, still believing it leads, so two schedulers briefly fire together. A run table that refuses a second active run limits the damage; fencing tokens are the general fix.

Tracking runs in a database #

On one box, "don't start this script if it is still running" is a file lock: the script takes a flock on a file, and a second copy that cannot get it exits. Slack's old cron did exactly that. It stops working the moment runs happen on different machines, because each machine has its own files.

So the state moves to a place everyone can see: a table with one row per run. Slack used a Vitess table (Vitess is sharded MySQL). A run's row moves through enqueued, in progress and done:

  1. On Fire, the scheduler queries the script's active runs. An index on the script name keeps this lookup cheap.
  2. If one is enqueued or in progress, the scheduler skips this run ("Already running").
  3. Otherwise it inserts a row with state enqueued, then enqueues the run. Writing first means a crash between the two leaves a visible row, not an invisible job.
  4. The worker sets the row to in progress before the script and to done (or an error) after it.

The same table answers the question engineers ask most: is that hour-long script still running, or did it fail? Slack put a small page on top of it showing each run's state and errors.

Why a database and not Redis or the scheduler's memory? The state must outlive any one machine. If it lived in the scheduler, a failover would forget every run in progress and start them again. If it lived in a cache, a restart would do the same. A durable, replicated store keeps it through both.

Sizing the worker pool #

The rate that matters is the peak: the top of the minute. Divide it by what one worker completes, then add headroom so the pool is not running hot, and one spare so that losing a machine does not tip it over:

Little's law is the same idea from the other side: the number of runs in flight equals the arrival rate times how long each run takes, and every run in flight needs a slot on some worker.

A pool sized for the average fills up at every burst. Runs then wait in the queue, scripts start late, and the backlog grows each minute if the burst never fully drains. In the simulation, the workers saturate and the p99 of Run script climbs past its limit.

Designing it step by step #

  1. Start from the use cases. Fire goes from clock to the scheduler and ends at the queue. Run script starts at the queue and ends with the worker acknowledging the job. Write the steps before choosing replica counts.
  2. Add the scheduler, and only make it schedule. One service, id scheduler, connected to the run table and the queue, and nothing else. No connection to backend: if the scheduler can reach it, someone will eventually run a script there.
  3. Add the queue. Any durable queue or broker works. It is what lets the scheduler answer in milliseconds while a run waits for a worker.
  4. Add the run table. A relational database, two replicas. In Fire, look up active runs first, then branch: "Enqueued" inserts the row and then enqueues; "Already running" answers without touching the queue.
  5. Add the worker pool, id workers. In Run script: update the row to in progress, call backend for the script's work, update the row to done, acknowledge the job.
  6. Add the standby. Two scheduler replicas. Write down in a decision that one leads and one waits.
  7. Size and check. Workers for the peak plus a spare, then run the tests and read the cost.

Common mistakes #

In the interview #

Open with the failure that motivates the design: "one cron box is a single point of failure, and two of them run everything twice." Then split scheduling from running, and say why each half needs different things. Expect the interviewer to push on the corners:

Have two numbers ready: the peak-to-average ratio (4×), and workers = peak ÷ per-worker rate (20 at full load, 30 with headroom and a spare).

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems