Slack ran its cron scripts the classic way: one big machine held every script and a single crontab with all their schedules. As the scripts and the data they processed grew, the machine was moved to bigger and bigger instances, and it stayed a single point of failure: a problem with provisioning, rotation or configuration on that one box stopped every script.
The replacement splits the job in three. A small scheduler service decides what is due and only enqueues it; other copies stand by, and leader election decides which one fires. Slack's existing job queue and its workers run the scripts. A table records every run and its state, so a script whose last run is still going is not started twice, and engineers can see what ran and what failed.
Functional requirements
Fire: every minute the clock ticks, and for each script whose cron string matches, the scheduler hands a run of it to the job queue. It starts at clock. Two scenarios:
"Enqueued": no run of the script is enqueued or in progress, so the scheduler records a new run and enqueues it.
"Already running": the previous run has not finished; the scheduler skips this one and enqueues nothing.
Run script: a worker takes the next run from the queue, marks it in progress, runs the script against Slack's systems (backend), and marks it done.
Name the scheduler scheduler and the worker pool workers: the tests and the workers' capacity in problem.proschi refer to them, and to the use case and scenario names above.
Scale
Slack has not published how many cron scripts it runs or how often; these numbers are assumptions for the exercise.
Most cron strings fire at the start of a minute, so runs come in bursts: 200 runs a second at the top of the minute, about 50 a second averaged over the whole minute. Size for the peak: that is the rate of both use cases.
About 1% of due scripts are still running from last time.
A worker replica completes about 10 runs a second. Most scripts are short; a few run for up to an hour.
Constraints
The scheduler never runs a script: no connection from the scheduler to backend, and the scripts run on the worker pool.
Every run is recorded in a durable database before it is enqueued, and the worker updates it before and after the script.
p99 of Fire under 100 ms.
p99 of Run script under 110 ms. In this model that is the worker's own handling of a run, not the script's run time; it climbs when the pool is too busy, so it stands for "a due script starts without waiting for a free worker".
Fire available 99.9% of the time.
Losing any single machine must not stop scripts from being scheduled or run.
At most $5,500 / month; backend costs nothing here.
What is given
problem.proschi declares the clock (the minute tick that starts every Fire) and backend (the databases and APIs the scripts work on, already running), the workers' capacity, and holds the traffic, requirements and tests. Add the scheduler, the job queue, the worker pool, the run-state database, the connections and the two use cases.
What this model leaves out
No overlapping runs is behaviour the simulation cannot check: it does not know which script a run belongs to or how long it lasts. The problem encodes it as a flow: the run is recorded before it is enqueued, and the "Already running" scenario never reaches the queue. Its 1% share is the scenario mix, not something the lookup earns.
Leader election is not simulated either. The model spreads Fire over both scheduler pods, where in the real service one leads and the other waits; the second pod counts here as the standby that takes over when the first is lost.
How long a script holds a worker (seconds to an hour) is folded into the per-replica rate of 10 runs a second, and the scripts' own run time is not part of any latency.
Retries of failed runs and the page that shows each run's state are not modelled.
Based on
Claire Adams, Executing Cron Scripts Reliably At Scale, Slack Engineering, September 2023: the single cron box with one crontab that kept moving to larger machines and was a single point of failure; the Scheduled Job Conductor, a Go service on Bedrock (Slack's Kubernetes wrapper) that keeps the cron string format, runs several pods with Kubernetes leader election so one schedules while the others stand by, and only schedules, leaving the CPU- and memory-heavy work to the job queue; one job per script on Slack's job queue (Kafka, then Redis, then a job worker); and a Vitess table with a row per run (enqueued, in progress, done), queried by an index on the script name before a run starts, which replaces the old flock and backs a page showing each run's state and errors, since some scripts run for up to an hour.
How your design is checked
You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.
Cron is the oldest scheduler there is: a crontab line such as */5 * * * * says "run this every five minutes", and the cron daemon on that machine starts the script when the minute matches. For a handful of scripts on one server it is perfect.
Slack ran its scheduled scripts this way: one node held every script and one crontab with every schedule. As the scripts and their data grew, the node moved to ever larger machines, and any problem with that one box (provisioning, rotation, configuration) stopped every script at once. Its replacement, described in 2023, has three parts: a scheduler, the company's existing job queue, and a database table of runs.
You build the same shape, in two use cases:
Fire starts at the clock, the minute tick. The scheduler looks up the script's active runs; in "Enqueued" there are none, so it records a new run and enqueues it; in "Already running" the last run has not finished and nothing is enqueued.
Run script starts at the queue: a worker takes the run, marks it in progress, does the work against backend (Slack's systems), and marks it done.
The tests check that the scheduler never reaches backend, that a run is inserted (INSERT) into a database before it is enqueued and updated (UPDATE) before and after the script runs, and that the scheduler has a standby.
The requirements add numbers: p99 of Fire under 100 ms and of Run script under 110 ms (p99 is the latency 99% of requests beat), Fire available 99.9% of the time, no single machine whose loss stops scheduling or running, and at most $5,500 a month. Run script's limit covers the worker's handling of a run, not the script's own run time: it climbs when the pool is too busy, so it stands for "a due script starts without waiting for a free worker".
The scale numbers are assumptions: Slack did not publish how many scripts it runs.
runs at the top of the minute
200 / s
averaged over the minute
50 / s
runs per worker replica
10 / s
workers at 100% busy
20
workers with headroom and a spare
30
Bursts, not averages. Cron strings have minute resolution, so every script due in a minute is due at its first second, and round times like 0 * * * * (every hour, on the hour) pile up. Here: 200 runs a second at peak against 50 over the whole minute, a 4× burst.
The pool. A worker replica completes about 10 runs a second (most scripts are short; how long each holds a worker is folded into that rate). At the peak:
Workers
Busy at 200 runs/s
One lost
8 (sized for the average)
250%
286%
20
100%
105%
22
91%
95%
30
67%
69%
Twenty is the bare minimum and saturates the moment one machine goes. Queueing delay climbs steeply as a pool nears 100%, so aim for under 70% busy: 200 ÷ (10 × 0.7) = 28.6, so 29 workers, and 30 stays under 70% even with one lost.
The run table. Every Fire reads the script's active runs and, when it enqueues, inserts a row: 200 reads and about 200 writes a second. Every run updates its row twice: 400 more writes. Six hundred writes a second is nothing for a primary that takes about 5,000; two replicas are for availability, not load.
The budget. Two scheduler pods ($200), a three-broker queue ($600), two database replicas ($800) and 30 workers ($3,000) come to about $4,600 of $5,500. Forty workers would not fit.
A single cron machine is both the clock that knows what is due and the engine that runs it, so losing it loses both, with every run in progress. A larger box fixes running out of memory or CPU, not that: it is still one machine, and slower to replace.
A second, identical box is no answer either: two daemons reading one crontab fire every script twice. Something has to decide which copy fires.
The two halves need opposite things. Scheduling is tiny: parse a few thousand cron strings, compare them with the clock once a minute, and send out messages. It must be always up and on time. Running scripts is heavy, varied and slow: some take a second, some an hour, some need lots of memory. It needs many machines and does not care which one a script lands on.
So the scheduler does nothing but decide and hand off. It writes each due run to a queue, and a pool of workers reads the queue and runs the scripts (the competing-consumers pattern: each message goes to one worker, not to all of them). Slack did not even build the second half: its job queue (Kafka for durability, Redis for the jobs being worked on, then job workers) already ran billions of jobs a day, and each cron run became one more job.
The scheduler stays small, so it fails over in seconds and a heavy script cannot delay the next minute. Losing a worker loses only the runs on it. The pool scales with the scripts without touching the scheduler. The cost: a run now waits in a queue, and its state is spread over three machines, which is why the next two concepts exist.
The scheduler must not be a new single point of failure, yet two schedulers firing at once would enqueue every script twice. The usual answer is leader election: run two or more copies, and let them agree that exactly one, the leader, fires. The others are standbys that watch the leader and take over if it stops renewing its claim.
Slack runs several scheduler pods on Kubernetes and uses Kubernetes leader election, which is built on a lease: the leader holds a record that expires unless renewed every few seconds. If the leader dies, the lease runs out and a standby takes it. This is active-passive: the standby does no scheduling until it leads. Slack decided against keeping several schedulers in sync: a new leader takes over quickly enough that downtime is unlikely, and with the scripts moved off, a pod has only scheduling to do.
The trade-off: a failover takes a few seconds, and runs due in that window fire late (or, if the scheduler does not catch up, are skipped). Timing matters: losing the leader just before the top of the minute is the worst moment, so Slack keeps a scheduler pod from going down then when it can.
Deep dive Why leases and not a lock held forever
A lock held until released never frees itself when the leader crashes or is cut off; someone would have to prove the leader dead. A lease expires on its own. Its weakness is a leader that pauses (a long garbage collection, a frozen VM) and wakes after its lease ran out, still believing it leads, so two schedulers briefly fire together. A run table that refuses a second active run limits the damage; fencing tokens are the general fix.
On one box, "don't start this script if it is still running" is a file lock: the script takes a flock on a file, and a second copy that cannot get it exits. Slack's old cron did exactly that. It stops working the moment runs happen on different machines, because each machine has its own files.
So the state moves to a place everyone can see: a table with one row per run. Slack used a Vitess table (Vitess is sharded MySQL). A run's row moves through enqueued, in progress and done:
On Fire, the scheduler queries the script's active runs. An index on the script name keeps this lookup cheap.
If one is enqueued or in progress, the scheduler skips this run ("Already running").
Otherwise it inserts a row with state enqueued, then enqueues the run. Writing first means a crash between the two leaves a visible row, not an invisible job.
The worker sets the row to in progress before the script and to done (or an error) after it.
The same table answers the question engineers ask most: is that hour-long script still running, or did it fail? Slack put a small page on top of it showing each run's state and errors.
Why a database and not Redis or the scheduler's memory? The state must outlive any one machine. If it lived in the scheduler, a failover would forget every run in progress and start them again. If it lived in a cache, a restart would do the same. A durable, replicated store keeps it through both.
The rate that matters is the peak: the top of the minute. Divide it by what one worker completes, then add headroom so the pool is not running hot, and one spare so that losing a machine does not tip it over:
Little's law is the same idea from the other side: the number of runs in flight equals the arrival rate times how long each run takes, and every run in flight needs a slot on some worker.
A pool sized for the average fills up at every burst. Runs then wait in the queue, scripts start late, and the backlog grows each minute if the burst never fully drains. In the simulation, the workers saturate and the p99 of Run script climbs past its limit.
Start from the use cases. Fire goes from clock to the scheduler and ends at the queue. Run script starts at the queue and ends with the worker acknowledging the job. Write the steps before choosing replica counts.
Add the scheduler, and only make it schedule. One service, id scheduler, connected to the run table and the queue, and nothing else. No connection to backend: if the scheduler can reach it, someone will eventually run a script there.
Add the queue. Any durable queue or broker works. It is what lets the scheduler answer in milliseconds while a run waits for a worker.
Add the run table. A relational database, two replicas. In Fire, look up active runs first, then branch: "Enqueued" inserts the row and then enqueues; "Already running" answers without touching the queue.
Add the worker pool, id workers. In Run script: update the row to in progress, call backend for the script's work, update the row to done, acknowledge the job.
Add the standby. Two scheduler replicas. Write down in a decision that one leads and one waits.
Size and check. Workers for the peak plus a spare, then run the tests and read the cost.
The scheduler runs the scripts itself. Two scheduler pods with leader election, but the leader executes every due script in-process: the old cron box, doubled. Losing the leader kills every run on it, and a memory-hungry script can make the next minute late. It fails The scheduler only enqueues and Scripts run on the worker pool. Enqueue, and let the workers run.
One scheduler, no standby. Moving the scripts to workers removed one single point of failure and kept another: while the lone scheduler restarts, nothing fires. Fire's availability drops to the scheduler's own 99.5%, below 99.9%, and A standby scheduler is ready to take over fails. Run two and let leader election pick.
No run table. The scheduler enqueues whatever is due, every time. A script that runs longer than its interval gets a second copy, then a third, and nobody can tell whether last night's run finished. It fails Every run is recorded before it is enqueued. Keeping the state in a cache instead fails the same test: it must survive a restart.
A pool sized for the average. Eight workers handle 50 runs a second, the minute's average, and saturate at 250% when 200 arrive at once. It fails the p99 of Run script and surviving a node failure. Size for the burst.
Doubling the pool to be safe. Forty workers pass every test but cost $5,600 with the rest, over the budget. Headroom is one spare and 70% busy, not double.
Open with the failure that motivates the design: "one cron box is a single point of failure, and two of them run everything twice." Then split scheduling from running, and say why each half needs different things. Expect the interviewer to push on the corners:
Have two numbers ready: the peak-to-average ratio (4×), and workers = peak ÷ per-worker rate (20 at full load, 30 with headroom and a spare).
Claire Adams, Executing Cron Scripts Reliably At Scale (Slack Engineering, 2023): the single cron box, the Scheduled Job Conductor with Kubernetes leader election, the job queue and the Vitess table of runs.
Saroj Yadav, Matthew Smillie, Mike Demmer and Tyler Johnson, Scaling Slack's Job Queue (Slack Engineering, 2017): the job queue the cron runs are handed to.
Štěpán Davidovič and Kavita Guliani, Distributed Periodic Scheduling with Cron, in Site Reliability Engineering (Google): how Google's distributed cron uses a replicated leader and records launches to avoid double runs.