System design practice

Notification Fan-out

medium queues async external-providers failover

Queue it, check preferences, fail over between providers.

Solve it in your browser Read the lesson first

The Order Service wants to tell customers when their order ships, is delayed or arrives. Build the notification system it hands these events to: it picks the channel each customer wants and sends the message through an external email, SMS or push provider.

Functional requirements

Use these use case and scenario names exactly: the traffic, requirements and tests in problem.proschi refer to them.

Scale

Constraints

What is given

problem.proschi declares the orders service (the caller), the existing prefs service (each user's channel and opt-outs) and the providers email, sms, smsBackup and push, with their rate limits. It also holds the traffic, requirements and tests. Add the components, the connections and the two use cases.

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 12 min read

Learn it: Notification Fan-out #

Notification Fan-out: accept fast, deliver patiently #

"Tell the customer their order shipped" sounds like one line of code: call the email API. Then the email provider has a bad afternoon, and every order update takes seconds or fails. Here you design a notification system that takes events from the Order Service in milliseconds, picks the right channel, respects opt-outs and survives a provider outage.

What you'll learn #

The problem, explained #

Who uses it. The Order Service, an internal caller, emits events like OrderShipped to be turned into an email, SMS or push notification, whichever the customer prefers.

Functional requirements.

Non-functional requirements.

What is given, and why. given.proschi declares the caller (orders, four replicas), the existing Preferences service (prefs, three replicas) and four providers: email, sms, smsBackup and push. Their capacity limits (2k, 500, 500 and 10k requests a second) stand for real rate limits. They are external: free in the model, about 200 ms per call, 99.9% available.

What the tests check.

Back-of-the-envelope #

events at peak, accepted and delivered once
1,500/s
email sends, about 40% of the 2k limit
825 rps
provider calls in flight
≈ 300
events queued after a 10-minute total outage
900,000
of the $2,000 already spent
$700

Peak is 1,500 events a second, each accepted and delivered once, split 55% email, 30% push, 11% SMS (10% by the primary, 1% failing over to the backup) and 4% opted out.

QuantityArithmeticResult
Queue writes (Notify)1,500 events/s1,500 rps
Preference lookupsevery delivered event1,500 rps
Email sends55% × 1,500825 rps (limit 2k)
Push sends30% × 1,500450 rps (limit 10k)
SMS sends10% × 1,500150 rps, plus 15 calls that time out (limit 500)
SMS failover: backup sends1% × 1,50015 rps (limit 500)
Opted out: nothing sent4% × 1,50060 events/s
Provider calls in flight1,500/s × ~0.2 s (Little's law, below)about 300 at once
Backlog if all providers are down for 10 minutes1,500/s × 600 s900,000 events

Rate limits. Every channel is under its provider's limit at peak; email is the closest, at about 40%. If the Order Service also called a provider inline, that load would come on top and could push email over its limit.

Concurrency. Little's law: requests in flight = arrival rate × time in the system. With 200 ms provider calls, workers hold about 300 calls open (a little fewer, since the 4% opted out call nobody), so they need asynchronous I/O or a big enough pool. Proschi does not model a worker waiting on a slow dependency (a service replica is one server with a 10 ms service time): point this out yourself.

Backlog. A ten-minute outage of everything leaves under a million small messages queued. Queues keep messages for days (SQS for up to 14), so "never lose an accepted event" comes from the queue, not from heroics in the workers.

Sizing the parts you add. A queue replica takes 50k writes a second, so two are lightly loaded and redundant. A service replica takes 2k requests a second against the worker's 1,500 events a second: divide by your target utilisation, and stay under 100% with one replica lost.

Latency in the model. Notify is the Order Service writing the queue, a 5 ms hop: lots of room under 50 ms, unless a provider is on the path, since one call is already 200 ms.

Deep dive Why a provider's tail must stay off the caller's path

Deliver has no latency requirement; the providers decide its p99. A provider is a single server in the model, so at 41% busy the email provider takes around 340 ms on average because of queueing: exactly the slow tail you keep off your caller's path.

Availability. Notify's availability is the queue's, and two replicas of a 99.99% queue are effectively always up in the model. Deliver's is capped by the providers (99.9% each), which is fine: the queue holds events until delivered.

Cost. The given services cost $700 a month ($100 per service replica); a queue replica is $200. There is room for a sensible design, not for doubling every tier.

Concepts #

Queue-based load levelling #

Put a durable queue between a fast caller and slow work. The caller's request ends when the queue stores the message; workers drain it at the pace the providers allow. The queue absorbs bursts (load levelling) and outages (buffering).

Trade-offs: delivery is asynchronous, so the caller only knows the email will be attempted, and you must monitor queue depth and message age. When not to use it: when the caller truly needs the result now. (A one-time password the user awaits may still go through a queue, with a priority lane and tight alerting.)

title "Accept now, work later"

caller "Caller"   [REST API]        x2
inbox  "Inbox"    [AWS SQS]         x2
worker "Worker"   [Worker]          x2
slow   "Slow API" [Third Party API]

caller -> inbox  : SendMessage
inbox  -> worker : deliver
worker -> slow   : call

usecase "Accept" {
  caller -> inbox  : SEND ReportRequested {"id": 7}
  inbox --> caller : 200 queued
}

usecase "Work" {
  inbox   -> worker : ReportRequested {"id": 7}
  worker  -> slow   : POST /render
  slow   --> worker : 200
  worker --> inbox  : delete
}

At-least-once delivery and idempotency keys #

Queues like SQS deliver at least once: every message arrives, and a few arrive more than once. A message is hidden for a visibility timeout while a worker handles it, and deleted only when the worker says so; if the worker crashes, it reappears for another worker. Occasionally a message arrives twice even without a crash, so the SQS documentation says to make consumers idempotent (safe to run twice).

Here a duplicate means a customer gets the same text twice. The defence is an idempotency key: a unique id (the event id) that the worker passes to the provider or records itself, so a repeated send does nothing. Stripe's write-up on idempotency explains the pattern for APIs in general.

Someone must store the keys for a while: the provider, if it accepts idempotency keys, or a small table or cache of recently sent event ids.

Provider failover and circuit breakers #

A third party will fail. For channels with alternatives (SMS has many vendors), the worker tries the primary and, on a timeout or error, sends through a backup. A circuit breaker makes this cheap: after enough failures it skips the primary for a while, so you do not pay a timeout on every message.

In Proschi a failed call is -x: a timeout (1,000 ms by default) and no answer. A scenario that calls a node with -x and still succeeds through another node is a fallback, what survive failure of … and handles failure of … look for.

Deep dive In Proschi: two providers
title "Two providers"

jobs    "Address Events"  [AWS SQS]         x2
app     "App"             [Worker]          x2
primary "Geocoder"        [Third Party API]
backup  "Backup Geocoder" [Third Party API]

jobs -> app     : deliver
app  -> primary : lookup
app  -> backup  : lookup

usecase "Geocode" {
  jobs -> app : AddressAdded
  alt "Primary" {
    app      -> primary : GET /geocode
    primary --> app     : 200
  } alt "Failover" when "the primary times out" {
    app     -x primary : GET /geocode
    app     -> backup  : GET /geocode
    backup --> app     : 200
  }
}

Do not fail over when both providers could act (a payment captured twice) or when the backup is much worse than a short delay: retry later from the queue instead.

Designing it step by step #

1. Scope. Clarify the channels (email, SMS, push), who picks one (the user's preferences), whether one event can produce several notifications (no, at most one), the peak rate, and the promise to the caller ("accepted" means stored, not sent). Ask about opt-outs: for legal and ethical reasons, they are honoured before anything goes out.

2. High-level design. Two flows, one boundary between them:

Name and reject the alternative, the Order Service calling providers itself: it couples order processing to the worst provider's latency and availability, which the requirements forbid.

3. Deep dive.

4. Wrap-up. Check: Notify is one queue write; no path from the Order Service to a provider; the failover scenario covers an SMS outage; every node you run has two or more replicas; the bill fits. Then extend: priority queues for urgent messages, per-user rate limits so nobody gets twenty texts in a minute, a dead-letter queue for events that keep failing, and provider delivery receipts to track what arrived.

Common mistakes #

A notifier that queues and then sends inline (wrong/notifier-calls-provider-inline). The notifier writes the queue, then calls the email provider before answering. The caller still waits for a 200 ms third party, and every provider outage becomes an order-processing outage. In the model it is worse: the inline sends come on top of the worker's, the email provider goes over its 2k rps limit, and Notify's p99 runs to many seconds. It fails The Order Service never waits for a provider, and also Notify's p99 and availability limits.

A worker that polls the queue (wrong/worker-polls-queue). Deliver starts with the worker sending ReceiveMessage to the queue. Common with SQS, but in this model it reads as the worker writing to the queue, and the flow no longer starts at the queue. It fails Workers take events from the queue. Model the queue handing the event over instead.

Other classic mistakes.

In the interview #

Give the core idea, draw the queue as the boundary, then walk one event through each scenario.

Follow-ups you should expect:

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems