The Order Service wants to tell customers when their order ships, is delayed or arrives. Build the notification system it hands these events to: it picks the channel each customer wants and sends the message through an external email, SMS or push provider.
Functional requirements
Notify: the Order Service hands over an event for a user (OrderShipped, …) and gets an answer as soon as the event is safely accepted.
Deliver: a queue hands an accepted event to a worker (the use case starts with a step sent by the queue). The user's preferences decide the channel; each event becomes at most one notification. Model five scenarios:
"Email", "Push", "SMS": the user prefers that channel and the notification goes out through its provider.
"SMS failover": the SMS provider does not answer; the message goes out through the backup SMS provider instead.
"Opted out": the user turned these notifications off; nothing is sent.
Use these use case and scenario names exactly: the traffic, requirements and tests in problem.proschi refer to them.
Scale
1,500 events per second at peak (evening delivery rounds), each one handed over once and delivered once.
55% of events go to users who prefer email, 30% push and 11% SMS; 4% are opted out. The SMS provider times out on about 1% of all events (one SMS in eleven), so 10% go out through it and 1% fail over.
Providers are slow and flaky: about 200 ms per call when they work, and they have outages.
Constraints
The Order Service must never wait for a provider: handing over an event takes under 50 ms at p99.
Accepting events available 99.95% of the time.
An accepted event is never lost, even if every provider is down for a while.
Losing any single machine, or the SMS provider, must not stop notifications.
At most $2,000 / month, the Order Service and the Preferences service included.
What is given
problem.proschi declares the orders service (the caller), the existing prefs service (each user's channel and opt-outs) and the providers email, sms, smsBackup and push, with their rate limits. It also holds the traffic, requirements and tests. Add the components, the connections and the two use cases.
How your design is checked
You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.
"Tell the customer their order shipped" sounds like one line of code: call the email API. Then the email provider has a bad afternoon, and every order update takes seconds or fails. Here you design a notification system that takes events from the Order Service in milliseconds, picks the right channel, respects opt-outs and survives a provider outage.
Who uses it. The Order Service, an internal caller, emits events like OrderShipped to be turned into an email, SMS or push notification, whichever the customer prefers.
Functional requirements.
Notify: the Order Service hands over an event and gets an answer as soon as the event is safely accepted.
Deliver: a queue hands an accepted event to a worker, which reads the user's preferences and sends at most one notification. Five scenarios: "Email", "Push", "SMS", "SMS failover" (the SMS provider does not answer, the backup sends it) and "Opted out" (nothing is sent).
Non-functional requirements.
Hand-over p99 under 50 ms (99% are faster); the Order Service never waits for a provider.
Accepting events is 99.95% available.
An accepted event is never lost, even if every provider is down for a while.
Losing any single machine, or the SMS provider, does not stop notifications.
At most $2,000 a month, including the given Order Service and Preferences service.
What is given, and why.given.proschi declares the caller (orders, four replicas), the existing Preferences service (prefs, three replicas) and four providers: email, sms, smsBackup and push. Their capacity limits (2k, 500, 500 and 10k requests a second) stand for real rate limits. They are external: free in the model, about 200 ms per call, 99.9% available.
What the tests check.
The Order Service never waits for a provider: Notify never waits for an external system, writes a queue before answering, and orders has no connection to a provider.
Workers take events from the queue: Deliver starts at a queue.
Preferences are checked before anything is sent: Deliver calls prefs before any provider, and the opted-out scenario never calls a provider.
Each channel goes out through its provider: Email calls email, Push calls push, SMS calls sms.
SMS fails over to the backup provider: Deliver survives a failed call to sms, and the failover scenario calls smsBackup after sms.
Peak is 1,500 events a second, each accepted and delivered once, split 55% email, 30% push, 11% SMS (10% by the primary, 1% failing over to the backup) and 4% opted out.
Quantity
Arithmetic
Result
Queue writes (Notify)
1,500 events/s
1,500 rps
Preference lookups
every delivered event
1,500 rps
Email sends
55% × 1,500
825 rps (limit 2k)
Push sends
30% × 1,500
450 rps (limit 10k)
SMS sends
10% × 1,500
150 rps, plus 15 calls that time out (limit 500)
SMS failover: backup sends
1% × 1,500
15 rps (limit 500)
Opted out: nothing sent
4% × 1,500
60 events/s
Provider calls in flight
1,500/s × ~0.2 s (Little's law, below)
about 300 at once
Backlog if all providers are down for 10 minutes
1,500/s × 600 s
900,000 events
Rate limits. Every channel is under its provider's limit at peak; email is the closest, at about 40%. If the Order Service also called a provider inline, that load would come on top and could push email over its limit.
Concurrency.Little's law: requests in flight = arrival rate × time in the system. With 200 ms provider calls, workers hold about 300 calls open (a little fewer, since the 4% opted out call nobody), so they need asynchronous I/O or a big enough pool. Proschi does not model a worker waiting on a slow dependency (a service replica is one server with a 10 ms service time): point this out yourself.
Backlog. A ten-minute outage of everything leaves under a million small messages queued. Queues keep messages for days (SQS for up to 14), so "never lose an accepted event" comes from the queue, not from heroics in the workers.
Sizing the parts you add. A queue replica takes 50k writes a second, so two are lightly loaded and redundant. A service replica takes 2k requests a second against the worker's 1,500 events a second: divide by your target utilisation, and stay under 100% with one replica lost.
Latency in the model. Notify is the Order Service writing the queue, a 5 ms hop: lots of room under 50 ms, unless a provider is on the path, since one call is already 200 ms.
Deep dive Why a provider's tail must stay off the caller's path
Deliver has no latency requirement; the providers decide its p99. A provider is a single server in the model, so at 41% busy the email provider takes around 340 ms on average because of queueing: exactly the slow tail you keep off your caller's path.
Availability. Notify's availability is the queue's, and two replicas of a 99.99% queue are effectively always up in the model. Deliver's is capped by the providers (99.9% each), which is fine: the queue holds events until delivered.
Cost. The given services cost $700 a month ($100 per service replica); a queue replica is $200. There is room for a sensible design, not for doubling every tier.
Put a durable queue between a fast caller and slow work. The caller's request ends when the queue stores the message; workers drain it at the pace the providers allow. The queue absorbs bursts (load levelling) and outages (buffering).
Trade-offs: delivery is asynchronous, so the caller only knows the email will be attempted, and you must monitor queue depth and message age. When not to use it: when the caller truly needs the result now. (A one-time password the user awaits may still go through a queue, with a priority lane and tight alerting.)
Queues like SQS deliver at least once: every message arrives, and a few arrive more than once. A message is hidden for a visibility timeout while a worker handles it, and deleted only when the worker says so; if the worker crashes, it reappears for another worker. Occasionally a message arrives twice even without a crash, so the SQS documentation says to make consumers idempotent (safe to run twice).
Here a duplicate means a customer gets the same text twice. The defence is an idempotency key: a unique id (the event id) that the worker passes to the provider or records itself, so a repeated send does nothing. Stripe's write-up on idempotency explains the pattern for APIs in general.
Someone must store the keys for a while: the provider, if it accepts idempotency keys, or a small table or cache of recently sent event ids.
A third party will fail. For channels with alternatives (SMS has many vendors), the worker tries the primary and, on a timeout or error, sends through a backup. A circuit breaker makes this cheap: after enough failures it skips the primary for a while, so you do not pay a timeout on every message.
In Proschi a failed call is -x: a timeout (1,000 ms by default) and no answer. A scenario that calls a node with -x and still succeeds through another node is a fallback, what survive failure of … and handles failure of … look for.
Deep dive In Proschi: two providers
title"Two providers"jobs "Address Events"[AWS SQS] x2app "App"[Worker] x2primary "Geocoder"[Third Party API]backup "Backup Geocoder"[Third Party API]jobs -> app : deliverapp -> primary : lookupapp -> backup : lookupusecase"Geocode" { jobs -> app : AddressAdded alt"Primary" { app -> primary : GET /geocode primary --> app : 200} alt"Failover"when"the primary times out" { app -x primary : GET /geocode app -> backup : GET /geocode backup --> app : 200 }}
Do not fail over when both providers could act (a payment captured twice) or when the backup is much worse than a short delay: retry later from the queue instead.
1. Scope. Clarify the channels (email, SMS, push), who picks one (the user's preferences), whether one event can produce several notifications (no, at most one), the peak rate, and the promise to the caller ("accepted" means stored, not sent). Ask about opt-outs: for legal and ethical reasons, they are honoured before anything goes out.
2. High-level design. Two flows, one boundary between them:
Notify: Order Service → queue → 200. Nothing else on that path.
Deliver: queue → worker → Preferences → one provider → delete the message.
Name and reject the alternative, the Order Service calling providers itself: it couples order processing to the worst provider's latency and availability, which the requirements forbid.
3. Deep dive.
Scenarios. The preference answer decides the branch: one per channel, a failover branch where the SMS call fails with -x and the backup is called, and an opted-out branch that only deletes the message. Spell the names exactly as the problem does; the traffic mix refers to them.
Ordering of steps. Preferences, then provider. Delete the message only after the provider accepted it (or after deciding not to send), so a crash causes a retry, not a loss.
Sizing. Two queue replicas, and enough workers for 1,500 events a second at well under 70%, still fine with one lost. Nothing needs to be big, and nothing should be: $700 of the $2,000 is already spent.
Duplicates. Use the event id as the provider's idempotency key; mention it though the model does not simulate redelivery.
4. Wrap-up. Check: Notify is one queue write; no path from the Order Service to a provider; the failover scenario covers an SMS outage; every node you run has two or more replicas; the bill fits. Then extend: priority queues for urgent messages, per-user rate limits so nobody gets twenty texts in a minute, a dead-letter queue for events that keep failing, and provider delivery receipts to track what arrived.
A notifier that queues and then sends inline (wrong/notifier-calls-provider-inline). The notifier writes the queue, then calls the email provider before answering. The caller still waits for a 200 ms third party, and every provider outage becomes an order-processing outage. In the model it is worse: the inline sends come on top of the worker's, the email provider goes over its 2k rps limit, and Notify's p99 runs to many seconds. It fails The Order Service never waits for a provider, and also Notify's p99 and availability limits.
A worker that polls the queue (wrong/worker-polls-queue). Deliver starts with the worker sending ReceiveMessage to the queue. Common with SQS, but in this model it reads as the worker writing to the queue, and the flow no longer starts at the queue. It fails Workers take events from the queue. Model the queue handing the event over instead.
Other classic mistakes.
Checking preferences after sending, or not at all for one channel. An opted-out user gets a message. Fails Preferences are checked before anything is sent.
Only retrying the primary SMS provider. Every SMS waits out the outage, and with no fallback survive failure of sms fails.
Deleting the message before the send. If the worker dies right after, the notification is lost: at-most-once by accident.
One queue replica or one worker. A single point of failure; fails survive any node failure.
One giant queue for every priority. A marketing blast of millions delays password resets: separate queues (or priorities) per class of message.
Give the core idea, draw the queue as the boundary, then walk one event through each scenario.
Follow-ups you should expect:
How do you avoid sending twice? At-least-once plus idempotency: the event id is the key, sent to the provider or checked in a small store of recently sent ids.
A provider is down for an hour? Messages stay queued (or are retried with backoff), a circuit breaker switches to the backup where one exists, and queue age alerts fire. Nothing is lost.
Provider rate limits? Limit worker concurrency per provider, or a token bucket per provider in the workers.
Templates and localisation? A template service the worker calls after preferences, keyed by event type and locale, heavily cached.
How do you know it was delivered? Provider callbacks (delivery receipts) written to a status store; for push, the app can acknowledge on display.
Bulk campaigns? A separate pipeline and queue, never competing with transactional messages.