DoorDash's search finds stores and dishes near a diner. Its index is built from the stores, menus and items that merchants and operators edit all day, plus scores and tags that a nightly ETL computes for every item. The old indexing was slow and unreliable: it could take up to a week to reindex all of the changes, mismatches between the index and the source of truth had to be fixed by hand, and a full reindex job typically took a month.
The rebuild separated the two sides completely. Edits go to the database, and a change event goes to Kafka. Flink jobs turn each event into a search document and write it to Elasticsearch in bulk. Diners query Elasticsearch only. Design that pipeline and the query path beside it.
Functional requirements
Search: a diner searches for stores and dishes and gets results from the search cluster.
Edit menu: a merchant changes a store, menu or item. The edit is saved in the db before the merchant hears back, and a change event (the store's id, not its contents) is published to a change stream. The edit never waits for the search cluster, and nothing but the stream leads into the index.
Index change: an indexer receives a change event from the stream, reads the store's current rows from db, builds the search document and adds it to a bulk buffer. Two scenarios:
"Buffered": the buffer has room; the document waits in it, and the search cluster is not called.
"Flush": the buffer is full (about 100 documents) or a second has passed; one _bulk request writes the whole batch.
Backfill: the nightly ETL's rows (scores and tags) are streamed in throttled batches by the given loader into the change stream, and from there to the indexers, with the same "Buffered" and "Flush" scenarios. The rows carry their values, so a backfill never reads db.
Use these use case and scenario names exactly: the traffic, requirements and tests in problem.proschi refer to them.
Scale
10k searches a second at peak.
2k edits a second, each one change event.
The backfill streams 10k ETL rows a second, spread over the day.
A bulk request carries about 100 documents, so 1% of the changes and rows trigger a flush.
An Elasticsearch node serves about 3k requests a second.
DoorDash did not publish these rates; they are assumptions for the exercise.
Constraints
Searches never call db, and no client connects to it.
p99 of a search under 120 ms, of an edit under 150 ms, with the backfill running.
Searches available 99.95% of the time, edits 99.9%.
Every shard of the index has at least two copies; losing any single machine must not break a latency limit.
At most $8,000 / month, the database and the loader included.
What is given
problem.proschi declares the two clients, merchant and diner, the source-of-truth database db (PostgreSQL, x2) and loader, the job that streams the ETL's rows. Add the search cluster (Elasticsearch: shards with capacity { search shards 3 } are yours to choose), the API in front of it, the API that saves edits, the change stream, the indexers and the four use cases.
What this model leaves out
Freshness. The point of the pipeline is that an edit shows up in search as quickly as possible; this exercise takes "within seconds" as its goal (DoorDash's post gives no figure). The simulation measures request latency, not how old the index is, so freshness is encoded as a test (edits are indexed from the stream) rather than as a number.
How Elasticsearch spends its nodes. A real query visits one copy of every shard, and every copy applies every write. The simulation spreads both over all the nodes, so it is a little optimistic about large clusters; the shard and replica arithmetic is in the lesson.
Lost events. Publishing after the commit leaves a small gap: a crash between the two loses the event, and the index stays stale until the store changes again or the next full reindex. An outbox or log-based change capture closes it; the simulation has no crashes in the middle of a request.
Two Flink jobs in one. DoorDash runs an Assembler that builds the document by calling the backend services, then a second Kafka topic and an Elasticsearch sink. Here one indexer does both and reads db directly.
Based on
Satish Saley, Danial Asif and Siddharth Kumar, Building Faster Indexing with Apache Kafka and Elasticsearch, DoorDash Engineering blog, July 14, 2021: the legacy indexing, where reindexing all of the changes could take up to a week, a full reindex job typically took a month, and mismatches with the source of truth were fixed by hand; a platform built on Kafka, Flink and Elasticsearch, fed from Postgres and a Snowflake warehouse; change events published by save hooks in the applications ("application level CDC") and carrying only the changed entities' ids, because Debezium change data capture showed too much overhead in performance tests; a Flink Assembler that dedupes events over a short window and fetches the full data in bulk from the services' REST APIs, and a Flink sink that writes Elasticsearch in bulk with rate limiting, a dead-letter queue and one consumer group per index; nightly ETL data streamed into Kafka in batches sized so downstream is not overwhelmed, with the goal of ingesting it within 24 hours; and the results: a store backfill from one week to 6.5 hours, an item backfill from two weeks to 6.5 hours, and a reindex of existing stores and items from one week to two hours.
How your design is checked
You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.
A food delivery app's search box answers "tacos near me, open now". The answer comes from stores, menus and items that merchants and operators edit all day, plus ranking scores and tags that a nightly ETL job (extract, transform, load: a batch job over the data warehouse) computes for every item. A relational database is the right home for those edits, with transactions and constraints, and the wrong tool for ranked text and location search over every menu. So the data lives twice: in the database as the source of truth, and in a search engine as an index built from it.
The hard part is keeping the copy in step. DoorDash's first indexing system did it slowly and unreliably: reindexing all of the changes could take up to a week, mismatches between the index and the database had to be fixed by hand, and a full reindex job typically took a month. Their rebuild, on Kafka (the change stream), Flink (a stream processor, which runs the indexing jobs) and Elasticsearch (the search cluster), made the index follow changes as they happen, cut the reindex of existing stores and items from a week to two hours, and cut the store and item backfills (one and two weeks) to 6.5 hours each.
The problem has four use cases:
Search: a diner queries the search cluster.
Edit menu: a merchant's change is saved in db and published as a change event.
Index change: an indexer consumes the event, reads the store from db, and either buffers the document ("Buffered") or flushes a bulk request ("Flush").
Backfill: the nightly ETL's rows enter the same stream through the given loader and reach the index in bulk too.
The tests check that searches never touch db, that an edit publishes to a broker and never calls the search cluster, that both indexing use cases come from the stream, that only flushes write the index, and that every shard has a replica.
The limits: p99 (the latency 99% of requests beat) under 120 ms for a search and 150 ms for an edit, both with the backfill running; searches available 99.95% of the time and edits 99.9%; no single machine whose loss breaks a latency limit; and at most $8,000 a month.
The numbers are assumptions for the exercise; DoorDash did not publish its rates.
Flow
Rate
Where it lands
Search
10k/s
Search API, then the search cluster
Edit menu
2k/s
Edit API, db (writes), Kafka
Index change
2k/s
indexers, db (reads); 20 bulk requests/s to the cluster
Backfill
10k/s
Kafka, indexers; 100 bulk requests/s to the cluster
The search cluster. At about 3k requests a second per node, the queries alone need 10k ÷ 3k ≈ 3.3 nodes. If every change were its own write, the cluster would take another 12k a second: 22k in total, more than seven nodes' worth before any headroom. A bulk request writes about 100 documents in one call, so with bulk requests the writes fall to about 120 a second, almost nothing.
Shards and copies. A shard is one piece of the index, and each shard is kept in several copies on different nodes (both are explained under Concepts). Three shards with three copies each is nine nodes: 27k requests a second of capacity for 10k queries, about 37% busy. The simulation gives each shard a third of the queries, 3.3k a second over its three copies, so losing a node leaves that shard at about 56%. That is a little optimistic: a real query visits one copy of every shard. Nine Elasticsearch nodes at $400 is $3,600, the largest line of the budget.
The rest. The Search API at 2k a second per instance needs 10k ÷ 2k = 5 instances; 8 keep it near 62% and 71% after a loss. The indexers handle 12k messages a second: 10 instances at 60%. The database sees 2k writes and 2k reads a second, well within one primary.
The writers (merchants) and the readers (diners) want different things from the data. Writers need transactions and a single source of truth; readers need ranked text and geographic search, fast, at ten times the rate. Serving both from one store forces one of them to lose: text search on the primary competes with edits for CPU and locks, and a relational index cannot rank "tacos" by relevance and distance anyway.
So keep two stores, each built for its side, and connect them only through a stream. This is the idea behind CQRS (command query responsibility segregation): the database takes commands, the index answers queries, and an asynchronous process turns one into the other.
What it costs is staleness. The index is always a little behind, here by seconds. For search that is fine: an old price in the result list is harmless, because the order itself is placed against the database.
An edit commits to the database, then a change event goes to Kafka, and the merchant gets the answer. Indexers consume the stream and update the index a moment later. The merchant never waits for the search cluster, and a slow or failing cluster only makes the index a little older; it never fails an edit.
DoorDash publishes these events from save hooks in the applications: code that runs whenever a store, menu or item is saved. They looked at the usual alternative, log-based change data capture with Debezium, and performance tests showed too much overhead, especially on a database that also serves online traffic. Both approaches have the same shape: the database commits first, and an event follows.
Two details make the stream robust:
Events carry ids, not data. An event says "store 812 changed". The indexer then reads the store's current state (DoorDash's Assembler calls the backend services; here it reads db). Events can be duplicated, reordered or replayed, and the indexer still writes the latest version. An event that carried the new price could arrive after a newer one and overwrite it with an old value.
The index can always be rebuilt. Because the index is derived, replaying the stream or streaming every row through it again repairs any drift. That is what the backfill does.
The gap to know about: publishing after the commit is a dual write. If the process dies between the two, the event is lost and the index stays stale until the store changes again. A transactional outbox (write the event in the same transaction, relay it afterwards) or log-based change capture closes the gap.
Deep dive Why not write to Elasticsearch in the request?
Writing the database and then the index in the same request looks simpler. It couples the edit's latency and availability to the search cluster: a cluster under a heavy reindex slows every edit, and a cluster that is down fails them. Worse, the two writes are not atomic. An edit that commits and then fails to index leaves the index wrong with nobody to retry it, which is the drift DoorDash's engineers were fixing by hand. A stream makes the retry someone's job: the consumer keeps its offset until the index has the document.
Every request to a search cluster has a fixed cost: a round trip, a thread, a write to the transaction log, and work to make new documents searchable. A _bulk request pays that once for hundreds of documents. So the indexer buffers: each change adds its document to an in-memory batch, and the batch is sent when it reaches a size (here about 100 documents) or a time limit (a second), whichever comes first.
That is what the two scenarios model. In "Buffered" the indexer only adds to the buffer and the cluster sees nothing. In "Flush", 1% of the time, one request writes the whole batch. The cluster takes 120 requests a second instead of 12k, and queries keep the nodes to themselves.
The time limit matters as much as the size: without it, a quiet hour would leave a half-full buffer waiting. DoorDash's sink also rate-limits its writes, so that a burst of changes cannot crowd out queries on the cluster. The backfill gets the same protection one step earlier: the ETL's nightly rows are streamed in batches sized so the pipeline downstream is not overwhelmed, spread out rather than in a spike, with the goal of ingesting them within 24 hours.
A search index is split into shards, each a self-contained piece of the index holding a subset of the documents. Shards let an index be bigger than one node and spread indexing work. Each shard then has replicas: full copies that can answer queries for it and take over when its node dies.
They scale different things. A query must visit one copy of every shard and merge the results, so more shards do not add query throughput; they add work per query. More replicas do: each copy can serve a share of the queries. But every copy must apply every write, so replicas multiply indexing work, which is one more reason to keep writes in bulk.
Count nodes as shards × copies. Three shards with one primary and two replicas each is nine nodes, and losing any one leaves every shard two copies. Nine shards with no replicas is also nine nodes, with the same total capacity in the simulation, and losing one node loses a ninth of the stores from every search.
The read path. Add the search cluster and an API in front of it. The diner calls the API, the API queries the cluster, and nothing on this path connects to db.
The edit. Add an API for merchants that writes db and then publishes a change event, the store's id, to a Kafka topic. Answer the merchant after the commit; the publish can be asynchronous.
The indexers. Add a pool of indexers that consume the topic. For each event, read the store's current rows from db and build the document. Split the use case into "Buffered" (add to the buffer, answer the stream) and "Flush" (one bulk write, then commit the offsets).
The backfill. Send the loader's rows into the same topic. The indexers handle them like changes, without reading db, since the rows carry their values.
Size the cluster. Choose shards and copies: with capacity { search shards 3 } and x3, nine nodes. Check that each shard's copies stay well under 70% busy with one copy gone.
Size the rest. The Search API and the indexers near 60% busy, at least two of everything, and a look at the budget: the search cluster is about half of it.
Explain the trade-offs in decision blocks: why the index is fed from a stream, why events carry ids, why bulk, why replicas.
Indexing inside the edit. The edit API writes the database, then the search cluster, then answers. Edits now wait for the cluster and fail with it, and an edit that saved but did not index leaves them out of step. It fails An edit is saved, then indexed in the background. Publish an event and index from the stream.
Searching the database. The Search API runs ILIKE queries on PostgreSQL. It cannot rank by relevance and distance, and it competes with every edit on the source of truth. It fails Searches never touch the source of truth.
One write per change. The indexers send each document as its own request: 12k writes a second on the nodes that answer queries. Search p99 climbs past the limit, and losing a node saturates a shard. It fails Documents are written to the index in bulk. Buffer and flush.
Shards without replicas. Nine shards with one copy each have enough capacity, but each shard lives on one node. Losing it breaks every search. It fails Every shard of the index has a replica and survive any node failure.
Start from the two sides. Writers and readers, their rates and their needs, then the stream between them. Saying "the read model is eventually consistent, the database decides" early frames everything that follows.
Expect "how fresh?" Seconds, in this exercise (DoorDash's post says "as quickly as possible" without a figure): the edit commits, the event is consumed within a second or two, and the bulk flush timer bounds the rest. Mention the search engine's refresh interval as the last step: Elasticsearch makes newly written documents searchable on a refresh, by default about once a second.
Expect "what if an event is lost?" Ids-only events and a full reindex make the index self-healing; an outbox or log-based capture closes the publish gap.
Expect "how do you reindex everything?" Stream every row through the same pipeline at a throttled rate, or build a new index alongside and switch an alias once it has caught up. DoorDash's rebuild brought the reindex of existing stores and items from a week to two hours.
Expect "shards or replicas?" Shards for the size of the data and to spread indexing, replicas for query throughput and failures; count nodes as shards × copies.
Martin Kleppmann, Designing Data-Intensive Applications, chapter 11, "Stream Processing" (the section on keeping systems in sync and change data capture).