System design practice

Explore Recommendations

hard Based on a system published by Meta ranking machine-learning caching read-heavy estimation real-world

Instagram Explore's ranking funnel: 500 candidates, then ever heavier models rank ever fewer.

Solve it in your browser Read the lesson first

Over half of the Instagram community visits Explore every month: a grid of photos and videos from accounts they do not follow yet, picked for them out of billions of posts. Meta's Explore system is a funnel. Candidate generation finds a few hundred posts that might interest the user, and a series of ranking passes, each more expensive per item than the last, narrows them to the 25 that fill the first page.

The expensive part is the ranking: a deep neural network that predicts likes, saves and "see fewer posts like this" is far too costly to run on every candidate. The funnel spends it only on the few that cheaper models already rated highly. Design the serving path for one Explore page, and the offline job that keeps its features and embeddings fresh.

Functional requirements

Name the nodes retrieval, embeddings, features, light, medium and heavy: the capacity and the tests in problem.proschi refer to them, and to the two use case names above.

Scale

The funnel's sizes (500, 150, 50, 25) and the three kinds of model are from Meta's post. The request rate, the model speeds and prices and the warehouse's numbers are assumptions for the exercise: Meta published only fleet-wide totals (65 billion features extracted and 90 million model predictions a second).

Constraints

What is given

problem.proschi declares the user, the warehouse of logged interactions (two nodes), the offline pipeline, the speed and price of each ranking model, the traffic, the requirements and the tests. Add the Explore API, retrieval and its embeddings index, the features store, the three ranking passes, their connections and the two use cases.

What this model leaves out

Based on

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 12 min read

Learn it: Explore Recommendations #

Recommendations: a ranking funnel that spends the best model on the fewest items #

What you'll learn #

The problem, explained #

Instagram's Explore tab shows each person a grid of posts from accounts they do not follow yet. Over half of the Instagram community visits it every month, and the posts come from a pool of billions. Showing everyone the same popular grid would be cheap and dull; ranking every post for every person is impossible. Meta's answer, described in its 2019 post, is a funnel.

Candidate generation comes first. It starts from the person's seed accounts, the accounts whose posts they liked or saved, and looks up accounts that are similar to them. Similarity comes from account embeddings: Meta trains vectors (ig2vec, a word2vec-like model) on the sequences of accounts people interact with in a session, so accounts that are visited together end up near each other. A nearest-neighbour lookup finds the closest accounts, and their recent posts, filtered for eligibility, become the candidates. Each request samples 500 of them.

Ranking then takes three passes. A distilled model with minimal features scores all 500 and keeps 150. A lightweight neural network with the full set of dense features keeps 50 of those. A deep neural network with dense and sparse features orders the final 25, which fill the first page. The deep model predicts several actions at once (like, save, "see fewer posts like this"), and a weighted sum of those predictions, with a penalty for showing the same author twice, is the final score.

The problem has two use cases. Load Explore runs the funnel while the user waits. Refresh features is the offline pipeline that reads the warehouse and refills the feature store and the embedding index. The tests check that retrieval runs before the first ranker and that the rankers run from cheapest to heaviest, that features are read from the online store and never from the warehouse while a user waits, and that users reach only the Explore API.

The limits: 1k Explore loads a second, p99 (the latency 99% of requests beat) under 300 ms with all three passes included, 99.95% availability, no latency limit broken by losing any single machine, and at most $20,000 a month, the warehouse and the pipeline included. The statement also gives each model's speed and price: one light machine scores 100k items a second for $200 a month, one medium machine 25k for $300, one heavy machine 5k for $600.

Back-of-the-envelope #

items for the first pass
500k / s
items for the deep model
50k / s
machines to run the deep model on all 500 candidates, near 70% busy
140
machines it needs in the funnel, with the same headroom
15
a month for the reference design
$16,100

The request rate and the model speeds are assumptions for the exercise; the funnel's sizes are Meta's.

PassItems per requestItems a secondPer machineMachines (no headroom)Machines with headroom
light (distilled)500500k100k58
medium (lightweight network)150150k25k610
heavy (deep network)5050k5k1015
heavy on every candidate500500k5k100about 140 (at 70% busy)

The deep model on everything. 1k requests × 500 candidates = 500k scores a second. At 5k a machine that is 100 machines before any headroom, about 140 to stay near 70% busy: some $85,000 a month at $600 each, four times the whole budget.

The funnel. The deep model sees only 50 items per request: 50k a second, 10 machines, 15 for headroom (67% busy, 71% after losing one). The two cheaper passes add 8 and 10 machines (63% and 60% busy). All three pools together cost $13,600, a sixth of the deep model alone on everything.

Latency. The page waits for each pass in turn, so their latencies add: about 10 ms for the API, 10 ms for retrieval plus 15 ms for the index, 1 ms for the features, then 5, 10 and 40 ms for the three passes. That is about 90 ms idle, a little more under load, and about 260 ms at p99 in the simulation, under the 300 ms limit. A warehouse query, at half a second, would not fit even once.

Concepts #

The ranking funnel #

A recommender faces a trade-off between how good a model is and how many items it can afford to score. The best model uses hundreds of features and a deep network, and it costs tens of times more per item than a simple one. The candidate pool is large, and most of it is obviously wrong for this user: a cheap model can tell that much.

So the work is split into passes. Each pass scores what the previous one kept and passes on only its best. Early passes are cheap and see many items; late passes are expensive and see few. The expensive model's load is set by the last cut, 50 items a request, not by the size of the pool. That is what keeps both the bill and the latency bounded: the latency of each pass is one batched call, and the slowest pass sees the smallest batch.

The cost of the funnel is recall: a post the first pass drops is never seen by the deep model, even if the deep model would have loved it. Meta reduces that loss by distillation: the first-pass model is trained on the recorded outputs of the main models, so it learns to imitate their ranking with far fewer features. A good funnel loses few of the items the final model would have chosen.

Candidate retrieval with embeddings #

Ranking needs a short list to start from. Retrieval builds it without scoring individual posts. Each account is a vector; accounts that people visit together have vectors that point the same way. Given the user's seed accounts, a k-nearest-neighbour (KNN) search over those vectors returns similar accounts, with cosine distance or a dot product as the measure. Meta runs this lookup with FAISS, its library for similarity search.

The index is built offline: the embeddings come out of a training job over logged sessions, and the index is reloaded when they change. At request time it only answers lookups, which is why it can be fast. The posts of the similar accounts are then filtered for eligibility (no content that is likely to violate policy, no spam) and sampled down to 500.

Deep dive Approximate nearest neighbours

An exact KNN search compares the query with every vector, which is too slow over millions of accounts. Approximate methods (inverted lists of clusters, graph indexes such as HNSW, product quantization to shrink the vectors) find nearly the same neighbours by looking at a small part of the index. They trade a few percent of recall for orders of magnitude of speed, the same trade the ranking funnel makes. Meta's 2023 post describes item embeddings generated daily offline and served from such an approximate index.

Precomputed features and embeddings #

A ranking model's inputs are features: how often this user liked posts from this author, how many saves the post got in the last day, the author's embedding. Most of them summarise long histories, so they change slowly and are expensive to compute. Computing them from raw logs while a user waits would take seconds.

So they are computed offline, in batches, from the warehouse where the interaction logs land, and written to an online feature store: a key-value store or cache that answers a batch of keys in about a millisecond. A request reads the features of all 500 candidates and of the user in one batched call, and each ranking pass uses the subset it needs. The warehouse stays an offline system: it answers in about half a second and serves a few hundred queries a second, fine for a pipeline and hopeless for 1k requests a second.

The cost is staleness: a feature is as old as the last batch that wrote it. Features that must be fresh (what the user tapped a minute ago) come from a streaming job instead; the simulation treats them all as one store.

Sizing each pass #

Each model has its own cost per item and sees its own number of items, so each gets its own pool, sized from its own numbers:

machines = requests a second × items per request ÷ items one machine scores a second

then enough extra that the pool stays near 70% busy after losing one machine. Pools that are the same size are wrong for at least two of the three: the distilled model idles while the deep model queues. Separate pools also scale independently: a new, heavier final model changes only its own pool.

Designing it step by step #

  1. The API. Add an Explore API that the user calls; it will run the funnel. Nothing else is reachable from the user.
  2. Retrieval. Add retrieval and its nearest-neighbour index embeddings. The API asks retrieval for candidates; retrieval looks up accounts near the seed accounts and returns 500 eligible posts.
  3. Features. Add the online features store. The API reads the features of the 500 posts and of the user in one batched call before ranking.
  4. The three passes. Add light, medium and heavy, and call them in that order. Use a fan-out label for the items each one scores, such as x500 SCORE, so that each pass carries the load of its batch.
  5. The offline pipeline. Write "Refresh features": the given pipeline reads the warehouse, then writes features and embeddings. The serving path never touches the warehouse.
  6. Size every pool. From the table in Back-of-the-envelope: 8, 10 and 15 machines for the passes, two or three of everything else, and a look at the budget, where the deep model is more than half of the bill.
  7. Explain the trade-offs in decision blocks: why a funnel, why precomputed features, why one pool per model.

Common mistakes #

In the interview #

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems