System design practice

View Counting

medium Based on a system published by Reddit streaming queues probabilistic caching real-world

Reddit's view counter: Kafka, a filter, and HyperLogLogs in Redis.

Solve it in your browser Read the lesson first

In 2017 Reddit started showing how many people viewed each post. A view count sounds like a counter, but it is a count of unique viewers: a user who opens a post ten times is one view. Keeping a set of user ids per post would take megabytes for every popular post. Reddit's answer was a streaming pipeline that filters views and counts them with HyperLogLog, a sketch of at most 12 KB per post that is typically off by less than 1%.

Functional requirements

Use these use case and scenario names exactly: the traffic, requirements and tests in problem.proschi refer to them.

Scale

Constraints

What is given

problem.proschi declares the reader and holds the traffic, requirements and tests. Add everything else: the services, Kafka, Redis, the database, the connections and the five use cases. The post gives no request rates; the scale above is an assumption for this exercise.

Based on

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 11 min read

Learn it: View Counting #

"How many people saw this post?" sounds like count += 1. It is not: a Reddit view count counts unique viewers within a time window, filters out bots and refreshes, and keeps up with tens of thousands of events a second. This lesson follows Reddit's 2017 design: a pipeline plus a 12 KB sketch rather than a table.

What you'll learn #

The problem, explained #

Clients report each view; later readers see a number next to the post. Reddit's post lists four requirements: counts in near real time, each user counted once within a short window, a displayed number within a few percent of the truth, and production scale.

The five use cases map onto a pipeline:

Limits: p99 of Record view and Read count under 60 ms; both 99.9% available; the view durable before the answer; any single machine loss survived; at most $10,000 a month.

The given file contains only the reader actor plus traffic, requirements and tests; you build everything else. The tests, in plain words:

Back-of-the-envelope #

views in
20k / s
counted after the filter
18k / s
database writes from 10-second persistence
2k / s
exact set vs HyperLogLog, 10M viewers
80 MB vs 12 KB
rows a day at the peak rate, the naive way
≈ 1.6 billion

Event rates. 20k views a second arrive; about 10% are filtered, so 18k a second are counted. About 1% of those find their counter evicted: 180 database reads a second. Roughly 20k posts change in any 10-second window, so persisting costs 20k / 10 = 2k database writes a second. Reads are 10k a second.

The exact-set alternative. At 8 bytes per unique viewer per post, a post with 10 million viewers needs 80 MB, for every popular post, on every replica. A Redis HyperLogLog tops out at 12 KB whatever the cardinality, with a standard error of 0.81%.

ApproachMemory for a 10M-viewer postError
Set of 8-byte ids~80 MB0
Redis HyperLogLog≤ 12 KB~0.81%

Write amplification avoided. Writing Cassandra once per counted view would be 18k writes a second; writing each changed counter every 10 seconds is 2k. That 9× reduction grows with popularity: a hot post getting 1,000 views a second still costs one write per 10 seconds.

Per-tier load and replicas. With services at about 2k requests a second per replica:

TierLoadReplicas at 100%
Event collector20k/s10
Filter consumer20k/s10
Count consumer18k/s9
Counts API10k/s5

Then add headroom: Proschi marks anything above 70% as hot, and "survive any node failure" re-runs the analysis with one replica fewer.

Cost. Services $100 per replica, Redis $150, Kafka $200, Cassandra $500, the load balancer $50. Four service tiers with headroom are most of the bill, leaving little room for spare tiers or a bigger database.

Rows per day, the naive way. 18k rows a second × 86,400 seconds ≈ 1.6 billion new rows a day if the peak held all day, and still hundreds of millions on an average day, forever. Keep that number handy for the interview.

Concepts #

HyperLogLog: counting distinct things in fixed memory #

Hash every user id to a bit string that looks random. A run of k leading zeros happens with probability 1 in 2^k, so if the longest run you have seen is 20, you have probably seen about a million (2^20) distinct values.

One such estimate is noisy. HyperLogLog splits the hash space into many buckets (Redis uses 16,384), keeps the longest run per bucket in a few bits each, and combines them with a harmonic mean (an average not thrown off by a few very large values). Flajolet and colleagues showed the relative error is about 1.04 / √m for m buckets, which is where Redis's 0.81% comes from.

Properties that matter in design:

Trade-offs. You cannot list who viewed, remove a viewer, or get an exact number. When not to use it: billing, quotas, anything where a user may dispute the count, or small sets where an exact set is cheap.

title "Distinct counter in a cache"

app   "App"            [Go]    x2
redis "Distinct Counts" [Redis] x2

app -> redis : RESP

usecase "Add member" {
  app    -> redis : PFADD visitors:2026-10-05 user-42
  redis --> app   : 1
}

usecase "Count members" {
  app    -> redis : PFCOUNT visitors:2026-10-05
  redis --> app   : 18342
}

A durable log first, the work later #

Rules, counting and persistence run in consumers that read the log at their own pace. Stages connected by topics let you change, scale or replay the filter without touching the counter, which sees only events that matter. The cost is more moving parts and end-to-end latency in seconds rather than milliseconds.

When not to use it. When the user must see the result of their own action immediately and exactly (a like button that must flip at once is usually updated optimistically in the client, with the real count arriving later).

Deep dive The pattern in Proschi: a two-stage stream
title "Two-stage stream"

client "Client"    [Actor]
api    "Ingest"    [Go]    x2
log    "Event Log" [Kafka] x3
stage1 "Enricher"  [Go]    x2
stage2 "Counter"   [Go]    x2

client -> api    : HTTPS
api    -> log    : produce
log    -> stage1 : consume
stage1 -> log    : produce
log    -> stage2 : consume

usecase "Ingest event" {
  client -> api    : POST /events
  api    -> log    : PRODUCE raw
  log   --> api    : ok
  api   --> client : 202
}

usecase "Enrich" {
  log     -> stage1 : RawEvent
  stage1  -> log    : PRODUCE enriched
  log    --> stage1 : ok
  stage1 --> log    : commit offset
}

Cache as primary, database as periodic backup #

Usually the database behind a cache is the truth. Here, for hot data, it is the other way around: counters live and are read in Redis, and Cassandra holds a copy refreshed every 10 seconds to restore an evicted or lost counter. This is write-behind caching: writes land in memory and reach durable storage in batches.

It works here because the counter is approximate anyway: losing up to 10 seconds of increments on a rare eviction is within "a few percent", and the raw events are still in Kafka for a rebuild.

title "Write-behind snapshot"

job   "Flusher"        [Go]        x2
hot   "Hot Counters"   [Redis]     x2
cold  "Counter Backup" [Cassandra] x3

job -> hot  : RESP
job -> cold : CQL

usecase "Flush" {
  job   -> hot  : GET counter:42
  hot  --> job  : value
  job   -> cold : UPSERT counters 42
  cold --> job  : ok
}

Designing it step by step #

Step 1: scope. Clarify "unique per what": per user per post within a short window. Ask how accurate (a few percent), how fresh (seconds), and whether anyone needs the viewer list (no). Those three answers unlock approximation and asynchrony.

Step 2: high-level design. Four paths:

PathFlow
Writeclient → load balancer → collector → Kafka → answer 202
ProcessingKafka → filter (with its own Redis for "has this user viewed this post recently?") → counted topic → counter → Redis HLL per post
Readclient → load balancer → counts API → Redis PFCOUNT
Backgrounda job that copies changed counters to the database

Step 3: deep dive.

Why two Redis clusters? The filter's state (recent views per user and post, short TTLs, very hot keys) and the counters (one HLL per post, read by the API) differ in shape and failure impact. Separating them keeps a bot storm in the filter from evicting counters.

Eviction. Under memory pressure Redis may drop a cold post's counter. The count consumer must handle a missing key: read the stored HLL from the database, write it back to Redis, then PFADD. Model it as its own scenario: it is the only time the count path touches the database.

Persistence. Track which counters changed (a dirty set); every 10 seconds read each from Redis and upsert it into the database. One write per active post per interval, not per view.

Sizing. Use the envelope's load table. Every service tier stays well under 70% and under 100% with one replica gone; Kafka and each Redis get at least two replicas. Check the total against the budget before adding anything optional.

Step 4: wrap-up. Name what the model misses: hot posts concentrating load on one Redis shard, consumer lag during spikes, Kafka retention for replays, and HLL accuracy at very small counts (Redis's sparse encoding and bias correction handle it). Offer exact counting for small creators if the product asks for it.

Common mistakes #

Counting in the request (wrong/count-in-request). The collector does a PFADD before answering. In production page latency now depends on Redis: a slow or failing counter cluster slows or fails view recording, and spikes hit Redis directly. It also skips the filter, so bots count.

Reading the count from Cassandra (wrong/read-count-from-cassandra). The database copy is up to 10 seconds behind Redis, every read hits the slower, costlier store, and the backup becomes a read-path dependency. Proschi still meets the latency limit, so the test "Redis serves the counts; the database is a backup written in batches" is what fails.

A row per viewer (wrong/row-per-viewer). Exact, and it "also lets us show who viewed". But it is a write per counted view (about 18k a second instead of 2k) and about 1.6 billion new rows a day that never shrink. The test "Unique viewers go into a HyperLogLog in Redis, not a database" fails because the normal count path now calls the database.

Classic mistakes beyond this problem:

In the interview #

Likely follow-ups:

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems