System design practice

File Storage

medium storage cdn egress async

Pre-signed uploads: the API never touches the bytes.

Solve it in your browser Read the lesson first

Design a small Dropbox: users upload files, list their folders and download files again, from any device. Files are anything from a 10 KB note to a 2 GB video, so their bytes must never stream through your own servers: the API hands out pre-signed URLs and the client talks to object storage (for uploads) and to a CDN (for downloads) directly.

Functional requirements

Use these use case and scenario names exactly: the traffic, requirements and tests in problem.proschi refer to them.

Scale

Constraints

What is given

problem.proschi declares the user and the bucket blobs (S3, two partitions) and holds the traffic, requirements and tests. Add the API, the metadata store, the CDN, how a stored object marks its file ready, and the four use cases.

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 13 min read

Learn it: File Storage #

File Storage: keep the bytes off your servers #

What you'll learn #

The problem, explained #

Who uses it. People with several devices who want their files everywhere: upload a report on a laptop, open it on a phone.

Functional requirements.

Non-functional requirements.

What is given. given.proschi declares the user and the bucket blobs (S3 with two partitions). Every cloud has object storage and it scales by itself; the interesting choices are everything around it.

What the tests check.

Back-of-the-envelope #

The problem gives the rates and an average file of 1 MB. A month in the model is 30 days, 2,592,000 seconds.

downloaded per month
≈ 1.3 PB
monthly egress if the bucket sends it
$116,600
monthly egress if the CDN sends it
$25,900
to move 1 MB at the user's 10 MB/s
100 ms
API requests
2.2k rps
new data per month
518 TB
QuantityArithmeticResult
Download bandwidth500 rps × 1 MB500 MB/s
Downloaded per month500 MB/s × 2,592,000 s1,296,000 GB ≈ 1.3 PB
Egress if the bucket sends it1,296,000 GB × $0.09about $116,600/month
Egress if the CDN sends it1,296,000 GB × $0.02about $25,900/month
Bucket reads with 90% CDN hits10% × 500 rps50 rps
Upload bandwidth (ingress, free)200 rps × 1 MB200 MB/s
New data per month200 MB/s × 2,592,000 sabout 518 TB
API requests2k list + 200 start2.2k rps
Metadata writes200 inserts + 200 updates400 rps
Transfer time for 1 MB at the user's 10 MB/s1 MB ÷ 10 MB/s100 ms

Transfer time sets the latency floor. A 1 MB file over 10 MB/s takes 100 ms no matter what. In Proschi, payload transfer is a fixed cost added once to every percentile, so a 200 ms p99 leaves about 100 ms for everything else, and every extra hop that carries the bytes eats into it.

Storage grows fast: about half a petabyte a month. Proschi's cost model has no storage-at-rest price, but in an interview mention lifecycle rules (cold files to cheaper storage classes) and deduplication.

Deep dive How the model sees each piece

Capacity per replica. A load balancer takes 100k rps, a service 2k, PostgreSQL 20k reads but 5k writes on its single primary, a queue 50k, a function 10k, storage 5k. 2.2k API requests a second is 1.1 replicas' worth of work. Divide by your target utilisation (about 70%), then check that one replica fewer still copes: three replicas run at 37%, and two at 55% when one is lost. Metadata writes (400 a second) are far below one primary's 5k, so no sharding.

Bandwidth. Object storage and CDNs "scale out behind one name": the model never saturates them on bandwidth, as in reality.

Availability. A write to a single-primary database with two replicas is modelled at 99.995% (failover costs 10% of the primary's downtime). That is the weakest link for Start upload, still well above 99.9%.

Concepts #

Object storage and pre-signed URLs #

Object storage (S3, GCS, Azure Blob) stores immutable blobs under keys. It is cheap per byte, extremely durable, and scales request rates and bandwidth for you. It is not a database you can query, nor a file system with cheap renames.

A pre-signed URL carries a signature made with your credentials. It allows one kind of operation on one key (say, a PUT of 9a1) and expires after a few minutes. Your API decides who may upload what, signs the URL locally (no network call), and the client then talks to the bucket directly. Your servers never see the bytes, so they need neither the bandwidth nor the long-lived connections a 2 GB upload would hold.

Trade-offs. You cannot inspect the bytes on the way in, so virus scanning and transcoding happen later, on an event. And something must tell your system the upload happened: the next concept. When not to use it: tiny payloads that are really part of an API call (an avatar crop, a 2 KB JSON), where an extra round trip costs more than proxying.

title "Direct-to-bucket upload"

phone  "Phone"     [Actor]
api    "Media API" [REST API]   x2
db     "Media DB"  [PostgreSQL] x2
bucket "Bucket"    [AWS S3]     x2

phone -> api    : HTTPS
phone -> bucket : PUT pre-signed
api   -> db     : SQL

usecase "Get upload URL" {
  phone -> api   : POST /media
  api   -> db    : INSERT media status=pending
  db   --> api   : ok
  api  --> phone : 201 {"putUrl": "https://bucket.example.com/m1?sig=..."}
}

usecase "Put bytes" {
  phone   -> bucket : ~5MB PUT /m1?sig=...
  bucket --> phone  : 200
}

Event-driven completion #

After the PUT, someone must flip the record from pending to ready. The naive answer is "the client calls /complete". But phones lose signal, laptops close, apps crash, and a vanished client leaves a file pending forever.

The robust answer: the bucket announces new objects. S3 event notifications deliver an ObjectCreated event to SQS, SNS, Lambda or EventBridge. Put a queue in between so events survive a slow or failing consumer, and let a worker or function mark the file ready; if it crashes, the queue redelivers. Delivery is at-least-once, so the update must be idempotent: setting status=ready twice is harmless.

Trade-offs. A short delay before the file appears ready; the client polls or is pushed the change. When not to use it: if you need the result synchronously (a server-side validation that must reject the file before the user moves on), you need a synchronous step, usually a proxy or a post-upload check the client waits for.

Deep dive Object events in Proschi
title "Object events"

bucket "Bucket"      [AWS S3]     x2
events "Object Feed" [AWS SQS]    x2
fn     "Indexer"     [AWS Lambda] x2
db     "Catalog"     [DynamoDB]   x2

bucket -> events : ObjectCreated
events -> fn     : trigger
fn     -> db     : write

usecase "Index new object" {
  bucket ->> events : ObjectCreated img/42.jpg
  events ->> fn     : ObjectCreated img/42.jpg
  fn      -> db     : UPDATE item 42 status=indexed
  db     --> fn     : ok
}

CDNs for large downloads #

A CDN is a fleet of caching proxies near users. On a hit, the edge serves the file from its cache; on a miss, it fetches from the origin (here, the bucket), keeps a copy and serves it. This is a pull CDN, the kind the System Design Primer recommends for heavy traffic. Its two benefits are often confused:

BenefitWhat it buys here
Offload and costHits never reach the origin, and CDN egress is cheaper than origin egress: the difference between blowing and fitting the budget.
LatencyEdges are closer, but for big files the user's bandwidth dominates. A 1 MB download takes at least 100 ms at the client's 10 MB/s, whether a CDN or a bucket sends it.

When not to use a CDN. Highly personalised content that is never reused, or tiny traffic where the fixed cost outweighs savings. Private files are fine on a CDN as long as links are signed and short-lived.

Designing it step by step #

1. Scope. Ask about file sizes (up to 2 GB, so streaming through servers is out), the read/write ratio (downloads plus listings far outnumber uploads), who may fetch a file (only someone holding a signed, short-lived link) and what egress costs. Confirm sync, versioning and conflict resolution are out of scope: great follow-ups, not this problem.

2. High-level design. Separate metadata from content. Metadata (folder, name, size, status, object key) is small, structured and queried by folder: a relational database behind an API, behind a load balancer. Content lives in the bucket and only moves between the user, the bucket and the CDN.

Use casePath
List filesuser → load balancer → API → database
Start uploadthe same path, inserting a pending row and answering 201
Uploaduser → bucket, then the bucket's event travels to a finisher
Downloaduser → CDN, and on a miss CDN → bucket

3. Deep dive.

4. Wrap-up. Walk the requirements: upload p99 is the 100 ms transfer plus the bucket's write; download p99 is the transfer plus a CDN hop, with 10% misses adding the origin fetch; nothing is a single replica; the client never sees the database. Then list improvements: multipart uploads for large files (resume after a dropped connection), content hashing for deduplication, a sweeper that deletes pending rows whose upload never arrived, and lifecycle tiers for cold data.

Common mistakes #

Proxying the upload through the API (wrong/upload-through-api). The most common instinct: the client sends bytes to your API, which forwards them to the bucket. In reality each 2 GB upload pins an API connection for minutes, and the API fleet scales with bandwidth instead of requests. In the model, the extra hops (load balancer, API, and a second transfer from API to bucket) push the upload's p99 over 220 ms. It fails File bytes never pass through your servers and p99 of Upload < 220 ms.

The client marks its own upload complete (wrong/client-marks-upload-complete). After the PUT, the client calls POST /files/{id}/complete. It works in the demo, then orphans a pending file every time a phone loses signal at the wrong moment. It fails Uploads are finished without the client: the user calls the load balancer during Upload.

Downloading straight from the bucket (wrong/download-from-bucket). Simple and correct, but every byte leaves at $0.09/GB: the model puts the bill around $119k a month, far over $45k. It fails Downloads are served by the CDN and cost ≤ $45,000/month.

A load balancer where the CDN should be (wrong/load-balancer-instead-of-cdn). A load balancer caches nothing, so every download still reaches the origin, and the bytes leave from a component you run at $0.09/GB: about $119k a month again. It fails Downloads are served by the CDN (the selector any cdn does not match a load balancer) and cost ≤ $45,000/month.

Other classic mistakes.

In the interview #

Open with the two numbers that drive everything, then the principle. Draw the metadata path and the byte path in different colours if you can.

Likely follow-ups:

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems