System design practice

API Gateway

medium Based on a system published by Uber api-gateway caching authentication availability real-world

Uber's Edge Gateway: a stateless pool that authenticates and routes every app call, with sessions cached.

Solve it in your browser Read the lesson first

Uber's apps make every call through one edge tier, the Edge Gateway. Hundreds of backend services, owned by many teams, sit behind it, and none of them is reached by an app directly. The gateway does the work every call needs, the same way for every endpoint: it decodes the request, runs middleware (authentication, bans, rate limits), routes the call to the service that owns the path, and encodes the answer.

Each endpoint is configuration, not hand-written code: a team declares the path, the target service and the middleware it needs, and Uber generates the gateway from that configuration. Every endpoint must use at least one authentication middleware, and when a middleware rejects a call, the rest of the stack is skipped and the middleware's answer goes back.

The catch is that everything depends on the edge. If authentication there calls the auth service for every request, the auth service carries the whole app's load; if the gateway pool cannot lose an instance, a single crash takes every endpoint down at once.

Functional requirements

Three endpoints, each routed to a different service:

Use these use case and scenario names exactly: the traffic, requirements and tests in problem.proschi refer to them.

Scale

Constraints

What is given

problem.proschi declares the rider app and four services owned by other teams: auth, pricing, trips and users, with their instances and capacity fixed by the problem. Add the edge (a load balancer, the gateway pool and the cache of sessions and bans), the connections and the three use cases. The tests find the edge by its kind: [AWS Load Balancer] or any load balancer, [API Gateway] or any gateway, [Redis] or any cache.

What this model leaves out

Based on

How your design is checked

You write the design as text in Proschi. Tests run in your browser: a simulation of the traffic above checks latency, availability, cost and what happens when a machine fails. How the simulation works.

Start designing

Lesson · 13 min read

Learn it: API Gateway #

API gateway: one edge for every call #

What you'll learn #

The problem, explained #

Uber's rider and driver apps talk to hundreds of backend services, owned by many teams. Letting the app call each of them directly would mean every service has to authenticate users, block abusers, limit rates and speak the app's protocols, and every app release would have to know where every service lives. Instead Uber puts one tier at the edge, the Edge Gateway: the apps know a single address, and the gateway forwards each call to the service that owns its path.

The gateway runs the same steps for every call. A protocol layer decodes the request (JSON, Thrift or Protobuf), then a stack of middleware runs: authentication, authorization, rate limits, bans. If a middleware rejects the call, the stack stops there and its answer goes back. Otherwise the endpoint handler maps the request to a backend call, and a generated client sends it with timeouts, retries and a circuit breaker. Endpoints are configuration that a team publishes, from which Uber generates the gateway's Go code.

The exercise keeps three endpoints:

The given has the rider app and four services owned by other teams: auth, pricing, trips and users. You add the load balancer, the gateway pool and a cache that holds sessions and bans. The limits: p99 under 100 ms for every endpoint, 99.95% availability, no latency limit broken by losing any one machine, and at most $2,800 a month. The tests check:

Back-of-the-envelope #

calls through the edge
40k / s
what the auth service can take
6k / s
reach auth with a 95% session hit rate
2k / s
gateway instances: under 70% busy with one gone
7
for the whole edge, of the $2,800 budget
$1,100

All the traffic numbers are assumptions for the exercise; Uber publishes its architecture, not one region's rates. They are scaled down, but the ratios are what matter.

FlowRateReaches
Fare estimates30k / spricing (29.4k after the 2% blocked)
Trip requests4k / strips
Profile reads6k / susers
Every call40k / sload balancer, gateway, cache
Session not cached (5%)2k / sauth

The auth service. Three instances at about 2k calls a second each make 6k. If the gateway asked it about every call, it would get 40k: almost seven times what it can serve. With 95% of sessions in the cache, it gets 5% of 40k, which is 2k a second, a third of its capacity, and still only half after losing one instance.

The gateway. One instance serves about 10k calls a second in the simulation. Four would run at 100%. Five run at 80% and pass on a normal day, but losing one puts the other four at 100%. To stay under about 70% busy with one instance gone, you need 40k ÷ 7k ≈ 6 instances left, so 7 in the pool (57% busy normally, 67% after a failure).

The cache. 40k reads (one per call) and 2k writes (one per miss) a second, on Redis instances that take 100k each: two of them, for redundancy more than for load.

Cost. The four services are fixed at $1,500. Two load balancers ($100), seven gateway instances ($700) and two cache nodes ($300) add $1,100, so $2,600 in all. The budget leaves room for two more gateway instances, not for a pool sized by guesswork.

Concepts #

What an API gateway does #

An API gateway is a reverse proxy that understands your API. It takes every call from the outside, and does the work that is the same for every endpoint:

ConcernWithout a gatewayWith one
AuthenticationEvery service validates tokensOnce, in middleware
Bans and rate limitsEvery service, or nobodyOnce, before routing
RoutingThe app knows every service's addressThe app knows one address
ProtocolsEvery service speaks the app's protocolThe gateway converts
Logs and auditingScatteredOne access log for every call

What stays out: business logic. Uber's previous gateway let product code creep in, and a few APIs' performance problems slowed the whole gateway, while the gateway team became the bottleneck for every product change. The third generation keeps the edge to cross-cutting work and moves product logic behind it.

Middleware that short-circuits #

Middleware are small steps that run in order before the endpoint's handler: authenticate, authorize, check bans, apply a rate limit. Each either lets the call continue or answers it. When one answers, the rest of the stack is skipped, including the backend call.

That is why a ban check belongs at the edge. A blocked rider's call that reaches pricing before anyone notices still costs a backend call, and depends on every service remembering to check. At the edge, it costs one cache read and a 403.

Uber keeps its short-term bans (users, app versions, other identifiers) in a central cache with a TTL, fed by its fraud and security systems. The gateway reads it on the way in.

Serving auth checks from a cache #

Every endpoint needs to know who is calling. If the token is opaque, someone must look it up; if it is signed, someone must check the signature and whether it was revoked. Doing that by asking the auth service makes the auth service carry the whole app's traffic: it becomes the busiest service you have, and a hiccup there fails every endpoint.

A session cache fixes that. When the gateway sees a token, it reads the cache first. On a hit, it has the user id and carries on. On a miss, it asks the auth service, writes the answer to the cache with a short TTL, and carries on. This is plain cache-aside, applied to identity.

The cost is revocation time. A session in the cache stays valid until its entry expires, so a token revoked at the auth service keeps working for up to one TTL. Three ways to live with it:

Uber's post does not say how its authentication middleware serves token checks; the session cache is this exercise's assumption, built on the same pattern as the ban cache the post describes.

Deep dive Cache-aside on another example

The same shape protects any expensive lookup. Here a feature-flag service sits behind a cache:

usecase "Load flags" {
  app -> flagsCache : GET flags:{user}
  alt "Hit" {
    flagsCache --> app : flags
  } alt "Miss" {
    app -> flags      : Evaluate(user)
    app -> flagsCache : SET flags:{user} EX 60
  }
}

Only the misses reach the flag service, so its load is the miss rate times the traffic.

The gateway must not become the bottleneck #

A service that only some calls reach can fail and take only its feature with it. The gateway is different: every call crosses it, so if it slows down, everything slows down, and if it fails, the app is down. Three habits keep it from becoming the single point of failure:

Designing it step by step #

1. Clarify. Which apps, which endpoints, how many calls a second? Who owns authentication, and what can the auth service take? Are there bans or rate limits to enforce, and must a rejected call stop at the edge? What latency and availability does the whole app need, given that everything passes the edge?

2. High level. Draw the app, one load balancer, a gateway pool and the services behind it. There is no line from the app to any service. Each endpoint is a route from the gateway to one service.

3. Middleware. Put the checks in order: read the session and the ban list from the cache in one round trip; on a ban, answer 403; on a miss, ask the auth service, then cache the session. Only then route the call. Write each endpoint's scenarios so the cached case never touches auth.

4. Size it. Count the calls on each tier: the load balancer, the gateway and the cache see every call; the auth service sees only the misses; each backend sees its own endpoint's calls. Size the gateway pool for a failure, not for the average, and keep two of the load balancer and the cache.

5. Check the budget and the tail. Add up the edge's cost against what is left after the services, then look at p99: a cached call is load balancer, gateway, cache and backend; a miss adds an auth call and a cache write.

6. Wrap up. Mention what else the edge gives you once it exists: one access log for auditing, rate limits per user or IP, protocol conversion, client SDKs generated from the same configuration, and a circuit breaker per backend.

Common mistakes #

Asking the auth service about every call (wrong/auth-on-every-call). Correct, and simple: the gateway validates every token with the auth service. That puts 40k calls a second on a service built for 6k; it saturates, and every endpoint's p99 goes with it. Caught by "Sessions are checked from a cache" and the latency limits.

Apps calling each service directly (wrong/apps-call-services-directly). No edge at all: each service validates tokens itself. Every service carries its own copy of the auth logic, the auth service still gets every call, and the app depends on every service's address. Caught by "Every call enters through the edge" and "The session is checked before the call is routed".

A gateway pool with no headroom (wrong/gateway-no-headroom). Five instances handle 40k calls at 80% and pass on a normal day. Losing one puts the remaining four at 100%, and every endpoint in the app fails together. Caught by survive any node failure. Even before a gateway is lost the pool is too tight: at 80% it queues enough that losing a single trips instance pushes Request trip's p99 past 100 ms.

Backends checking the ban list (wrong/backend-checks-bans). The gateway authenticates but leaves bans to each service, so a blocked rider's call is routed to pricing before anyone says no. Caught by "Blocked riders never reach a backend".

Classic mistakes beyond the tests:

In the interview #

Likely follow-ups:

Further reading #

Now design it

Review cards for this problem

Short questions on the ideas it needs, from the review cards. Review them daily so they stick.

More system design problems