Performance Metrics & Vocabulary - Complete Deep Dive

Prerequisites: Back-of-Envelope Estimation, Terminology Used in: every design — this is the language you use to state and defend non-functional requirements

Why this matters more than it looks

Every system design interview opens with non-functional requirements. Candidates say “it should be fast and scalable.” Strong candidates say “p99 read latency under 200ms at 50K rps, 99.9% availability, and I’ll size the thread pools from Little’s Law.”

Same design, completely different signal. This page is that vocabulary.

Latency vs Response time vs Service time

These three get used interchangeably and they are not the same thing.

        ┌──────────────── response time (what the user feels) ─────────────────┐
        │                                                                     │
client →│ network out │ queue wait │ service time │ serialize │ network back │→ client
        │             └── latency (waiting, not working) ──┘                  │

Say this: “Response time is what the user feels; service time is what my server logs. The gap is queueing — and under load queueing dominates, which is why p99 explodes long before CPU reaches 100%.”

⚠️ The classic gotcha: your server-side p99 is 40ms but users complain. The 40ms excludes queue wait at the LB, TLS handshakes, DNS, and mobile radio wake-up. Always be clear about where you measured.

Latency vs Throughput

Not opposites, and improving one usually costs the other.

Latency    = time per operation     (ms/op)     → lower is better
Throughput = operations per second  (ops/s)     → higher is better

Throughput needs a unit before it means anything. Requests, queries, and transactions are three different counts at three different tiers — see QPS vs TPS vs RPS for the fan-out multipliers between them.

The highway analogy that actually works: latency is how long your car takes to travel the road. Throughput is how many cars pass per minute. Adding lanes raises throughput without making your trip faster. Raising the speed limit lowers latency. And at high enough traffic, latency degrades even though throughput is at maximum — that’s congestion, and it’s exactly what happens to a saturated service.

Little’s Law — the single most useful formula

L = λ × W

L = average number of requests in the system (concurrency)
λ = arrival rate (requests per second)
W = average time in the system (response time)

Worked example:

10,000 rps × 0.2 s average response time = 2,000 concurrent requests in flight

→ Thread-per-request model needs ~2,000 threads (at ~1MB stack = 2GB just for stacks)
→ Across 20 servers: 100 threads each
→ Each with a 20-connection DB pool: 400 total DB connections — fine
→ If a downstream slows to 1s: L = 10,000 × 1 = 10,000 in flight
  → 500 threads/server needed; pools exhaust; requests queue; p99 collapses
  → An unrelated endpoint on the same pool now times out too

That last line is the whole point. One slow dependency causes a fleet-wide outage through pool exhaustion, and Little’s Law is how you predict it. It’s also the argument for bulkheads (separate pools per dependency) and for aggressive timeouts.

Say this: “I size pools with Little’s Law: at 10K rps and 200ms, that’s 2,000 requests in flight. I’d also set a timeout budget so a slow downstream can’t inflate W without bound — otherwise concurrency grows until the pool exhausts and takes down endpoints that don’t even use that dependency.”

Percentiles — and why averages lie

Requests (ms): 5, 6, 6, 7, 7, 8, 8, 9, 10, 30000

average = 3,009 ms   ← "our average is 3 seconds!" — nonsense
p50     = 7 ms       ← the truth for typical users
p99     = 30,000 ms  ← the truth for the user who is suffering

⚠️ You cannot average percentiles. Host A’s p99 and host B’s p99 averaged is not the fleet p99. To aggregate you need mergeable structures — histograms with fixed buckets, HDR histograms, or t-digest — then compute the percentile from the merged distribution. This is a genuinely common interview follow-up, and getting it right is a strong signal.

⚠️ Percentiles don’t average over time either. The mean of 60 one-minute p99s is not the hour’s p99.

Tail latency amplification — the fan-out killer

If one request fans out to N services in parallel and waits for all of them, the slowest one determines your response time.

Each service: 99% fast, 1% slow (p99 tail)

N = 1  → P(no slow call) = 0.99      → 1%  of requests are slow
N = 10 → P = 0.99^10  ≈ 0.904        → ~10% of requests are slow
N = 100→ P = 0.99^100 ≈ 0.366        → ~63% of requests are slow (!)

A p99 problem in one service becomes a p37 problem for the user at 100-way fan-out. This is why large-scale systems obsess over tail latency rather than average.

Mitigations, in order of how often they’re the right answer:

Say this: “With fan-out I’d worry about tail amplification — 10 parallel calls each with a 1% slow tail means roughly 1 in 10 user requests is slow. I’d propagate a deadline, return partial results past 150ms, and consider hedged requests on the read path where the work is idempotent and cheap.”

Utilization and the queueing wall

The single most counterintuitive fact in performance: queue wait grows non-linearly with utilization and goes vertical near 100%.

For an M/M/1 queue:

Average wait W = S / (1 - ρ)        where S = service time, ρ = utilization

ρ = 50%  →  W = 2 × S       (wait equals service time)
ρ = 80%  →  W = 5 × S
ρ = 90%  →  W = 10 × S
ρ = 95%  →  W = 20 × S
ρ = 99%  →  W = 100 × S     ← the wall

latency
   ↑                                    ║
   │                                   ║
   │                                 ╱
   │                            ╱
   │____________________╱
   └──────────────────────────────────→ utilization
   0%       50%       80%   90%  95% 99%

Practical consequences:

• Target 60-70% utilization, not 95%. “We’re only at 70% CPU, why is it slow?” — because at 70% you’re already waiting 2.3× service time, and a traffic spike puts you on the vertical part of the curve.
• Variance makes it worse. The formula above assumes exponential service times. Bursty arrivals (which real traffic is) and high service-time variance push the wall left.
• This is why autoscaling on 80% CPU is late. By the time the new instance boots (60-180s), you’re at 95%+ and p99 has already blown the SLO. Scale on queue depth or p99 latency, and keep headroom.

Throughput vocabulary

Availability math

The nines

⚠️ Five nines is less than one bad deploy per year. You cannot reach it with a human in the recovery path — it requires automated failover that’s faster than a page. If you claim five nines in an interview, be ready to justify it.

Serial dependencies multiply

Service chain: gateway → auth → orders → inventory → payments
Each at 99.9%:

0.999^5 = 0.995  →  99.5%  →  1.8 DAYS of downtime per year

This is the strongest technical argument against deep synchronous call chains. Five hops of “very reliable” services produce a mediocre system.

Redundancy adds nines (if failures are independent)

Two independent instances at 99% each:
P(both down) = 0.01 × 0.01 = 0.0001  →  availability 99.99%

⚠️ “Independent” is doing enormous work in that sentence. Same AZ, same deploy, same config push, same certificate expiry, same poisoned cache entry, same dependency — correlated failure eats the benefit. Say this out loud: “redundancy only adds nines to the extent the failures are actually independent, which is why I’d spread across AZs and stagger deploys.”

SLI vs SLO vs SLA vs Error budget

Three rules worth knowing:

1. SLA is always looser than SLO. You need internal headroom before money is at stake.
2. The error budget is a budget — spend it. Untouched budget means you’re over-investing in reliability and under-investing in features. Exhausted budget means freeze deploys until it recovers.
3. A good SLI measures the user’s experience, not your server’s comfort. “CPU < 80%” is not an SLI.

Say this: “I’d define the SLO on the user-facing indicator — p99 latency and success rate at the edge — and use the error budget to govern release velocity. That keeps the reliability conversation quantitative instead of ‘the site feels slow.’”

Failure and recovery metrics

Availability = MTBF / (MTBF + MTTR)

The insight: halving MTTR improves availability exactly as much as doubling MTBF — and it’s usually 10× cheaper. Better alerting, faster rollback, and rehearsed runbooks beat heroic engineering to prevent all failure.

Latency numbers to have memorized

The three ratios that actually matter in an interview:

• Memory is ~100× faster than SSD, SSD is ~100× faster than a disk seek. That’s why caching works.
• Sequential is ~100× faster than random on any storage. That’s why LSM trees, WALs, and Kafka append.
• Cross-continent round trips are ~200ms and physics won’t fix it. That’s why you need CDNs and regional replicas — no amount of optimization beats the speed of light.

Interview application

Stating non-functional requirements: “Reads: p99 < 200ms at 50K rps. Writes: p99 < 500ms at 5K rps. Availability 99.9% for reads, 99.99% for the payment path. Consistency: strong for balances, eventual (< 2s) for feeds. Peak-to-average 3×, so I’ll provision for 150K rps read capacity with 30% headroom.”

Defending a design: “I keep the read path to two hops because five 99.9% services in series is 99.5% — a day and a half of downtime a year. Anything not on the critical path moves to async via a queue.”

Sizing: “At 50K rps with 40ms service time, Little’s Law gives 2,000 concurrent requests. With 200 concurrent per instance that’s 10 instances, and I’d run 15 for headroom and AZ loss tolerance.”

Common Interview Questions

Q: “What latency should we target?” A: “Depends on the interaction. Under 100ms feels instant; 100-300ms feels responsive; past 1s users notice and start abandoning. I’d set p99 < 200ms for the read path and be explicit that I’m measuring at the edge, not server-side.”

Q: “Why p99 and not average?” A: “Averages hide the tail — one 30s timeout disappears into thousands of 5ms requests. At 10K rps, p99 is 100 users every second having a bad experience. And with fan-out, a 1% tail per service becomes a 10% tail for the user across 10 parallel calls.”

Q: “How do you compute p99 across 50 servers?” A: “You can’t average their p99s. Each host exports a histogram (or t-digest), you merge the distributions centrally, then compute the percentile. Averaging percentiles is a common and wrong shortcut.”

Q: “Your CPU is at 70% and latency is bad. Why?” A: “Queueing. At 70% utilization average wait is already about 2.3× service time, and the curve is non-linear — bursty arrivals put you briefly at 95%+ where wait is 20× service time. Also check whether you’re actually CPU-bound: lock contention, GC pauses, or a saturated connection pool all produce latency at moderate CPU.”

Q: “Latency vs throughput — can you have both?” A: “Caching gives you both. Almost everything else trades: batching raises throughput and hurts latency, and adding capacity raises throughput without changing per-request latency. The one thing that helps both is removing work — fewer hops, fewer bytes, fewer round trips.”

Q: “How much capacity do you provision?” A: “Peak, not average — typically 2-5× average for consumer traffic — targeting 60-70% utilization at peak so I stay off the vertical part of the queueing curve, plus enough spare that losing an AZ doesn’t push the survivors past that line.”

Q: “What’s the difference between an SLO and an SLA?” A: “SLO is the internal target, SLA is the contractual promise with penalties. The SLO is always stricter, so you find out you’re in trouble before your customers get a credit.”

Decision Framework

Where This Shows Up

This concept is load-bearing in these designs - each link goes straight to the design that leans on it:

• Design a Metrics and Monitoring System - percentiles computed over a stream, and why averages hide the problem
• Design a Twitter Feed - a read-latency budget that forces fan-out on write
• Design a URL Shortener - a redirect latency target that decides the whole storage choice