Designing a URL Shortener Like Bitly or TinyURL

Difficulty: Beginner Prerequisites:Caching and CDN

TL;DR

A URL shortener maps short codes to long URLs and redirects billions of clicks per day using tiered caching (CDN → Redis → DB).

flowchart LR
    USER["Browser"]:::client
    CDN["CDN Edge"]:::edge
    RS["Redirect Service"]:::service
    REDIS[("Redis Cache")]:::data
    DB[("Database")]:::data

    USER -->|"1. Click short URL"| CDN
    CDN -->|"2. Cache miss"| RS
    RS -->|"3. Lookup short code"| REDIS
    RS -->|"4. Fallback read"| DB

    classDef client fill:#4c3a5e,stroke:#818cf8,color:#e2e8f0
    classDef edge fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    classDef service fill:#1a3a2a,stroke:#4ade80,color:#e2e8f0
    classDef data fill:#3b3520,stroke:#fbbf24,color:#e2e8f0

In 3 sentences: User submits a long URL → system generates a unique short code (base62 of a Snowflake ID) → stores the mapping. On redirect, the system looks up the short code through CDN → Redis → DB tiers and returns a 302. Analytics events fire async to Kafka without slowing the redirect.

💡 Base62 encoding uses [0-9a-zA-Z] (62 characters) to represent numbers compactly. A 7-char Base62 string gives 62^7 = 3.5 trillion unique short codes.

Understanding the Problem

🔗 What is a URL shortener? A service that takes a long URL like https://example.com/products/2024/summer-sale?utm_campaign=email&ref=newsletter and gives you back a short one like https://sho.rt/aB3xY9. When someone clicks the short link, they’re redirected to the original. Despite the tiny API surface, a URL shortener has to handle billions of redirects a day, generate unique codes without collisions, survive traffic spikes on viral links, and provide analytics.

Think Bitly, TinyURL, t.co (Twitter), or Google’s short-lived goo.gl.

Naive First Cut

A 30-second whiteboard sketch:

Receive a URL, generate a random code, store {short, long} in a DB, look it up on redirect. Works for 100 users. Breaks at scale:

• Random code generation collides more often than you’d think at 100M+ links.
• Every redirect is a DB read, so 1M redirects/sec melts the DB.
• Hot links (a tweet goes viral) hammer a single row.
• No analytics beyond “it existed.”
• One DB region → half the planet sees 200ms+ redirect latency.

The rest of this doc evolves this into a system that serves billions of redirects globally at low latency.

Prior Art We’re Drawing From

• Bitly - the canonical URL shortener. Public writeups describe a heavy read-cache tier, base62 encoding over numeric IDs, and a dedicated analytics pipeline separate from the redirect hot path. (Educative overview)
• Twitter Snowflake - 64-bit distributed ID generator producing time-ordered unique IDs without coordination per request. Widely adopted as an alternative to DB auto-increment for short-code generation. (paper / repo)
• Base62 encoding - the standard way to turn a numeric ID into a short alphanumeric code. 7 characters of base62 give 3.5 trillion unique codes, enough for decades of links.
• Counter-based ranges (Zookeeper / DB) - pre-allocate ranges of IDs to each service instance to avoid per-request coordination. Instagram’s photo ID generation is a famous variant.
• CDN-cached 301 redirects - t.co and goo.gl both served redirects from edge POPs. Bitly uses its own edge infrastructure with aggressive caching.

Functional Requirements

Core Requirements

1. Users should be able to submit a long URL and get back a unique short URL.
2. Anyone with a short URL should be redirected to the original long URL, fast.
3. Short URLs should have a configurable expiry (default: forever) and support click analytics.

Below the line (out of scope)

• User accounts, API keys, billing, dashboards
• Custom aliases / vanity URLs (simple extension once core works)
• Password-protected or expiring-on-click links
• Real-time dashboards with sub-second freshness
• QR codes, link previews, malware scanning
• Custom domains per customer

Non-Functional Requirements

Core Requirements

• Read-heavy workload - 100:1 reads to writes. Every design decision is driven by the redirect hot path.
• Low redirect latency - P99 under 100ms globally. This is user-facing; slow redirects feel broken.
• High availability - 99.99% on the redirect path. A dead redirect breaks someone else’s tweet.
• Uniqueness - no two long URLs can accidentally share a short code.
• Scale - 100M new links/day, 10B redirects/day, 5-year retention → ~200B rows at peak.

In simple terms: ten billion clicks a day, from everywhere on earth, each expected back in under 100ms. Read that list again and notice that none of it is about shortening URLs — the hard part of this system is entirely on the redirect path, and it is where every deep dive ends up.

Below the line

• Strong consistency on analytics counts (eventual is fine)
• Strict ordering of click events
• Sub-100ms link creation latency (creation is rare; can be 500ms)

Scale Estimation (Back-of-Envelope)

• Users: 100M DAU (link creators + clickers combined)
• Write QPS: 1K new URLs/sec (100M new links/day)
• Read QPS: 100K redirects/sec (100:1 read-write ratio, 10B redirects/day)
• Storage: 500GB URL mapping data/year (~200B rows at 5-year retention)
• Bandwidth: ~50 Gbps at peak for redirect responses + analytics event ingestion

Core Entities

• Long URL - the original destination URL the user submitted.
• Short Code - the unique alphanumeric suffix (aB3xY9) that identifies a mapping.
• Link - the stored mapping of short_code → long_url with metadata (creator, created_at, expires_at).
• Click Event - a record of a single redirect, with timestamp, IP-derived country, referrer, user agent.

API / System Interface

POST /v1/links                                 → Link
     Body: { longUrl, customAlias?, expiresAt? }
     Header: Authorization: Bearer <api_key>

GET  /:shortCode                               → 302 redirect
     Public endpoint on the short domain (sho.rt/aB3xY9)

GET  /v1/links/:shortCode/stats                → ClickStats
     Aggregated clicks by time bucket, country, referrer

DELETE /v1/links/:shortCode                    → 204
     Soft delete - short code becomes 410 Gone

Security notes:

• Creation is authenticated via API key; redirects are public.
• customAlias (if supported) must be validated for reserved words and rate-limited per user.
• Submitted URLs should be screened for phishing/malware (out of scope for this HLD, but hook point needed).
• Never trust the Referer header for anything beyond analytics; it’s user-controlled.

High-Level Design

Let’s build up service by service.

1) User creates a short URL

A long URL comes in, a short code goes back, and the pair is remembered. Creation runs at about 1.2K/sec, which is a rounding error next to the read path, so this is the part of the system where we can afford to be boring.

Build the boring version. A gateway, a service, and one Postgres table. The short code is the row’s own auto-increment id, base62-encoded — the database is already handing us a unique number on every insert, so taking it costs nothing and needs no new component.
💡 Base62 = writing a number using 0-9, a-z and A-Z. It packs a big integer into few characters, which is why 7 characters covers 62^7 ≈ 3.5 trillion codes.

New components we need:

1. API Gateway - the front door. Authenticates API keys, applies per-user rate limits, and routes requests.
   💡 Think of it as a security guard plus receptionist for your backend.
2. Write Service - validates the URL, inserts the row, encodes the returned id, and hands back the short link.
3. Postgres links table - id, long_url, user_id, created_at, expires_at. The id column is both the primary key and the source of the short code.

flowchart LR
    CLIENT["Client"]:::client
    GW["API Gateway"]:::edge
    WS["Write Service"]:::service
    DB[("Postgres links table")]:::data

    CLIENT -->|"1. POST a long URL"| GW
    GW -->|"2. Auth and rate limit"| WS
    WS -->|"3. Insert row and return id"| DB
    WS -->|"4. Base62 encode the id"| CLIENT

    classDef client fill:#4c3a5e,stroke:#818cf8,color:#e2e8f0
    classDef edge fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    classDef service fill:#1a3a2a,stroke:#4ade80,color:#e2e8f0
    classDef data fill:#3b3520,stroke:#fbbf24,color:#e2e8f0

Step-by-step flow:

1. User calls POST /v1/links with a long URL and an API key
2. The Gateway checks the key is valid and the user is inside their rate limit
3. The Write Service runs INSERT INTO links (long_url, ...) RETURNING id
4. It base62-encodes that id into a 7-character code and returns sho.rt/aB3xY9
5. No uniqueness check is needed anywhere, because the database’s own sequence cannot issue the same id twice

Why derive the code from the row id instead of generating a random string? A random string has to be checked against what already exists, so every creation costs a read before its write, and as the table fills the retry rate climbs. Deriving the code from the id means uniqueness is a property of the insert rather than something we verify.

What we have deliberately left broken. This creates working short links, and it makes two commitments we cannot keep:

• Every code in the system comes from one sequence. A single primary hands out every id, so link creation has exactly one bottleneck and no way to add a second writer without risking a duplicate code. That is Deep Dive 1.
• The codes are a sequence, so they are guessable. aB3xY9 is followed by aB3xYA. Anyone can walk the space and enumerate every link ever created, and because short links are unlisted rather than access-controlled, people treat them as private. Enumeration is a disclosure bug, not an inconvenience. That is also Deep Dive 1.
• 200 billion rows are going in one Postgres table. At 100M links/day over the 5-year retention the requirements state, this table does not fit on one machine. That is Deep Dive 5.

2) Anyone hits the short URL and gets redirected

Someone clicks sho.rt/aB3xY9. We decode the code back to an id, look up the row, and tell the browser where to go. Functionally that is one read and one HTTP header.

New components we need:

1. Redirect Service - decodes the short code, reads the row, and returns a 302 Found with the long URL in the Location header.

Nothing else. No cache and no CDN yet: both are answers to the non-functional latency and throughput targets rather than to the requirement, and asserting them here would leave the deep dives with nothing to argue.

flowchart LR
    USER["Browser"]:::client
    RS["Redirect Service"]:::service
    DB[("Postgres links table")]:::data

    USER -->|"1. GET the short code"| RS
    RS -->|"2. Read row by id"| DB
    DB -->|"3. Return the long URL"| RS
    RS -->|"4. Respond 302 Location"| USER

    classDef client fill:#4c3a5e,stroke:#818cf8,color:#e2e8f0
    classDef service fill:#1a3a2a,stroke:#4ade80,color:#e2e8f0
    classDef data fill:#3b3520,stroke:#fbbf24,color:#e2e8f0

Step-by-step flow:

1. Browser issues GET /aB3xY9
2. The Redirect Service base62-decodes aB3xY9 back to the numeric id — no lookup table needed, the code is the key
3. It reads that row from Postgres by primary key
4. If the row is missing or past expires_at, return 404
5. Otherwise return 302 Found with Location: <long_url>

Why 302 and not 301? A 301 is a permanent redirect, so the browser caches it and goes straight to the destination on every later click — we never see it, and the click analytics in requirement 3 quietly stop working. 302 keeps us in the path. It is worth being explicit that this is a deliberate trade of some redirect latency for the ability to count, and it is also what makes the caching in Deep Dive 2 harder than it first looks.

What we have deliberately left broken. One indexed primary-key read per redirect is about as cheap as a database read gets, and it is still nowhere near enough:

• The volume is 100x what creation is. 10B redirects/day is roughly 120K/sec average and 500K/sec at peak, all landing on the single Postgres primary that FR1 is also writing to. That instance cannot serve it, and read replicas move the problem without solving the next point.
• Distance is most of the latency budget. The requirement is a P99 under 100ms globally. A user in Singapore hitting an origin in us-east-1 spends around 200ms on the network before the query is even parsed, so the target is unreachable from one region no matter how fast the database is.

Both are Deep Dive 2. A single link going viral concentrates all of that on one row, which survives the fixes in Deep Dive 2 and is Deep Dive 3.

3) Expiry and click analytics

Requirement 3 is two small things: a link can expire, and the owner can see how many people clicked it. Expiry is already done — FR2 step 4 checks expires_at, and a column plus a comparison is the whole feature.

That leaves counting. The simplest thing that counts is a counter.

New components we need:

1. Stats API - returns the click count for a link to its owner.

Plus one column on the existing table: click_count. The Redirect Service increments it as part of serving the redirect.

flowchart LR
    USER["Browser"]:::client
    RS["Redirect Service"]:::service
    DB[("Postgres links table<br/>with click count")]:::data
    API["Stats API"]:::service
    OWNER["Link owner"]:::client

    USER -->|"1. GET the short code"| RS
    RS -->|"2. Read row and bump count"| DB
    RS -->|"3. Respond 302"| USER
    OWNER -->|"4. Ask for stats"| API
    API -->|"5. Read click count"| DB

    classDef client fill:#4c3a5e,stroke:#818cf8,color:#e2e8f0
    classDef service fill:#1a3a2a,stroke:#4ade80,color:#e2e8f0
    classDef data fill:#3b3520,stroke:#fbbf24,color:#e2e8f0

Step-by-step flow:

1. A redirect arrives and is served exactly as in FR2
2. In the same transaction, the service runs UPDATE links SET click_count = click_count + 1 WHERE id = ?
3. The 302 goes back to the browser
4. The owner calls GET /v1/links/:code/stats
5. The Stats API reads click_count off the row and returns it

Why keep the counter on the link row rather than in its own table? Because at this stage the only question anyone asks is “how many clicks?”, and that answer lives one primary-key read away. Inventing a clicks table now would mean a COUNT(*) over billions of rows to answer it, which is slower and more complex for no gain in what we can report.

What we have deliberately left broken. The count is accurate and the way we get it is indefensible:

• A read path just became a write path. FR2’s redirect was a single indexed read; now every one of the 120K redirects/sec also writes a row, and that write is inside the request the user is waiting on. We have taken the highest-volume, most latency-sensitive path in the system and given it a database mutation.
• Every click on one link contends on one row. A popular link means thousands of transactions per second all trying to update the same row, serialising behind each other on the same lock. The more successful the link, the slower its own redirects get.
• The count is all we will ever be able to report. A single integer cannot answer “how many from Germany last Tuesday”, and there is nowhere in this design that the information to answer it is being kept.

All three are Deep Dive 4.

Technology Choices

Each component in this design is generic - here are real options for each. Pick per your stack and ops capacity.

Rule of thumb: managed > self-hosted when the workload isn’t a differentiator. We’re not in the “run Kafka” business.

Data Modeling

DynamoDB / Cassandra (Global KV — source of truth for short_code → URL mapping):

Table: links
  PK: short_code (string, 7-char base62)
  Attributes: long_url, user_id, created_at, expires_at, status (active/deleted)
  
GSI: user_links (for "show all my links")
  PK: user_id
  SK: created_at (DESC)

Redis (Regional Cache — hot redirect lookups):

Key: "link:{shortCode}" → String (long_url)
TTL: 24 hours
Eviction: LRU when memory pressure

CDN (Edge Cache):

Cache key: URL path (e.g., /aB3xY9)
Cache-Control: public, max-age=300
Value: 302 redirect response with Location header

Kafka (Click Events — analytics pipeline):

Topic: link-clicks
  Key: short_code (partition by link for ordering)
  Value: { short_code, timestamp, ip_country, referrer, user_agent, device_type }

ClickHouse (Serving Store — pre-aggregated analytics):

CREATE TABLE click_rollups (
    short_code String,
    time_bucket DateTime,   -- 1-hour granularity
    country String,
    referrer_domain String,
    click_count UInt64
) ENGINE = SummingMergeTree()
ORDER BY (short_code, time_bucket, country, referrer_domain);

Access Patterns:

How the Bloom Filter Rejects Invalid Codes:

1. All valid short codes are inserted into an in-memory Bloom filter on each Redirect Service pod (rebuilt periodically from DB scan)
2. On redirect request: probe the Bloom filter first. If “definitely not present” → return 404 immediately (no Redis/DB hit)
3. If “maybe present” → proceed to Redis/DB lookup normally
4. At 200B links, the Bloom filter is ~2GB RAM with 0.1% false positive rate — a cheap way to reject 30%+ of requests (bots, typos, scanners)

Deep Dives

1) How do we generate short codes at 1K/sec without collisions or letting people enumerate them?

Problem. Every code must be globally unique, and at 100M links/day we mint about 1,200/sec sustained. FR1 got uniqueness for free and paid for it twice over.

Bad - the database’s auto-increment id, base62-encoded. This is what FR1 built, and the uniqueness really is free: one sequence, no coordination, no collision possible. Two things are wrong with it.

• One sequence means one writer. Every code in the system comes from a single primary’s counter, so creation cannot be scaled out horizontally — a second writer would either need the same sequence, reintroducing coordination on every insert, or its own, which can hand out an id the first has already used. The write path has a hard ceiling and no way past it.
• Sequential codes are enumerable. aB3xY9 is followed by aB3xYA. Anyone can walk the keyspace and harvest every link ever shortened, and since short links are unlisted rather than permissioned, people put things behind them they would not put on a public page. This is a disclosure bug and it is the more serious of the two.

The other instinct — take an MD5 or SHA-256 of the long URL and keep the first 7 characters — is worse on both counts. Truncating 128 bits to the 42 that fit in 7 base62 characters makes collisions likely enough that you need a check-and-retry loop, so writes cost an unpredictable number of round trips. It also makes the same URL always produce the same code, which quietly means two unrelated users sharing one link and one set of click stats. Only reach for hashing if “same URL always gives the same code” is a stated product requirement.

Good - hand out ids in pre-allocated ranges. Keep the numeric-id idea but stop asking the database for one id at a time. Each Write Service instance claims a block of 1M ids up front and serves from that block in memory, returning for another when it runs low. Coordination drops from once per write to once per million, so the single-writer ceiling disappears, and an instance can keep minting codes through a brief outage of the allocator. Codes are still sequential within a block, so the enumeration problem is untouched.

Great - distributed ID generation with Snowflake + base62 + bit-shuffling.

Options, each with a use case:

1. Twitter Snowflake - 64-bit ID = [41 bits timestamp][10 bits machine ID][12 bits sequence]. Each service pod generates its own IDs, no coordination, time-ordered, collision-free across ~1K machines. Base62-encode the lower 42 bits for a 7-char code. Used in production by Twitter, Instagram, Discord.
2. Zookeeper range allocator - a service instance requests a range of 1M IDs at once from Zookeeper, burns through them locally, and requests the next range. Handles DB unavailability, no per-request coordination.
3. Database with pre-allocated ranges - same as (2) but using a single counter table row with SELECT ... FOR UPDATE SKIP LOCKED. Simpler infra, slightly slower.

To defeat enumeration scraping, we can bit-shuffle the numeric ID before base62-encoding using a fixed, reversible permutation (Feistel network or multiply-by-large-prime mod 2^42). Codes become unpredictable without adding any lookup cost - we still decode back to the real ID by reversing the permutation.

For custom aliases (if we support them), we use a separate write path: first try INSERT ... ON CONFLICT DO NOTHING on the alias; reserve reserved words (admin, api, login) in a denylist.

2) How do we serve 10B redirects a day at under 100ms P99 globally?

Problem. 10B redirects/day is ~120K/sec average and ~500K/sec at peak, against a P99 budget of 100ms for users anywhere on earth.

Bad - what FR2 built: one primary-key read against the single origin database. The query itself is not the problem; it is an indexed point lookup and it is genuinely fast. The problem is that it is the only thing absorbing 500K requests/sec, on the same instance FR1 writes to and FR3 now updates a counter on, so the read path is competing with both. Adding read replicas raises the ceiling and does nothing about the second half of the budget: a request from Singapore to an origin in us-east-1 spends roughly 200ms in flight before any query runs. The target is 100ms. No amount of database tuning closes a gap that is made of distance, so the data has to stop being in one place.

Good - Redis read-through cache in front of the DB. Check Redis first on every redirect. Cache hit → return instantly (sub-ms). Cache miss → read from DB, backfill Redis, return. Since links are mostly immutable (write-once, read-forever), cache hit rate climbs to 95%+ quickly. Most traffic never touches the DB.

Great - CDN layer in front of Redis + DB. For truly hot links (viral tweets), the CDN edge (Cloudflare, CloudFront, Fastly) serves the 302 redirect directly from the user’s nearest edge server — without ever reaching your origin. Sub-10ms worldwide.

flowchart LR
    USER["Browser"]:::client
    CDN["CDN Edge"]:::edge
    RS["Redirect Service"]:::service
    REDIS[("Redis Cache")]:::data
    DB[("Database")]:::data

    USER --> CDN
    CDN -->|"miss"| RS
    RS -->|"Lookup short code"| REDIS
    RS -->|"miss"| DB

    classDef client fill:#4c3a5e,stroke:#818cf8,color:#e2e8f0
    classDef edge fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    classDef service fill:#1a3a2a,stroke:#4ade80,color:#e2e8f0
    classDef data fill:#3b3520,stroke:#fbbf24,color:#e2e8f0

The lookup path: CDN (edge, ~5ms) → Redis (in-memory, ~1ms) → DB (disk, ~10ms). Each tier absorbs traffic so the next one sees less load. Cache invalidation is simple since links are immutable — set a TTL (e.g., CDN 5 min, Redis 24h) and on delete, purge both.

3) One link gets 100K clicks a second. What happens the moment its cache entry expires?

Problem. Deep Dive 2 put a CDN and a Redis tier in front of the database, and in the steady state that handles a viral link comfortably — the edge serves nearly everything. The hazard is not the steady state. It is the instant the cached copy stops being valid, because that instant is shared by every edge at once.

Bad - lean on the cache tiers as built, and add Redis nodes if it hurts. With a 5-minute CDN TTL, all edges holding that link expire within the same moment. Every one of them misses simultaneously and forwards to origin, so a link running at 100K clicks/sec delivers something close to 100K requests/sec to one Redis key, and on a Redis miss, to one Postgres row. Adding Redis nodes cannot help: consistent hashing maps a given key to exactly one node by design, so the extra capacity sits idle while the one node holding the hot key saturates. This is the thundering-herd shape — traffic that the cache was absorbing arrives all at once because the cache expiry was synchronised. Worse, the requests pile up behind the single fill, so a link gets slower precisely when it is most popular, and the retry storm from timed-out clients adds to the load that caused it.
💡 Cache stampede = many simultaneous misses for the same key, all racing to recompute the same value.

Good - a local LRU in every pod, and jittered TTLs. Give each Redirect Service pod a small in-process cache, say 10K entries with a 60-second TTL. A viral code is served from process memory without a network hop at all, which turns the hottest keys into the cheapest requests and takes them off Redis entirely. Then stop the misses lining up: set each cached entry’s TTL to its nominal value plus a random offset, so expiries spread across a window instead of firing together. Jitter converts one 100K/sec spike into a manageable trickle of refreshes.

Great - collapse duplicate misses instead of just spreading them. Add single-flight at each tier: when a pod takes a miss for a code, the first request goes to fetch it and every other request for that same code waits on that one in-flight fetch rather than starting its own. A thousand concurrent misses become one origin read and 999 waiters, and the guarantee holds no matter how badly the jitter is tuned. Pair it with stale-while-revalidate — serve the expired copy immediately while refreshing in the background — so an expiry stops being a latency event for anybody. Because short links are immutable once created, serving a slightly stale copy is risk-free here in a way it would not be for mutable data, which is what makes this the right tool rather than a compromise.

4) How do we count every click without making the redirect pay for it?

Problem. Requirement 3 asks for click analytics. FR3 delivered them by doing the counting inside the redirect, which is the one request in this system that cannot afford extra work.

Bad - what FR3 built: UPDATE links SET click_count = click_count + 1 in the redirect transaction. This turns the read path into a write path. FR2’s redirect was a single indexed read that a replica or a cache could serve; now all 120K redirects/sec must reach the primary, because a cache cannot absorb a write. Everything Deep Dive 2 and Deep Dive 3 just built is undone by it — a CDN cannot serve a request that has to increment a counter.

It is also self-throttling in the worst place. Every click on one link is a transaction against one row, so a popular link’s redirects serialise behind each other on that row’s lock, and the more traffic a link gets the slower it gets. A link at 10K clicks/sec is asking Postgres for 10K serialised updates/sec to a single row.

And the reporting ceiling is a single integer. “How many clicks from Germany last Tuesday” is unanswerable, not because the query is hard but because the information was thrown away at the moment of the click.

Good - fire the click onto a queue and return. The redirect publishes a small event and answers the browser without waiting; a consumer applies the counts behind the scenes. The hot path goes back to being read-only and cacheable, the count becomes eventually consistent by a second or two, and because the event carries the full request context — country, referrer, user agent, timestamp — the reporting ceiling disappears with it. The redirect no longer has any idea how expensive analytics is, which is the point.

Great - event bus + stream processor for rollups + object storage for raw events + columnar serving store.

• Every click becomes a Protobuf message on the event bus (Kafka / Kinesis / Pub/Sub) topic link-clicks, keyed by short_code for partition-level ordering.
• A stream processor (Flink / Kafka Streams / Spark Structured Streaming) maintains tumbling-window aggregates: 1-min, 1-hour, 1-day buckets per (short_code, country, referrer) tuple. Writes to a columnar serving store (ClickHouse / Pinot / Druid / Timestream) for fast per-link queries.
• Same stream is tee’d via a sink connector to object storage (S3 / GCS) as hourly Parquet files - for ad-hoc queries via Athena / BigQuery / Trino, ML training, and long-term retention.
• Stats API reads only pre-aggregated rollups. GET /links/:code/stats?range=30d runs in milliseconds.

Fully-serverless shortcut if team wants less ops: swap Kafka+Flink for Kinesis Data Streams + Kinesis Data Firehose + Kinesis Data Analytics, or Pub/Sub + Dataflow on GCP.

Eventual consistency tradeoff is explicit: dashboard numbers lag the true count by ~1-2 minutes. For link analytics this is never a problem; nobody sits watching their click count refresh every second expecting real-time precision.

5) Where do 200 billion rows live?

Problem. 100M new links/day at ~1,200 writes/sec is modest. Keeping them for the 5 years the requirements specify is not: that is roughly 200B rows in one table.

Bad - what FR1 built: one Postgres table, with read replicas added when it hurts. A single instance is fine for the write rate and loses to the row count. The primary key index over 200B rows is far too large to stay in memory, so the point lookups that FR2 depends on stop being cache hits and start being disk seeks, and the tail latency the requirements care about degrades as a function of how successful the product has been. Replicas do not help, because nothing here is bottlenecked on read capacity — they each carry the same oversized index. The operational side is worse than the performance side: a table that large takes many hours to back up and restore, VACUUM and index rebuilds stop fitting in any maintenance window, and adding a column becomes an outage. The failure is gradual, which is what makes it dangerous — nothing breaks on a particular day, it just gets steadily slower until it is unfixable without a migration.

Good - shard Postgres by short_code hash. N shards, each smaller and faster. Application routes by hash. Works but operationally heavy - you now own a sharding layer, cross-shard queries, and painful resharding.

Great - pick a store built for this: DynamoDB, Cassandra, or TiDB.

The access pattern is a perfect fit for a KV store:

• Writes: PUT short_code → {long_url, user_id, created_at, expires_at} - no joins, no transactions beyond a single row.
• Reads: GET short_code - point lookup by primary key.
• Deletes: UPDATE status = 'deleted' - also point-key.

No relational queries on the redirect path. DynamoDB (fully managed, auto-scales, Global Tables for multi-region), Cassandra (self-managed, cheaper at massive scale), or TiDB (SQL-compatible if you want it) all fit.

Schema:

PK: short_code  (partition key, hashed)
Attrs: long_url, user_id, created_at, expires_at, status

User-side queries like “all links created by user X” are a different, secondary access pattern. Serve them from a GSI (global secondary index) on user_id or a separate materialized view updated via CDC. Don’t warp the primary schema for it.

Core Flows

Flow 1 - Create a short link

sequenceDiagram
    autonumber
    participant C as Client
    participant GW as API Gateway
    participant WS as Write Service
    participant ID as ID Generator
    participant DB as Links DB

    C->>GW: POST v1 links with API key
    GW->>GW: auth and rate limit
    GW->>WS: forward
    WS->>WS: validate URL format and deny list
    WS->>ID: next id
    ID-->>WS: snowflake id
    WS->>WS: base62 encode and bit shuffle
    WS->>DB: put short_code to long_url
    DB-->>WS: success
    WS-->>C: 201 with short URL

1. Client submits the long URL with an API key.
2. Gateway authenticates, rate-limits, and forwards.
3. Write Service validates URL format, blocks obvious junk and phishing denylist.
4. Asks ID Generator for a fresh Snowflake ID; base62-encodes with bit-shuffling so the code isn’t enumerable.
5. Writes to the KV store. Collisions are impossible - Snowflake IDs are unique by construction.
6. Returns the short URL.

Failure worth calling out: if the KV store write fails, we retry with the same ID (the ID has already been generated, there’s no benefit to burning a new one). If it keeps failing, the ID is silently wasted - Snowflake has 42 bits of address space so waste is irrelevant.

Flow 2 - Redirect a short link

sequenceDiagram
    autonumber
    participant U as Browser
    participant CDN as CDN Edge
    participant RS as Redirect Service
    participant BF as Bloom Filter
    participant L as Local Pod Cache
    participant R as Regional Redis
    participant KV as Global KV
    participant K as Event Bus

    U->>CDN: GET short code
    alt CDN cache hit
        CDN-->>U: 302 to long URL
        CDN->>K: click event
    else CDN miss path
        CDN->>RS: forward to origin
        RS->>BF: probe bloom filter
        BF-->>RS: unknown or definitely not
        RS->>L: read local pod cache
        L-->>RS: hit or miss
        RS->>R: GET on local miss
        R-->>RS: hit or miss
        RS->>KV: GET on Redis miss
        KV-->>RS: long URL or not found
        RS->>R: backfill SETEX 24h
        RS->>L: backfill 60s
        RS-->>U: 302 with Location header
        RS->>K: click event fire and forget
    end

1. User’s browser hits the short URL. CDN edge checks its cache first.
2. On CDN hit (vast majority), browser gets the 302 instantly and the CDN logs the click.
3. On CDN miss, origin service checks a bloom filter to cheaply reject codes that definitely don’t exist.
4. Local pod cache → regional Redis → global KV, each populated on miss.
5. 302 returned to browser with Location: <long_url> header.
6. Click event fired to Kafka asynchronously; the redirect does not wait.

Failure handling: if all caches miss AND the KV is slow, we time out the read at 50ms and return 503 Try Again. We never serve a wrong URL; stale-if-error is risky because the link’s destination may have been changed.

Flow 3 - Analytics ingestion

sequenceDiagram
    autonumber
    participant RS as Redirect Service
    participant K as Kafka clicks
    participant SP as Flink Stream Processor
    participant CH as ClickHouse
    participant S3 as S3 Parquet
    participant API as Stats API
    participant D as Dashboard

    RS->>K: produce click event
    K->>SP: consume
    K->>S3: hourly sink
    SP->>SP: window by short_code + country + bucket
    SP->>CH: write aggregates
    D->>API: GET stats range 30d
    API->>CH: SELECT aggregated
    CH-->>API: rollup rows
    API-->>D: chart data

1. Every redirect produces a click event - {short_code, ts, ip_country, referrer, user_agent}.
2. Kafka Connect sinks raw events to S3 for long-term retention and ad-hoc analytics.
3. Flink maintains per-link rollups in ClickHouse across 1-min, 1-hour, and 1-day windows.
4. Dashboards query ClickHouse via the Stats API - fast even for a link with a billion clicks because we’re reading aggregates, not raw events.

Link state machine

stateDiagram-v2
    [*] --> ACTIVE
    ACTIVE --> EXPIRED: TTL reached
    ACTIVE --> DELETED: user deletes
    EXPIRED --> [*]: purged after 90d
    DELETED --> [*]: purged after 90d

ACTIVE links redirect normally. EXPIRED and DELETED return 410 Gone (not 404), so clients can distinguish “this link intentionally ended” from “this link never existed.”

Final Architecture

flowchart TD
    CLIENT["Client or Browser"]:::client
    CDN["CDN Edge"]:::edge
    GW["API Gateway"]:::edge

    WS["Write Service"]:::service
    RS["Redirect Service"]:::service
    IDG["Snowflake ID Gen"]:::service
    CONSUMER["Analytics Consumer"]:::service

    REDIS[("Redis Cache")]:::data
    DB[("Database")]:::data
    Q["Message Queue"]:::async

    CLIENT -->|"Request"| CDN
    CDN -->|"Cache miss"| GW
    GW -->|"3a. POST /links"| WS
    GW -->|"3b. GET /code"| RS
    WS -->|"Get unique ID"| IDG
    WS -->|"Store mapping"| DB
    RS -->|"Lookup short code"| REDIS
    RS -->|"DB fallback"| DB
    RS -->|"Fire click event"| Q
    Q -->|"Aggregate counts"| CONSUMER
    CONSUMER -->|"Write rollups"| DB

    classDef client fill:#4c3a5e,stroke:#818cf8,color:#e2e8f0
    classDef edge fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    classDef service fill:#1a3a2a,stroke:#4ade80,color:#e2e8f0
    classDef async fill:#AB47BC,stroke:#4A148C,color:#fff
    classDef data fill:#3b3520,stroke:#fbbf24,color:#e2e8f0

All requests enter through CDN → API Gateway. The gateway routes writes (POST /links) to the Write Service and redirects (GET /:code) to the Redirect Service. The CDN caches popular redirects at the edge for sub-10ms responses.

Key Technologies

What’s Expected at Each Level

This section helps you calibrate your depth. You don’t need to cover everything - just know what’s expected for your level.

Mid-level

Design basic URL creation and redirect flow. Propose a database for storing long-to-short mappings. Understand Base62 encoding for generating short codes from numeric IDs. Explain the difference between 301 (permanent) vs 302 (temporary) redirects and when each is appropriate.

Senior

Propose a counter-based or hash-based ID generation strategy (Snowflake or similar). Discuss caching with Redis for popular URLs and CDN for redirect responses at the edge. Explain horizontal scaling of the write service with a distributed counter (range allocation per instance). Articulate the 100:1 read-write ratio and how it drives the architecture.

Staff+

Address multi-region deployment with counter range allocation per region (no cross-region coordination on writes). Discuss the analytics pipeline - click tracking without adding latency to the redirect hot path (fire-and-forget to Kafka, process async). Cover URL expiration cleanup (background TTL sweeper vs lazy deletion), and the security implications of predictable sequential codes (enumeration attacks, phishing detection).

🎯 Key Takeaways

• Base62 encoding turns numeric IDs into short 7-char codes (3.5T unique codes)
• 302 redirect enables analytics; 301 for permanent redirects without tracking
• CDN caching at the edge handles redirect QPS without hitting origin
• Snowflake ID eliminates the need for a centralized ID counter

Related Designs

• Rate Limiter - protecting high-QPS endpoints
• Leaderboard - Redis-based caching patterns
• Stock Broker - idempotency keys for write operations

Related Concepts

Understand the building blocks used in this design:

• Consistent Hashing → — distributes the short-code keyspace across nodes with minimal reshuffling
• Caching → — hot short-URL lookups are served from Redis instead of the database
• CDN → — edge nodes handle redirects for popular links close to users
• Database Sharding → — partitions the code-to-URL mapping table as it grows to billions of rows