High-Level Design

System design is the process of defining the architecture, components, and data flow of a system to meet specific requirements. Think of it like being an architect for software — we decide what pieces we need, how they talk to each other, and how the whole thing holds up when millions of people use it.

Why Do Companies Ask System Design Questions?

Coding interviews test if we can write correct code. System design interviews test if we can think big.

At 3+ years of experience, companies expect us to do more than just write functions. They want to know:

• Can we break a vague problem into concrete pieces?
• Can we think about trade-offs (there’s no perfect answer)?
• Can we communicate our thought process clearly?
• Do we understand how real-world systems actually work?

The only difference between a coding interview and a system design interview is this: coding has a “right answer,” system design does not. It’s a conversation, not a test.

What Interviewers Actually Look For

Here’s what most people get wrong — interviewers are NOT looking for the “perfect architecture.” They’re evaluating:

1. Communication — Can we explain our thinking clearly?
2. Problem breakdown — Can we go from vague to specific?
3. Trade-off awareness — Do we know why we picked X over Y?
4. Breadth of knowledge — Do we know what tools exist?
5. Depth when needed — Can we dive deep into one area?

A candidate who designs a simpler system but explains trade-offs well will beat someone who draws a complex diagram but can’t explain why.

The Journey: Single Server to Distributed System

This is basically what system design is about — understanding when and why we move from one stage to the next. We don’t need a distributed system for 100 users. But we do need to know how to build one for 100 million.

When Does System Design Matter?

• Junior roles (0-2 YOE): Rarely asked. Focus on coding.
• Mid-level (3-5 YOE): Common. Expected to know the building blocks.
• Senior+ (5+ YOE): Critical. Expected to lead design discussions and make architectural decisions.

Even if we’re not interviewing, understanding system design makes us better engineers. We start seeing why things are built the way they are.

Key Takeaway

System design isn’t about memorizing architectures. It’s about understanding building blocks (load balancers, caches, queues, databases) and knowing when to use what. The rest of this collection covers exactly that — one building block at a time.

The biggest mistake in system design interviews? Jumping straight to drawing boxes. We need a framework — a repeatable process that works for any problem, from “Design a URL Shortener” to “Design Netflix.”

The 5-Step Framework

Let’s walk through each step.

Step 1: Requirements & Scope (5 min)

Never start designing without asking questions first. The interviewer gives us a vague prompt on purpose — they want to see if we can narrow it down.

Ask about:

• Users: How many? What kind?
• Features: Which ones are core? Which can we skip?
• Scale: Is this for 1,000 users or 1 billion?
• Constraints: Latency requirements? Availability needs?

For “Design Twitter,” we might say: “Let’s focus on the tweet feed, posting tweets, and following users. We’ll target 200M daily active users.”

Step 2: Back-of-the-Envelope Estimation (5 min)

Quick math to understand the scale. We’re not aiming for exact numbers — just the right order of magnitude.

• QPS (Queries Per Second): How many requests per second?
• Storage: How much data do we store per day/year?
• Bandwidth: How much data flows in and out?

This tells us whether we need one server or a thousand. We cover this in detail in a separate note.

Step 3: High-Level Design (15 min)

This is where we draw the architecture. Start simple and add complexity only when needed.

• Draw the core components (clients, servers, databases, caches)
• Define the API endpoints (what calls what)
• Show the data flow (how a request travels through the system)
• Pick the database type (SQL vs NoSQL, and why)

Keep it simple. We can always add more later.

Step 4: Deep Dive (15 min)

The interviewer will say something like “Let’s dig into X” or we can proactively pick 2-3 areas. This is where we show depth.

Common deep-dive areas:

• Database schema and indexing
• Caching strategy
• How we handle failures
• Data partitioning / sharding
• Consistency vs availability trade-offs

This is the most important step. It’s where trade-off discussions happen.

Step 5: Wrap Up (5 min)

Summarize and show we’re thinking about the future:

• Bottlenecks: What could break first?
• Monitoring: How would we know something is wrong?
• Scaling: What changes at 10x or 100x current scale?
• Nice-to-haves: Features we scoped out but could add later

Common Mistakes

1. Jumping to the solution — Not asking questions first. This is the #1 red flag.
2. Over-engineering — Designing for Google scale when the problem says “small startup.”
3. Silence — Not talking while thinking. The interviewer can’t read our mind.
4. No trade-offs — Every decision has pros and cons. Always mention them.
5. Ignoring the interviewer — If they’re nudging us somewhere, follow the hint.

The Golden Rule

In simple language, system design interviews are a conversation. We talk, the interviewer responds, we adjust. It’s collaborative, not adversarial. The framework gives us structure, but we should stay flexible and follow where the discussion goes.

Every system design problem starts with the same question: what exactly are we building? The interviewer gives us a vague prompt like “Design a URL shortener” — it’s our job to turn that into a clear list of requirements before drawing a single box.

Two Types of Requirements

Functional requirements describe features — what the user can do. Non-functional requirements describe quality — how well it works under the hood.

Key Non-Functional Requirements

These come up in almost every system design interview. We need to know them cold:

Scalability — Can the system handle growth? From 1,000 to 1,000,000 users without a rewrite.

Availability — Is the system up when users need it? Measured in “nines” — 99.9% means ~8.7 hours of downtime per year, 99.99% means ~52 minutes.

Consistency — Does every user see the same data at the same time? Strong consistency means yes. Eventual consistency means “give it a second.”

Latency — How fast does the system respond? A search engine needs <200ms. A batch report can take minutes.

Durability — Once we store data, does it stay stored? We can never lose a user’s photo or a bank transaction.

How to Ask the Right Questions

Don’t just sit there and guess. Ask questions like:

• “How many users are we designing for?”
• “What’s the read-to-write ratio?”
• “Do we need real-time or is a few seconds of delay OK?”
• “What’s more important — consistency or availability?”
• “Should we handle this feature, or can we scope it out?”

The interviewer expects us to ask. Not asking is a red flag.

Example: URL Shortener Requirements

Let’s say the interviewer says “Design a URL shortener like bit.ly.” Here’s how we’d gather requirements:

Questions we’d ask:

• How many URLs shortened per day? → “100 million”
• Read-heavy or write-heavy? → “100:1 read-to-write ratio”
• Do short URLs expire? → “Optional, default no expiry”
• Custom short URLs? → “Nice to have, not core”
• Analytics? → “Not in scope for now”

Functional requirements (after discussion):

1. Given a long URL, generate a short URL
2. Given a short URL, redirect to the original
3. Short URLs should be as short as possible

Non-functional requirements:

1. Very low latency for redirects (<100ms)
2. High availability — links must always work
3. Shortened URLs should not be predictable (security)

Now we have a clear scope. We know what to build and what quality bar to hit. Everything after this — estimation, design, deep dives — flows from these requirements.

The Trade-Off Question

Here’s a pro tip: non-functional requirements often conflict with each other. We can’t have perfect consistency AND perfect availability (that’s the CAP theorem, which we’ll cover later). When we state our requirements, we should also state which one wins when they clash.

For a URL shortener: “Availability is more important than consistency. If a link takes a few seconds to propagate globally, that’s fine. But a link should never be down.”

In simple language, requirements gathering is about turning a vague question into a concrete plan. It takes 5 minutes in an interview but sets the direction for everything that follows. Skip it, and we’re building blind.

Back-of-the-envelope estimation is quick, rough math to figure out the scale of our system. We’re not trying to be exact — we just need to know if we’re dealing with thousands or billions, megabytes or petabytes. That changes everything about our design.

Why Estimation Matters

If our system gets 10 requests per second, a single server is fine. If it gets 100,000 requests per second, we need load balancers, caching, sharding, and a whole different architecture. Estimation tells us which world we’re in.

Numbers Every Engineer Should Know

And for storage:

Handy shortcut: There are about 86,400 seconds in a day. For quick math, round to ~100,000 (10^5). A month is about 2.5 million seconds.

The Four Key Calculations

1. QPS (Queries Per Second)

QPS = Daily Active Users × Queries per User / 86,400

Peak QPS = QPS × 2  (or ×3 for spiky traffic)

Example: 10 million DAU, each user makes 5 requests/day.

QPS = 10M × 5 / 86,400 ≈ 580 QPS
Peak QPS ≈ 1,160 QPS

2. Storage

Storage = Daily New Records × Record Size × Retention Period

Example: 100M new URLs per day, each URL record is 500 bytes, keep for 5 years.

Daily  = 100M × 500 bytes = 50 GB/day
Yearly = 50 GB × 365 ≈ 18 TB/year
5 years = ~90 TB total

3. Bandwidth

Incoming = QPS × Request Size
Outgoing = QPS × Response Size

4. Memory for Cache

We usually cache the hot data — the most frequently accessed items. A common rule of thumb is to cache 20% of daily requests (the 80/20 rule: 20% of data handles 80% of traffic).

Cache Memory = Daily Requests × 0.2 × Average Response Size

Full Example: Estimate Twitter’s Storage

Let’s say Twitter has:

• 300M monthly active users, 50% are daily → 150M DAU
• Each user posts 2 tweets/day on average
• Each tweet: 140 chars (~280 bytes) + metadata (~200 bytes) = ~500 bytes
• 10% of tweets have a photo (~500 KB average)

Tweet storage per day:

Text:  150M × 2 × 500 bytes = 150 GB/day
Photos: 150M × 2 × 0.10 × 500 KB = 15 TB/day

Per year:

Text:   ~55 TB/year
Photos: ~5.5 PB/year

That tells us we need a serious storage strategy — object storage (like S3) for media, and sharded databases for tweet metadata.

Powers of Two — Quick Reference

Tips for the Interview

1. State assumptions clearly. “I’m assuming 100M DAU” — the interviewer can correct us.
2. Round aggressively. Use 10^5 instead of 86,400. Nobody expects exact math.
3. Focus on order of magnitude. The difference between 50 TB and 90 TB doesn’t change our design. The difference between 50 GB and 50 TB does.
4. Don’t spend more than 5 minutes. Estimation supports the design, it’s not the main event.

In simple language, back-of-the-envelope estimation is about getting a feel for the scale. Are we building a bicycle or a spaceship? The math takes 5 minutes but saves us from designing something wildly over- or under-engineered.

System design is all about trade-offs. There’s no perfect system — every choice we make comes with a cost. The key is understanding what we’re gaining and what we’re giving up with each decision.

Key Design Principles

Scalability

The ability to handle more load by adding resources. Two flavors:

• Vertical scaling (scale up): Bigger machine. More RAM, faster CPU. Simple but has a ceiling.
• Horizontal scaling (scale out): More machines. Harder to implement but virtually unlimited.

Think of it like a restaurant. Vertical scaling = get a bigger kitchen. Horizontal scaling = open more locations.

Availability

The system is up and working when users need it. Measured in “nines”:

Reliability

The system does what it’s supposed to do correctly. A system can be available (it’s responding) but unreliable (it’s giving wrong answers). We need both.

Performance

How fast the system responds. Two key metrics:

• Latency — Time to handle a single request (usually p50, p95, p99)
• Throughput — How many requests we handle per second

Maintainability

Can other engineers (or future us) understand and modify this system? Simple designs beat clever ones.

The Big Trade-offs

Consistency vs Availability

This is the famous CAP theorem in disguise. In simple language: when a network issue happens, we have to choose — do we give users potentially stale data (availability) or do we tell them “try again later” (consistency)?

• Bank account: Must be consistent. We can’t show the wrong balance.
• Social media feed: Availability wins. If a like takes 2 seconds to show up globally, nobody cares.

Latency vs Throughput

We can make one request super fast (low latency) or handle a massive number of requests (high throughput), but optimizing for one often hurts the other. Batching requests improves throughput but increases latency for individual requests.

Single Point of Failure (SPOF)

A SPOF is any component whose failure takes down the entire system. Every part of our design should have a backup plan:

• One server? Add another behind a load balancer.
• One database? Add a replica.
• One data center? Deploy across multiple regions.
• One load balancer? Use active-passive failover.

The rule is simple: if it can fail, assume it will. Then plan for it.

Stateless vs Stateful Services

Stateful services store information about the current session (like “this user is logged in”). If that server goes down, the state is lost.

Stateless services don’t remember anything between requests. Every request carries all the information needed (like a JWT token). Any server can handle any request.

Stateless services are much easier to scale — we just add more servers behind a load balancer and it just works. That’s why we push state to external stores (databases, Redis, sessions stores) and keep our application servers stateless.

Key Takeaway

In simple language, there’s no “best” architecture. There’s only the right architecture for our specific requirements. A system design interview is our chance to show we understand these trade-offs and can make informed decisions — not just draw boxes on a whiteboard.

Before we can design any system, we need to understand how a request gets from a user’s browser to our server. It all starts with DNS — the phone book of the internet.

What Happens When We Type a URL?

In simple language, DNS translates human-readable names (like google.com) into IP addresses (like 142.250.80.46) that computers understand. Without DNS, we’d have to memorize IP addresses for every website.

How DNS Resolution Works

DNS has a hierarchy, like asking directions from more and more knowledgeable people:

1. DNS Resolver (our ISP or something like 8.8.8.8) — The starting point. It does the legwork.
2. Root Name Server — Knows which servers handle .com, .org, .io, etc. There are only 13 root server clusters worldwide.
3. TLD Name Server — Handles a specific top-level domain (like all .com domains). Points to the authoritative server.
4. Authoritative Name Server — The final answer. This server actually knows what IP google.com maps to.

The result gets cached at every level (browser, OS, resolver) with a TTL (Time To Live). That’s why DNS changes take time to propagate — old cached entries have to expire first.

DNS Record Types We Should Know

Why DNS Matters in System Design

DNS isn’t just “it resolves names.” In system design, DNS is a powerful tool:

Load distribution — DNS can return different IPs for the same domain, spreading traffic across multiple servers (DNS round-robin).

Geo-routing — DNS can return the IP of the server closest to the user. A user in India gets routed to the Mumbai server, while a user in the US hits the Virginia server.

Failover — If a server goes down, DNS can stop returning its IP. Health checks detect the failure, and DNS automatically routes traffic to healthy servers.

CDN routing — Services like CloudFront and Cloudflare use DNS to route users to the nearest edge server.

DNS in Our System Designs

When we’re drawing system design diagrams, DNS is usually the very first step:

User → DNS → Load Balancer → Application Servers → Database

We don’t usually deep-dive into DNS in interviews unless asked, but we should always mention it. It shows we understand the full picture — not just the backend.

Quick Gotchas

• DNS propagation delay: Changing DNS records can take up to 48 hours because of caching at various levels. In practice, it’s usually faster.
• DNS is a SPOF (sort of): If our DNS provider goes down, nobody can reach us. That’s why companies like Netflix use multiple DNS providers.
• TTL trade-off: Short TTL = faster failover but more DNS queries. Long TTL = fewer queries but slower to react to changes.

In simple language, DNS is the first thing that happens in any web request. It’s simple in concept but powerful in practice — it can do load balancing, geo-routing, and failover, all before a single HTTP request is made.

A load balancer is a device (or software) that sits in front of our servers and distributes incoming traffic across multiple servers. Think of it like a host at a restaurant — they don’t let everyone pile into one table; they spread diners evenly across the floor.

Why We Need Load Balancers

Without a load balancer, all traffic goes to one server. That’s bad for two reasons:

1. Single point of failure — If that server crashes, the whole system is down.
2. Limited capacity — One server can only handle so much traffic.

A load balancer solves both problems. If one server dies, traffic goes to the others. If traffic grows, we add more servers.

Where Load Balancers Sit

We can place load balancers at multiple points:

• Between users and web servers (most common)
• Between web servers and application servers
• Between application servers and databases

L4 vs L7 Load Balancing

L4 (Transport Layer) — Routes based on IP address and port. Fast but dumb. It doesn’t look at the content of the request. Think of it like sorting mail by zip code.

L7 (Application Layer) — Routes based on the actual content: URL path, HTTP headers, cookies. Slower but smarter. We can send /api/* requests to API servers and /static/* to file servers.

Most modern load balancers (NGINX, AWS ALB) operate at L7.

Load Balancing Algorithms

Round Robin is the simplest and works great when all servers are identical and requests take roughly the same time.

Least Connections is better when some requests take much longer than others (like file uploads vs simple GETs).

IP Hash ensures the same client always hits the same server. Useful when we have session data on the server (though we should prefer stateless servers + external session stores).

Health Checks

Load balancers periodically ping each server to check if it’s alive. If a server stops responding, the load balancer removes it from the rotation. When it comes back, it gets added again.

LB → GET /health → Server 1: 200 OK  ✓ (keep in rotation)
LB → GET /health → Server 2: timeout ✗ (remove from rotation)
LB → GET /health → Server 3: 200 OK  ✓ (keep in rotation)

Load Balancer Redundancy

Wait — if the load balancer is a single point of failure, what do we do? We use two load balancers in an active-passive setup:

• Active LB handles all traffic
• Passive LB monitors the active one
• If active goes down, passive takes over (using a floating IP or DNS failover)

Some setups use active-active, where both LBs handle traffic simultaneously.

Popular Load Balancers

• NGINX — Software LB, very popular, can also be a reverse proxy and web server
• HAProxy — High-performance software LB, used by GitHub and Stack Overflow
• AWS ALB/NLB — Managed cloud LBs (ALB = L7, NLB = L4)
• Caddy — Modern LB with automatic HTTPS

In simple language, a load balancer is like a traffic cop for our servers. It spreads the work evenly, routes around failures, and lets us add more servers whenever we need more capacity. It’s one of the first things we add when scaling beyond a single server.

Caching is storing a copy of data in a faster location so we don’t have to fetch it from the slower source every time. Think of it like keeping a sticky note on our desk instead of walking to the filing cabinet every time we need a phone number.

It’s probably the single most impactful thing we can do for performance.

Where to Cache

The closer the cache is to the user, the faster the response. A browser cache hit is instant. A CDN hit avoids the trip to our data center. A Redis hit avoids the slow database query.

Cache Hit vs Cache Miss

• Cache hit — The data is in the cache. We return it immediately. Fast.
• Cache miss — The data is NOT in the cache. We fetch from the source, store it in the cache, then return it. Slow (but next time it’ll be a hit).

The hit ratio tells us how effective our cache is. If 95% of requests are cache hits, our cache is doing great. Below 80%, we should rethink our strategy.

Cache Eviction Policies

The cache has limited memory. When it’s full and new data comes in, we have to kick something out. The question is: what do we evict?

LRU is the default choice in most system design interviews. It’s simple and works well for most access patterns.

Cache Invalidation Strategies

The hardest problem with caching: keeping the cache in sync with the database. If the database changes but the cache still has the old value, users see stale data.

Cache-Aside (Lazy Loading)

The most common pattern. The application manages the cache directly:

1. Read: Check cache first → miss → read from DB → write to cache → return
2. Write: Write to DB → delete from cache (next read will re-populate it)

Pros: Only caches what’s actually requested. Cache failure doesn’t break the system. Cons: First request after a miss is slow. Potential for stale data between DB write and cache delete.

Write-Through

Every write goes to the cache AND the database at the same time.

Pros: Cache is always up to date. No stale data. Cons: Higher write latency (two writes per operation). Cache may fill with data that’s never read.

Write-Behind (Write-Back)

Write to the cache first, then asynchronously write to the database later.

Pros: Super fast writes. Great for write-heavy workloads. Cons: Risk of data loss if the cache crashes before persisting to DB.

When NOT to Cache

Caching isn’t always the answer:

• Frequently changing data — If data changes every second, the cache is constantly stale.
• Low-traffic data — If it’s rarely accessed, the cache miss rate is high and we’re wasting memory.
• Write-heavy workloads — More writes than reads means the cache is constantly being invalidated.
• Data that must be perfectly consistent — Like account balances. Stale cache = real problems.

Popular Caching Tools

• Redis — In-memory key-value store. Supports data structures (lists, sets, sorted sets). Most popular choice.
• Memcached — Simpler than Redis, pure key-value. Slightly faster for simple caching.
• Varnish — HTTP reverse proxy cache. Great for caching entire HTTP responses.

Cache in System Design Interviews

When an interviewer asks “how would you improve performance?”, caching is almost always part of the answer. Common cache use cases:

• Cache database query results to reduce DB load
• Cache user session data for fast authentication
• Cache API responses from third-party services
• Cache computed results (like feed generation or recommendations)

In simple language, caching trades memory for speed. We store a copy of frequently accessed data in a fast place so we don’t keep hammering the slow place. It’s the difference between a 2ms response (Redis) and a 200ms response (database query with joins).

A CDN (Content Delivery Network) is a geographically distributed network of servers that delivers content from the location closest to the user. Think of it like having copies of a book in libraries across the country instead of one central warehouse — people get the book faster because they go to the nearest library.

How a CDN Works

Here’s the flow:

1. User in India requests image.png from our site
2. DNS routes the request to the nearest CDN edge server (Mumbai)
3. If the edge has it cached → return immediately (cache hit)
4. If not → edge fetches from origin server, caches it, returns to user (cache miss)
5. Next user in India requesting the same file gets it from Mumbai instantly

The first request is slow (has to go to origin). Every subsequent request from that region is fast.

Push CDN vs Pull CDN

Pull CDN — The CDN fetches content from our origin server on demand (when someone requests it). Content is cached after the first request and expires based on TTL.

• We don’t manage what’s on the CDN
• Works great for high-traffic sites (content gets cached quickly)
• Simpler to set up

Push CDN — We explicitly upload content to the CDN. We decide what’s cached and when it’s updated.

• Full control over what’s on the CDN
• Better for content that changes infrequently (firmware files, large media)
• More work to manage

Most websites use Pull CDN. It’s simpler and handles most use cases well.

What to Put on a CDN

Great for CDN:

• Images, videos, audio files
• CSS, JavaScript bundles
• Fonts
• Static HTML pages
• Large downloadable files (PDFs, installers)

Not great for CDN:

• Dynamic API responses (changes per user/request)
• Private/personalized content
• Real-time data (stock prices, live scores)
• Very rarely accessed content (wastes edge storage)

CDN Invalidation

When we update content on our origin server, the CDN still has the old version cached. We need to invalidate it:

1. TTL expiration — Set a Time To Live. After it expires, the edge fetches fresh content. Simple but has a delay.
2. Explicit invalidation — Tell the CDN to purge specific files. Instant but costs money per request on some providers.
3. Versioned URLs — Use style.v2.css instead of style.css. The new URL is a cache miss, so fresh content is fetched. This is the most reliable approach.

Popular CDNs

CDN in System Design

In interviews, mention CDN whenever the system serves static content to a global audience. It’s a quick win that dramatically reduces latency and server load.

A typical architecture mention:

User → CDN (static assets) → Load Balancer → App Server → DB
         ↑
   Serves images, JS, CSS
   without hitting our servers

The performance impact is massive. Instead of a user in Tokyo making a round trip to our US server (150ms+), they hit a Tokyo edge server (5ms). For media-heavy sites like Instagram or Netflix, CDNs handle the vast majority of traffic — our origin servers barely break a sweat.

In simple language, a CDN puts copies of our content closer to users around the world. Less distance = less latency = happier users. For any system that serves files or media globally, a CDN is a no-brainer.

A message queue is a component that sits between two services and holds messages until the receiver is ready to process them. Think of it like a mailbox — the sender drops a letter in, and the receiver picks it up when they’re available. The sender doesn’t have to wait for the receiver to be home.

This is the foundation of asynchronous processing — doing work later instead of right now.

Why We Need Message Queues

Without a queue, services talk to each other synchronously. Service A calls Service B and waits. If B is slow or down, A is stuck.

With a queue:

• A drops a message in the queue and moves on immediately
• B picks it up whenever it’s ready
• If B crashes, the message stays in the queue — nothing is lost

This gives us decoupling, resilience, and scalability.

The Core Pattern

• Producer — The service that creates and sends messages
• Queue — The buffer that holds messages
• Consumer — The service that reads and processes messages

Point-to-Point vs Pub/Sub

Point-to-Point (Queue) — Each message is consumed by exactly one consumer. Once processed, the message is removed. Like a task queue where each task is done once.

Pub/Sub (Topic) — Each message can be consumed by multiple subscribers. The message stays available for all subscribers. Like a broadcast — everyone who’s listening gets the message.

When to Use Message Queues

Decoupling services — The order service doesn’t need to know about the email service. It just publishes “order placed” and moves on. The email service subscribes and sends the confirmation.

Handling traffic spikes — During a flash sale, we get 100x the normal orders. The queue absorbs the spike. Workers process orders at a steady rate.

Retry and error handling — If processing fails, the message goes back to the queue. It’ll be retried instead of lost. We can even have a dead letter queue (DLQ) for messages that fail repeatedly.

Heavy async work — Sending emails, generating reports, processing images, encoding videos — none of these need to happen during the user’s request. Drop a message in the queue and respond to the user immediately.

Popular Tools

Kafka

• Distributed event streaming platform
• Extremely high throughput (millions of messages/sec)
• Messages are persisted to disk and retained for days/weeks
• Consumers can replay messages from any point in time
• Great for: event sourcing, log aggregation, real-time analytics

RabbitMQ

• Traditional message broker
• Supports complex routing (exchanges, bindings)
• Messages are removed after consumption
• Lower throughput than Kafka but more flexible routing
• Great for: task queues, RPC patterns, complex routing

Amazon SQS

• Fully managed queue service from AWS
• No infrastructure to manage
• Two flavors: Standard (at-least-once, unordered) and FIFO (exactly-once, ordered)
• Great for: AWS-native apps, simple queuing needs

Quick Comparison

Message Queues in System Design

In interviews, bring up message queues whenever we have:

• Work that doesn’t need to happen immediately
• Services that should be independent of each other
• Traffic that’s bursty or unpredictable
• Operations that might fail and need retries

A common pattern in system design interviews:

User uploads video → API Server → Queue → Video Processing Workers → Store in S3
                         ↓
                   Return "processing..." to user immediately

In simple language, a message queue lets us say “I’ll deal with this later” instead of doing everything right now. It makes our systems more resilient, more scalable, and better at handling the unpredictable real world.

A proxy is a server that sits between a client and another server, acting as an intermediary. There are two types, and they sit on opposite sides of the connection. Let’s clear up the difference once and for all.

Forward Proxy vs Reverse Proxy

The only difference is which side they protect:

• Forward proxy sits in front of clients. The server doesn’t know who the real client is. Example: a VPN or corporate proxy.
• Reverse proxy sits in front of servers. The client doesn’t know which server actually handled the request. Example: Nginx routing to backend servers.

In system design, we almost always talk about reverse proxies.

What a Reverse Proxy Does

A reverse proxy is incredibly useful. It handles a bunch of cross-cutting concerns so our application servers don’t have to:

SSL/TLS Termination

The reverse proxy handles HTTPS encryption and decryption. Traffic between the proxy and our internal servers can be plain HTTP (since it’s within our network). This offloads CPU-intensive crypto work from our app servers.

Load Balancing

Distribute requests across multiple backend servers. Most reverse proxies have load balancing built in (round robin, least connections, etc.).

Compression

Compress responses (gzip, brotli) before sending them to clients. Reduces bandwidth and speeds up page loads.

Caching

Cache static content (or even dynamic responses) and serve them directly without hitting the backend. Huge performance boost.

Request Routing

Route different URL paths to different backend services:

• /api/* → API servers
• /static/* → File server
• / → Frontend server

This is how we can run multiple services behind a single domain.

Rate Limiting & Security

Block malicious traffic, limit requests per IP, add security headers — all at the proxy level before requests reach our application.

Popular Reverse Proxies

Forward Proxy Use Cases

While less common in system design interviews, forward proxies are still important:

• Corporate networks — Control and monitor employee internet access
• Bypassing restrictions — VPNs are forward proxies that let us access geo-blocked content
• Anonymity — Hide our IP address from websites we visit
• Caching for clients — A school might cache frequently accessed educational sites to save bandwidth

Reverse Proxy in System Design

In every system design interview, there’s a reverse proxy somewhere. It’s usually sitting right behind the DNS/load balancer:

User → DNS → Reverse Proxy / LB → App Servers → Database

Sometimes the reverse proxy IS the load balancer (Nginx and Caddy can do both). In cloud setups, managed load balancers (AWS ALB) take over this role.

When discussing our architecture, we can mention the reverse proxy handles:

• SSL termination (so internal traffic is faster)
• Static file serving (so app servers focus on business logic)
• Request routing (so we can run multiple services behind one domain)

Proxy vs Load Balancer vs API Gateway

These terms overlap a lot. Here’s the quick distinction:

• Reverse Proxy — General purpose: SSL, caching, compression, routing
• Load Balancer — Specifically distributes traffic across servers
• API Gateway — Like a reverse proxy but for APIs: auth, rate limiting, request transformation, analytics

Many tools (like Nginx, Caddy, Kong) can act as all three. In system design, we don’t need to be pedantic about the labels — just explain what the component does.

In simple language, a reverse proxy is the bouncer at the club. It stands at the door, handles the crowd, checks IDs (SSL), and directs people to the right area inside. Our application servers just focus on doing their job without worrying about all that.

This is one of the most common system design questions. “Should we use SQL or NoSQL?” The answer is always “it depends” — but let’s learn what it depends on.

SQL (Relational Databases)

SQL databases store data in tables with rows and columns. Every row has the same structure. Think of it like a spreadsheet with strict rules — every column has a type, and we define relationships between tables.

Examples: PostgreSQL, MySQL, SQLite, SQL Server, Oracle

Key traits:

• Fixed schema — we define the structure upfront
• Relationships via foreign keys (joins)
• ACID transactions (more on this later)
• Great for structured, predictable data

NoSQL (Non-Relational Databases)

NoSQL means “Not Only SQL.” These databases break free from the rigid table structure. There are actually four types, and they’re quite different from each other.

Key-Value Stores (Redis, DynamoDB)

The simplest NoSQL type. It’s literally a giant hash map — a key maps to a value. Super fast for lookups, but we can’t query by value. Think of it like a locker room: we know the locker number (key), we get the contents (value).

Use case: Caching, sessions, rate limiting, leaderboards.

Document Stores (MongoDB, CouchDB)

Data is stored as JSON-like documents. Each document can have a different structure. Think of it like filing cabinets where each folder can hold different papers.

Use case: Product catalogs, content management, user profiles — anything with varied structure.

Columnar Stores (Cassandra, HBase)

Data is stored by columns instead of rows. This is great when we need to read specific columns across millions of rows. Think of it like reading one column of a spreadsheet without loading all other columns.

Use case: Analytics, time-series data, write-heavy workloads.

Graph Databases (Neo4j, ArangoDB)

Data is stored as nodes and edges (relationships). When the relationships between data are the most important thing, graph databases shine. Think of it like a social network map.

Use case: Social networks, recommendation engines, fraud detection.

ACID vs BASE

SQL databases follow ACID — strong guarantees. NoSQL often follows BASE — looser guarantees in exchange for performance and availability.

• ACID: Atomicity, Consistency, Isolation, Durability — everything is correct and reliable
• BASE: Basically Available, Soft-state, Eventually consistent — the system stays available, and data will be consistent eventually (not immediately)

In simple language, ACID says “every transaction is perfect or it doesn’t happen.” BASE says “the system keeps running and catches up later.”

Decision Framework

Pick SQL when:

• Data is structured and relationships matter (e-commerce orders, banking)
• We need complex queries with joins
• Data integrity is critical (financial transactions)
• Schema is mostly known upfront

Pick NoSQL when:

• Schema changes frequently or data is unstructured
• We need massive horizontal scaling (millions of reads/writes per second)
• Data is naturally hierarchical or document-shaped (JSON APIs)
• Availability matters more than immediate consistency

Most real systems use both. PostgreSQL for core business data, Redis for caching, maybe Elasticsearch for search. The question isn’t SQL vs NoSQL — it’s which tool fits which part of our system.

Imagine we have a book with 1,000 pages and we need to find every mention of “caching.” We could read every single page (slow), or we could check the index at the back of the book (fast). Database indexes work exactly the same way.

What Is an Index?

An index is a separate data structure that the database maintains alongside our table. It stores a sorted reference to rows based on specific columns, so the database can jump directly to the right rows instead of scanning the entire table.

Without an index, a query like SELECT * FROM users WHERE email = 'a@b.com' has to check every row in the table. This is called a full table scan. With an index on email, the database finds the row almost instantly.

How Indexes Work

B-Tree Index (The Default)

Most databases use B-trees (balanced trees) as the default index type. A B-tree keeps data sorted and allows searches, insertions, and deletions in O(log n) time.

Think of it like a phone book. We don’t read every name — we jump to the right letter section, then narrow down. That’s a B-tree.

Hash Index

A hash index uses a hash function to map keys to locations. It gives us O(1) lookups for exact matches, but it can’t handle range queries (WHERE age > 25). Only useful for equality checks.

Types of Indexes

Single-Column Index

The most basic kind. We index one column.

CREATE INDEX idx_users_email ON users(email);

Now WHERE email = 'a@b.com' is fast.

Composite Index (Multi-Column)

We can index multiple columns together. The order matters a lot.

CREATE INDEX idx_orders_user_date ON orders(user_id, created_at);

This index helps queries that filter by user_id, or by user_id AND created_at. But it does not help queries that filter only by created_at. Think of it like a phone book sorted by last name, then first name — we can’t look up just by first name.

This is called the leftmost prefix rule.

Covering Index

A covering index includes all the columns a query needs, so the database doesn’t even have to look at the actual table. It gets everything from the index itself.

-- If we frequently run this query:
SELECT email, name FROM users WHERE email = 'a@b.com';

-- This covering index handles it entirely:
CREATE INDEX idx_users_email_name ON users(email) INCLUDE (name);

Unique Index

Enforces uniqueness on a column. A primary key automatically creates a unique index.

CREATE UNIQUE INDEX idx_users_email_unique ON users(email);

The Trade-Off

Here’s the thing nobody tells us about indexes: they aren’t free.

Indexes speed up reads but slow down writes. Every time we INSERT, UPDATE, or DELETE a row, the database also has to update every index on that table. More indexes = more work on writes.

Indexes also consume disk space. A heavily indexed table might have indexes that are larger than the table itself.

When NOT to Index

• Small tables — if the table has a few hundred rows, a full scan is fine
• Write-heavy tables — if we’re inserting thousands of rows per second, indexes add overhead
• Low-cardinality columns — a boolean column (is_active) has only two values; an index barely helps
• Columns we never query on — if we never filter or sort by a column, don’t index it

Practical Tips

-- See how the database plans to run our query
EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'a@b.com';

The EXPLAIN output tells us if the database is using an index (Index Scan) or scanning the whole table (Seq Scan). If we see a sequential scan on a large table with a WHERE clause, we probably need an index.

In simple language, an index is a cheat sheet the database keeps on the side. It makes finding data fast, but we pay for it on every write. The trick is indexing the right columns — the ones we actually search and filter on.

If our database server dies, all our data is gone. If traffic spikes, one server can’t handle all the reads. Replication solves both problems by keeping copies of our data on multiple servers.

What Is Replication?

Replication means keeping the same data on multiple database servers (called replicas or nodes). When we write data to one server, that change gets copied to the others.

The goals are simple:

• High availability — if one server crashes, others take over
• Read scaling — spread read queries across multiple servers
• Low latency — place replicas closer to users in different regions

Single-Leader Replication (Master-Slave)

This is the most common setup. One node is the leader (master) and the rest are followers (replicas/slaves).

• All writes go to the leader
• The leader sends changes to all followers
• Reads can go to any follower (or the leader)

This is what PostgreSQL, MySQL, and MongoDB use by default. It’s simple and works well when reads far outnumber writes (which is true for most applications).

The downside? The leader is a single point of failure for writes. If it goes down, we need to promote a follower to become the new leader (failover).

Multi-Leader Replication

Instead of one leader, we have multiple leaders that all accept writes. Each leader replicates its changes to all others.

Use case: Multi-region deployments. We put a leader in each data center so users write to the nearest one.

The big problem is write conflicts. If two leaders update the same row at the same time, which one wins? We need conflict resolution strategies:

• Last write wins — the latest timestamp wins (simple but can lose data)
• Merge values — combine both changes (complex but preserves data)
• Let the application decide — store both versions and let the app resolve it

In simple language, multi-leader replication is like having two people editing the same Google Doc at the same time. It works, but we need a plan for when their edits conflict.

Leaderless Replication (Dynamo-Style)

No leader at all. Any node can accept reads and writes. The client sends writes to multiple nodes at once. This is the approach Amazon DynamoDB and Apache Cassandra use.

It uses a concept called quorum. If we have n replicas:

• Write to w nodes
• Read from r nodes
• As long as w + r > n, we’re guaranteed to read the latest value

For example, with 3 replicas: write to 2, read from 2. At least one of the nodes we read from will have the latest data.

Synchronous vs Asynchronous Replication

Synchronous: The leader waits until the follower confirms it saved the data before telling the client “write successful.” Guarantees the follower is up to date, but makes writes slower.

Asynchronous: The leader tells the client “done” immediately and replicates in the background. Faster writes, but the follower might be slightly behind.

Most systems use semi-synchronous — one follower is synchronous (so at least two nodes always have the latest data), and the rest are asynchronous.

Replication Lag

With asynchronous replication, followers might be seconds (or even minutes) behind the leader. This causes some weird problems:

Read-after-write inconsistency: A user writes a comment, then immediately reads the page — but the read goes to a follower that hasn’t received the write yet. The user’s own comment is missing.

Fix: Route reads for “own data” to the leader, or track the latest write timestamp and don’t read from a follower that’s too far behind.

Monotonic read inconsistency: A user reads from Follower 1 (which is up to date), then reads from Follower 2 (which is behind). It looks like data disappeared.

Fix: Ensure each user always reads from the same replica (sticky sessions).

Causality violations: User A writes “How’s the weather?” then User B replies “Sunny!” — but a follower receives B’s message before A’s, so the reply appears before the question.

In simple language, replication lag is the gap between what the leader knows and what the followers know. Most of the time it’s milliseconds, but under heavy load it can grow and cause confusing bugs.

When to Use What

There comes a point when a single database server just can’t handle it all. The data is too big to fit on one disk, or the queries are too many for one CPU. That’s when we need sharding.

What Is Sharding?

Sharding is splitting our data across multiple database servers. Each server holds a subset of the data (called a shard or partition). Together, all shards hold the complete dataset.

Think of it like splitting a phone book into volumes: A-F goes to Server 1, G-M goes to Server 2, N-Z goes to Server 3. Each server only deals with its portion.

Horizontal vs Vertical Partitioning

Horizontal sharding (row-based): We split rows across servers. All servers have the same columns, but different rows. This is what people usually mean when they say “sharding.”

Vertical partitioning (column-based): We split columns across servers. One server holds user_id, name, email and another holds user_id, profile_pic, bio. This is more like splitting a big table into smaller, focused tables.

Most of the time, we’re talking about horizontal sharding.

Sharding Strategies

Hash-Based Sharding

We run the shard key through a hash function: shard = hash(user_id) % number_of_shards. This distributes data evenly across shards.

Pros: Even distribution, simple. Cons: Range queries become expensive (we have to query all shards). Adding a new shard means rehashing everything. This is why consistent hashing exists (next topic).

Range-Based Sharding

We assign ranges to each shard. Users with IDs 1-1M go to Shard 1, 1M-2M go to Shard 2, and so on. Or we could shard by date: January data on one shard, February on another.

Pros: Range queries are efficient (all data in a range lives on one shard). Cons: Can create hotspots. If most new users sign up this month, the “current month” shard gets hammered while older shards sit idle.

Directory-Based Sharding

We maintain a lookup table that maps each key to its shard. A separate service says “user 42 lives on Shard 3.”

Pros: Flexible — we can move data between shards without changing the sharding logic. Cons: The lookup service becomes a single point of failure and a bottleneck.

Problems with Sharding

Sharding introduces real complexity. Here’s what we have to deal with:

Cross-Shard Joins

If a user is on Shard 1 and their orders are on Shard 2, doing a JOIN across shards is a nightmare. We either have to query both shards and merge in the application, or denormalize our data so related data lives on the same shard.

Resharding

When we add or remove shards, we have to redistribute data. With hash-based sharding, hash(key) % 3 gives completely different results than hash(key) % 4. So adding one shard means moving most of our data. This is why consistent hashing was invented — it minimizes data movement.

Hotspots

Even with good distribution, some keys get way more traffic. A celebrity’s profile on a social network gets millions of reads while a regular user gets a few. That one shard handling the celebrity becomes a hotspot.

Mitigation: Add a random suffix to hot keys to spread them across shards, or handle hot keys specially in the application.

Referential Integrity

Foreign key constraints don’t work across shards. We lose the database’s help in enforcing relationships and have to handle it in application code.

When Do We Actually Need Sharding?

Sharding is a last resort, not a first step. Before sharding, we should try:

1. Vertical scaling — get a bigger machine (more RAM, faster SSD)
2. Read replicas — offload reads to follower databases
3. Caching — put Redis in front of the database
4. Query optimization — add indexes, rewrite slow queries

If all of that isn’t enough and our database is still struggling, then it’s time to shard.

In simple language, sharding is like splitting a library into multiple buildings. Each building has a portion of the books. It lets us serve way more readers, but finding a book now requires knowing which building it’s in. And moving books between buildings is a headache.

We just learned about hash-based sharding: shard = hash(key) % N. The problem? When we add or remove a server (N changes), almost every key maps to a different server. We have to move nearly all the data. Consistent hashing fixes this.

The Problem with Regular Hashing

Say we have 3 servers and we use hash(key) % 3 to decide where data goes.

hash("user:1") % 3 = 1  → Server 1
hash("user:2") % 3 = 0  → Server 0
hash("user:3") % 3 = 2  → Server 2

Now we add a 4th server. Everything becomes hash(key) % 4:

hash("user:1") % 4 = 1  → Server 1 (same)
hash("user:2") % 4 = 2  → Server 2 (MOVED!)
hash("user:3") % 4 = 3  → Server 3 (MOVED!)

Most keys end up on different servers. If these are cache servers, we just invalidated almost our entire cache. If these are database shards, we need to migrate a huge amount of data. With a million keys, roughly (N-1)/N of them move. That’s brutal.

How Consistent Hashing Works

The core idea: instead of hash % N, we arrange everything on a ring (a circle from 0 to some max value).

1. Hash each server onto the ring (using the server’s name or IP)
2. Hash each key onto the same ring
3. Walk clockwise from the key’s position — the first server we hit owns that key

The magic: when we add a new server D to the ring, only the keys between D and the next server counter-clockwise need to move to D. Everything else stays put. Instead of moving ~75% of keys (with modular hashing), we move roughly 1/N of keys. Same thing when a server is removed — only its keys redistribute to the next server clockwise.

Virtual Nodes

There’s a catch with basic consistent hashing. If servers aren’t evenly spaced on the ring, some servers end up with way more keys than others. And with just 3-4 servers, the distribution is often terrible.

The fix is virtual nodes (vnodes). Instead of placing each server once on the ring, we place it many times at different positions.

Server A gets virtual nodes A-1, A-2, A-3, ... A-100 spread across the ring. More virtual nodes = more even distribution. In practice, systems use 100-200 virtual nodes per physical server.

Another cool benefit: if a server is more powerful, we give it more virtual nodes so it handles more keys. Simple and elegant.

Where Consistent Hashing Is Used

It’s everywhere in distributed systems:

• Distributed caches (Memcached, Redis Cluster) — deciding which cache server holds which key
• CDNs (Akamai) — routing content to the nearest/appropriate edge server
• Load balancers — sticky sessions where the same user always hits the same backend
• Distributed databases (Cassandra, DynamoDB) — partitioning data across nodes
• Distributed file systems — deciding which node stores which chunk of a file

The Key Insight

With regular hashing, adding or removing a server reshuffles almost everything. With consistent hashing, only K/N keys move on average (where K is total keys and N is total servers). That’s the difference between a catastrophe and a minor adjustment.

In simple language, consistent hashing is like arranging servers in a circle and letting each key find its server by walking clockwise. When a server joins or leaves, only its immediate neighbors are affected. Everyone else doesn’t even notice.

A transaction is a group of database operations that should succeed or fail as a unit. Transfer $100 from Account A to Account B? That’s two operations (debit A, credit B) — and we never want only one of them to happen.

ACID is the set of guarantees that make transactions reliable. Let’s break each one down.

The ACID Properties

Atomicity — All or Nothing

Think of it like a light switch: it’s either ON or OFF. There’s no half-ON. If any operation in a transaction fails, all operations are rolled back. The database looks like the transaction never started.

Real-world analogy: We’re transferring money. Debit Account A by $100, credit Account B by $100. If the credit fails, the debit is undone. We never lose $100 into thin air.

BEGIN;
  UPDATE accounts SET balance = balance - 100 WHERE id = 'A';
  UPDATE accounts SET balance = balance + 100 WHERE id = 'B';
COMMIT;
-- If anything fails between BEGIN and COMMIT, everything rolls back

Consistency — Rules Are Always Respected

After every transaction, the database must be in a valid state. All constraints, foreign keys, and rules we defined must hold.

Real-world analogy: If we have a rule that account balance can never go negative, the database will reject any transaction that violates it. We can’t transfer $500 from an account with $300.

The only difference from the other properties: consistency is partly the application’s responsibility. The database enforces constraints we define, but our application logic also needs to make sense.

Isolation — Transactions Don’t Peek at Each Other

Even when multiple transactions run at the same time, each one behaves as if it’s the only one running. One transaction’s uncommitted changes are invisible to other transactions.

Real-world analogy: Two people at different ATMs both checking the same account balance. Neither should see the other’s half-finished transaction.

This is the trickiest property. Full isolation is expensive, so databases offer different isolation levels (more on this below).

Durability — Committed Means Permanent

Once the database says “committed,” the data survives power outages, crashes, and panics. It’s written to non-volatile storage (disk), not just sitting in memory.

Real-world analogy: Once we get the receipt, the transaction is done. Even if the bank’s computers crash right after, our money is safe.

Databases achieve this by writing to a write-ahead log (WAL) before making changes. Even if the system crashes mid-write, the WAL can replay and recover.

Transaction Isolation Levels

Full isolation (Serializable) is safe but slow. So databases give us four levels to choose from, each trading safety for speed.

Read Uncommitted (Weakest)

A transaction can see another transaction’s uncommitted changes. This is almost never used because it allows dirty reads — we might read data that gets rolled back.

Read Committed

We can only see data that has been committed. No dirty reads. This is the default in PostgreSQL and most databases.

But we can still have a non-repeatable read: we read a row, another transaction modifies and commits it, and when we read it again within the same transaction, we get a different value.

Repeatable Read

Once we read a row, we’ll always see the same value for the rest of our transaction, even if another transaction changes it. No dirty reads, no non-repeatable reads.

But we can still get phantom reads: we run a query and get 10 rows. Another transaction inserts a new row that matches our query. If we run the same query again, we get 11 rows. The new row is a “phantom.”

Serializable (Strongest)

Transactions behave as if they ran one after another, in sequence. No dirty reads, no non-repeatable reads, no phantom reads. Complete safety.

The trade-off? It’s the slowest level. The database may need to lock rows, detect conflicts, or abort and retry transactions.

Read Anomalies Cheat Sheet

When ACID Matters vs When We Can Relax

ACID is critical for:

• Financial systems (bank transfers, payments)
• Inventory management (don’t oversell)
• Booking systems (don’t double-book a seat)
• Anything where wrong data = real money lost

We can relax for:

• Social media feeds (seeing a post 2 seconds late is fine)
• Analytics and reporting (slightly stale data is acceptable)
• Caching layers (eventual consistency is OK)
• Logging and metrics (losing one log line isn’t the end of the world)

This is the ACID vs BASE trade-off we mentioned in the SQL vs NoSQL topic. Many NoSQL databases choose availability and partition tolerance (BASE) over strict ACID, because for their use cases, it’s the right call.

In simple language, ACID is a promise from the database: “Your transaction will either happen completely and correctly, or not at all. And once it’s done, it’s done forever.” The isolation levels let us tune how strict that promise is, depending on how much speed we need.

Scaling is what we do when our system can’t handle the load anymore. Users are growing, requests are piling up, and the server starts sweating. We have exactly two options: make the machine bigger, or add more machines.

What Is Scaling?

Scaling means increasing our system’s capacity to handle more traffic, more data, or more users. Every system hits a ceiling at some point. The question is — how do we raise that ceiling?

Vertical Scaling (Scale Up)

Vertical scaling means upgrading our existing machine. More CPU, more RAM, bigger disk. We take our one server and make it beefier.

Think of it like replacing a small car with a truck. Same number of vehicles, just a more powerful one.

Pros:

• Dead simple — no code changes needed
• No distributed system complexity
• Data consistency is easy (one machine = one source of truth)
• Lower latency between components (everything is local)

Cons:

• There’s a hard ceiling — even the biggest machine on AWS has limits
• Single point of failure — if that one machine dies, everything dies
• Expensive — high-end hardware gets disproportionately pricier
• Downtime during upgrades (usually need to restart)

Horizontal Scaling (Scale Out)

Horizontal scaling means adding more machines to share the load. Instead of one beefy server, we run 10 smaller ones behind a load balancer.

Think of it like adding more cars to a delivery fleet instead of buying one mega-truck.

Pros:

• No theoretical limit — just keep adding machines
• Better fault tolerance — one machine dies, others keep running
• Cost-effective — commodity hardware is cheap
• Can scale on demand (add machines during peak, remove after)

Cons:

• Distributed system complexity (network failures, data consistency)
• Need a load balancer to distribute traffic
• Session management gets tricky (which server has the user’s session?)
• Data synchronization across machines is hard

Visual Comparison

When to Use What?

Why Most Large Systems Go Horizontal

Here’s the reality: vertical scaling buys us time, but horizontal scaling is what the big players use.

Netflix, Google, Amazon — they all run thousands of small machines, not one supercomputer. The reasons:

1. No single point of failure — a server dying is expected, not catastrophic
2. Linear cost scaling — 10 small machines cost less than 1 giant one
3. Geographic distribution — we can place machines closer to users worldwide
4. Cloud-native — modern cloud platforms are built for horizontal scaling

Real-World Examples

• Instagram started on a single server. As they grew, they moved to horizontally scaled web servers + vertically scaled database servers (before eventually sharding the DB too).
• Databases often scale vertically first because distributing data is harder than distributing stateless logic.
• Kubernetes is essentially a tool for managing horizontal scaling — run more pods when traffic increases.

Key Takeaway

In simple language, vertical scaling is buying a bigger box, horizontal scaling is buying more boxes. Start vertical for simplicity, but design our code to be stateless so we can go horizontal when the time comes. Most production systems end up using a mix of both.

When we build a backend, we have a fundamental choice: put everything in one application (monolith) or split it into many small, independent services (microservices). Both are valid. The right choice depends on where we are, not where we want to be.

What Is a Monolith?

A monolith is a single application where all the code lives together. User auth, payments, notifications, search — everything is in one codebase, one deployment, one process.

Think of it like a restaurant where one chef does everything: takes orders, cooks, plates, and serves. Simple when it’s a small restaurant.

Advantages:

• Easy to develop and debug — everything is in one place
• Simple deployment — one build, one deploy
• No network calls between components — just function calls
• Easy to test end-to-end
• Great for small teams (2-10 devs)

Disadvantages:

• Gets messy as the codebase grows (the “big ball of mud”)
• One bug can bring down the entire system
• Scaling means scaling everything, even the parts that don’t need it
• Slow CI/CD — small change requires full rebuild and redeploy
• Hard for multiple teams to work on without stepping on each other

What Are Microservices?

Microservices split the application into small, independent services. Each service owns one piece of functionality, has its own database, and can be deployed independently.

Think of it like a food court — each stall specializes in one thing. The pizza stall doesn’t need to know how the sushi stall works.

Advantages:

• Independent deployment — update one service without touching others
• Scale individual services based on demand
• Teams can own services end-to-end
• Tech diversity — each service can use the best tool for its job
• Fault isolation — one service crashing doesn’t kill everything

Disadvantages:

• Distributed system complexity (network failures, latency)
• Data consistency across services is genuinely hard
• Debugging a request that spans 5 services is painful
• More infrastructure to manage (service discovery, monitoring, logging)
• Operational overhead — we need DevOps maturity

The Architecture Difference

Communication Between Services

When services are split, they need to talk to each other. Two main approaches:

Synchronous (HTTP/gRPC)

Service A calls Service B and waits for a response. Like a phone call.

Order Service --HTTP POST--> Payment Service
               <--200 OK---

• Simple to understand and implement
• But creates tight coupling — if Payment Service is down, Order Service is stuck
• Cascading failures are a real risk

Asynchronous (Message Queues)

Service A puts a message on a queue and moves on. Service B picks it up whenever it’s ready. Like sending a text message.

Order Service --publish--> [Message Queue] --consume--> Payment Service

• Services are decoupled — they don’t need to be online at the same time
• Better fault tolerance — messages wait in the queue
• But adds complexity (message ordering, duplicate handling, eventual consistency)

Most production systems use a mix of both. Synchronous for things that need an immediate response (login, checkout), asynchronous for things that can happen later (send email, generate report).

Service Discovery

In a monolith, calling another module is just a function call. In microservices, we need to know where the other service lives. That’s service discovery.

Two approaches:

• Client-side discovery — the caller asks a registry (like Consul or Eureka) for the service address
• Server-side discovery — a load balancer or API gateway handles routing (Kubernetes does this with its built-in DNS)

Kubernetes makes this almost free. Each service gets a DNS name like payment-service.default.svc.cluster.local, and K8s handles the rest.

The “Monolith First” Approach

Martin Fowler’s advice (and most experienced engineers agree): start with a monolith.

Why? Because:

1. We don’t know our domain boundaries well enough at the start
2. Splitting too early creates the wrong service boundaries
3. Monoliths are faster to iterate on when we’re finding product-market fit
4. We can always split later once we understand the pain points

The pattern looks like this:

1. Start: Monolith with clean module boundaries
2. Grow: Identify which modules cause the most scaling/deployment pain
3. Extract: Pull those modules out into separate services one at a time
4. Repeat: Keep extracting as needed

Amazon, Netflix, and Uber all started as monoliths before moving to microservices.

When to Go Microservices

Microservices make sense when:

• Multiple teams need to deploy independently
• Different parts of the system have wildly different scaling needs
• We need fault isolation (one feature crashing can’t kill the whole app)
• The codebase is so large that build times are painful
• We need technology diversity (ML in Python, API in Go, etc.)

Key Takeaway

In simple language, a monolith is one big app, microservices are many small apps that talk to each other. Monoliths aren’t bad — they’re the right starting point. Microservices aren’t magic — they trade code complexity for operational complexity. Pick based on team size, scale needs, and how well we understand our domain.

When we move from a monolith to microservices, our clients suddenly need to know about 10 different services with 10 different URLs. That’s a mess. An API Gateway solves this by giving clients one single door to walk through. Behind that door, it routes requests to the right service.

What Is an API Gateway?

An API Gateway is a server that sits between clients and our backend services. Every request goes through it first. It’s the receptionist of our microservices architecture — it knows where everything is and handles common tasks so individual services don’t have to.

What Does It Do?

An API Gateway handles a bunch of cross-cutting concerns:

• Request Routing — Routes /api/users to the User Service, /api/orders to the Order Service
• Authentication & Authorization — Validates JWT tokens, API keys, or OAuth before the request even reaches a service
• Rate Limiting — Stops a single client from hammering our services (e.g., 100 requests/minute)
• Load Balancing — Distributes requests across multiple instances of a service
• Request/Response Transformation — Converts between protocols (REST to gRPC, XML to JSON)
• Response Aggregation — Combines responses from multiple services into one response for the client
• Caching — Caches frequent responses to reduce load on services
• Logging & Monitoring — Centralized request logging, metrics, and tracing

Where It Sits in the Architecture

Without a gateway, every client would need to know every service URL, handle auth separately, and manage its own retry logic. That’s a nightmare, especially for mobile apps.

API Gateway vs Load Balancer

This comes up a lot in interviews. They’re related but different:

In simple language, a load balancer says “here are 5 copies of the same thing, pick one.” An API gateway says “what are we looking for? Let me take us to the right place.”

In practice, we often use both. The API gateway routes to the right service, and a load balancer in front of each service distributes traffic across its instances.

Response Aggregation — The Big Win

One of the most valuable things an API gateway does is aggregation. Say a mobile app’s home screen needs data from 3 services:

Without a gateway: the app makes 3 separate HTTP calls (slow on mobile, battery drain).

With a gateway: the app makes 1 call, the gateway fans out to 3 services internally, combines the results, and returns a single response. Way faster for the client.

Client → GET /api/home

Gateway internally:
  → GET /users/123      → User Service
  → GET /orders/recent  → Order Service
  → GET /recommendations → Rec Service

Gateway combines all three and returns one response.

Popular API Gateway Tools

• Kong — Open-source, plugin-based, built on Nginx. Very popular.
• AWS API Gateway — Fully managed, integrates with Lambda and other AWS services. Pay per request.
• Nginx — Can act as a basic API gateway with reverse proxy config. Lightweight and fast.
• Traefik — Cloud-native, auto-discovers services in Docker/Kubernetes.
• Express Gateway — Node.js-based, good for teams already in the JS ecosystem.

For Kubernetes, an Ingress Controller (like Nginx Ingress or Traefik) often serves as the API gateway.

Potential Downsides

The API gateway is powerful, but it’s not free:

• Single point of failure — If the gateway goes down, everything goes down. We need it highly available (multiple instances + health checks).
• Added latency — Every request goes through an extra hop. Usually negligible, but it adds up.
• Can become a bottleneck — All traffic funnels through it. We need to scale the gateway itself.
• Complexity — More infrastructure to maintain, configure, and monitor.

Key Takeaway

An API Gateway is the front door to our microservices. It handles the boring-but-critical stuff (auth, routing, rate limiting, logging) so our services can focus on business logic. In a monolith we don’t need one. In a microservices architecture, we almost always do.

In a normalized database, we avoid duplicate data. That’s great for writes and data integrity. But when our app grows and reads start dominating (most apps are 90% reads, 10% writes), all those JOINs across perfectly normalized tables start crushing performance. That’s when denormalization and read-write separation come in.

What Is Denormalization?

Denormalization is intentionally adding redundant data to our database to speed up reads. We’re trading storage space and write complexity for faster queries.

In a normalized database, we’d store a user’s name in the users table only. If we need it on a post, we JOIN. In a denormalized database, we store the user’s name directly on the post too.

Normalization vs Denormalization

Example: Normalizing a User Feed

Say we’re building a social feed. Normalized approach:

-- Normalized: 3 tables, need JOINs to display a post
SELECT p.content, p.created_at, u.name, u.avatar_url,
       COUNT(l.id) as like_count
FROM posts p
JOIN users u ON p.user_id = u.id
LEFT JOIN likes l ON l.post_id = p.id
WHERE p.id = 123
GROUP BY p.id, u.id;

This works fine for 1,000 users. At 10 million users with millions of posts, this JOIN is painful. Denormalized approach:

-- Denormalized: everything we need is right on the posts table
SELECT content, created_at, author_name, author_avatar, like_count
FROM posts
WHERE id = 123;

One table, zero JOINs, blazing fast. The only difference is we duplicated author_name, author_avatar, and pre-computed like_count onto the posts table.

The trade-off? When a user changes their name, we need to update it on all their posts too. That’s the price of denormalization.

When to Denormalize

Denormalization makes sense when:

• Read queries are way more frequent than writes
• JOINs are becoming the bottleneck (check query plans)
• We need very low latency reads (real-time feeds, dashboards)
• The duplicated data changes infrequently (user names don’t change every minute)

Don’t denormalize when:

• Our data changes frequently and consistency is critical (bank accounts)
• We’re still small — premature optimization is the root of all evil
• We haven’t profiled our queries to confirm JOINs are the actual bottleneck

Read-Write Separation (CQRS Lite)

The idea is simple: use separate database instances for reads and writes.

The primary (master) database handles all writes. One or more read replicas handle all reads. The primary replicates data to the replicas asynchronously.

Writes → Primary DB ──replication──→ Read Replica 1
                                  ──→ Read Replica 2
                                  ──→ Read Replica 3
Reads  → Load Balancer → [Replica 1, Replica 2, Replica 3]

This works because most apps are read-heavy. If 90% of our queries are reads, we just offload them to replicas and our primary DB breathes again.

Setting It Up

Most managed databases make this easy:

• AWS RDS — Click “Create read replica” and we’re done
• PostgreSQL — Streaming replication to standby servers
• MySQL — Built-in master-slave replication

Our application code needs to know which database to talk to:

# Pseudocode
def get_user(user_id):
    db = read_replica()  # reads go to replica
    return db.query("SELECT * FROM users WHERE id = %s", user_id)

def update_user(user_id, data):
    db = primary()  # writes go to primary
    db.execute("UPDATE users SET name = %s WHERE id = %s", data.name, user_id)

The Replication Lag Problem

Here’s the catch: replication isn’t instant. There’s a small delay (usually milliseconds, sometimes seconds) between a write hitting the primary and showing up on replicas.

This means: a user updates their profile, refreshes the page, and sees the old data because the read hit a replica that hasn’t caught up yet.

Solutions:

• Read-your-own-writes — After a write, route that user’s reads to the primary for a few seconds
• Sticky sessions — Route a user to the same replica consistently
• Accept it — For many use cases (feeds, dashboards), slight staleness is fine

Full CQRS (The Advanced Version)

CQRS (Command Query Responsibility Segregation) takes read-write separation further. Instead of just separate DB instances, we use entirely different data models for reads and writes.

• Write model — Normalized, optimized for data integrity
• Read model — Denormalized, optimized for query patterns (maybe even a different database like Elasticsearch)

Events sync data from the write model to the read model. This is powerful but complex — we only need full CQRS for systems with very different read and write patterns (think: analytics dashboards reading from a normalized transactional DB).

For most systems, simple read replicas are more than enough.

Key Takeaway

In simple language, denormalization is copying data so we read faster but write slower. Read-write separation is splitting our database so reads don’t compete with writes. Both are tools for scaling read-heavy systems. Start normalized, measure our bottlenecks, then denormalize strategically where it hurts most.

Every app eventually needs to store files — profile pictures, uploaded documents, video content, log exports. We can’t just throw these into our database. That’s where object storage comes in. It’s purpose-built for storing massive amounts of unstructured data cheaply and reliably.

What Is Object Storage?

Object storage treats every file as a standalone object with three parts:

1. Data — the actual file bytes (an image, a PDF, a video)
2. Metadata — info about the file (content type, upload date, custom tags)
3. Unique key — a flat identifier like users/123/avatar.jpg

There’s no folder hierarchy like a regular filesystem. The “folders” we see in S3 are just key prefixes — it’s all a flat namespace under the hood.

Three Types of Storage Compared

The key insight: block storage is for OS and databases (needs to be fast, attached to one machine). File storage is for shared access between servers. Object storage is for everything else — and that “everything else” is usually the bulk of our data.

When to Use Object Storage

Pretty much anytime we deal with files:

• User uploads — profile pictures, documents, attachments
• Media — images, videos, audio files
• Static assets — CSS, JS, fonts for a website
• Backups — database dumps, log archives
• Data lakes — raw data for analytics pipelines
• ML artifacts — model files, training datasets

The rule of thumb: if it’s a file and we don’t need to query inside it, object storage is the answer.

Popular Object Storage Services

• Amazon S3 — The OG. “S3” has basically become a generic term for object storage. 11 nines of durability (99.999999999%). Pretty wild.
• Google Cloud Storage (GCS) — Google’s version. Very similar API. Tight integration with BigQuery and other GCP services.
• Azure Blob Storage — Microsoft’s offering. Three tiers: Hot, Cool, and Archive based on access frequency.
• MinIO — Open-source, S3-compatible. Great for self-hosting or local development.
• Cloudflare R2 — No egress fees (a big deal — S3 egress costs add up fast).

Pre-Signed URLs — The Smart Pattern

Here’s a common mistake: uploading files through our backend server. The file goes Client —> Our Server —> S3. Our server becomes a bottleneck and wastes bandwidth.

The better approach is pre-signed URLs. Our server generates a temporary, signed URL that lets the client upload directly to S3.

1. Client asks our server: "I want to upload a file"
2. Server generates a pre-signed URL (valid for 15 minutes)
3. Client uploads directly to S3 using that URL
4. Client tells our server: "Upload done, here's the key"
5. Server saves the file reference in the database

Same pattern works for downloads. Instead of streaming the file through our server, we generate a pre-signed download URL and redirect the client to it.

# Generating a pre-signed upload URL (Python + boto3)
import boto3

s3 = boto3.client('s3')
url = s3.generate_presigned_url(
    'put_object',
    Params={'Bucket': 'my-bucket', 'Key': 'uploads/photo.jpg'},
    ExpiresIn=900  # 15 minutes
)
# Give this URL to the client — they can PUT directly to S3

Benefits:

• Our server doesn’t touch the file data — no bandwidth or CPU wasted
• Scales better — S3 handles the heavy lifting
• Secure — URL expires, and we control who can generate them

CDN Integration

Object storage and CDNs are best friends. The pattern:

1. Store files in S3 (or any object storage)
2. Put a CDN (CloudFront, Cloudflare) in front of it
3. Users download from the CDN edge server closest to them instead of hitting S3 directly

User in Tokyo → CloudFront Edge (Tokyo) → S3 (us-east-1)
                ↑ cached here after first request

This gives us:

• Lower latency — files served from the nearest edge location
• Lower cost — CDN egress is often cheaper than S3 egress, and caching reduces S3 requests
• Less load on S3 — the CDN absorbs most of the traffic

For static websites, this combo (S3 + CloudFront) is the go-to architecture. Fast, cheap, and almost infinitely scalable.

Storage Classes and Costs

Most providers offer tiered storage for different access patterns:

• Standard/Hot — Frequently accessed data. Highest storage cost, lowest retrieval cost.
• Infrequent Access — Data accessed less than once a month. Cheaper storage, small retrieval fee.
• Archive/Cold — Data rarely accessed (compliance, old backups). Very cheap storage, expensive and slow retrieval (hours).

Setting up lifecycle policies to automatically move old data to cheaper tiers is one of the easiest cost optimizations we can make.

Key Takeaway

In simple language, object storage is where we put files that our application needs but our database shouldn’t hold. Use pre-signed URLs so files go directly between the client and storage without touching our server. Pair it with a CDN for fast delivery worldwide. It’s one of the simplest and most impactful architectural decisions we’ll make.

The CAP theorem is one of the most important ideas in distributed systems. It says that when we build a system spread across multiple machines, we can only guarantee two out of three properties at the same time. Understanding this helps us make smart trade-offs when picking databases and designing architectures.

The Three Properties

Consistency (C) — Every read gets the most recent write. If we write “balance = $500,” every node in the system immediately returns $500. No stale data, ever.

Availability (A) — Every request gets a response (success or failure), even if some nodes are down. The system never says “sorry, I can’t respond right now.”

Partition Tolerance (P) — The system keeps working even when network communication between nodes breaks. Messages get dropped or delayed, and the system handles it.

Why We Can Only Pick Two

Think of it like this. We have two database nodes, Node A and Node B, that are supposed to stay in sync. Suddenly the network between them breaks (a partition).

Now a write comes in to Node A: “update balance to $500.”

We have two choices:

1. Stay consistent (CP) — Node B rejects reads because it can’t confirm it has the latest data. We sacrifice availability.
2. Stay available (AP) — Node B serves reads with its old data. We sacrifice consistency.

We can’t have both. If the network is broken, we either serve stale data (lose C) or refuse to respond (lose A). There’s no trick around this.

Partition Tolerance Is Not Optional

Here’s the thing most people miss: we always have to pick P. Network partitions happen in the real world. Cables get cut. Switches fail. Cloud regions have outages. If our system can’t handle a network partition, it’s not a real distributed system.

So the real choice is between CP and AP. It’s never “CA” in a distributed system.

CP vs AP Systems

CP Systems (Consistency + Partition Tolerance)

These systems prioritize correctness. During a partition, they’d rather refuse some requests than serve stale data.

• MongoDB — In a replica set, if the primary goes down, writes are refused until a new primary is elected
• HBase — Strong consistency via a single region server per row
• Redis (single node) — All reads/writes go to one node, so it’s always consistent
• Zookeeper — Used for distributed coordination where correctness is everything

Best for: Banking, inventory systems, anything where serving wrong data is worse than being temporarily unavailable.

AP Systems (Availability + Partition Tolerance)

These systems prioritize uptime. During a partition, they keep serving requests even if the data might be slightly stale.

• Cassandra — Writes succeed on any available node, syncs later
• DynamoDB — Designed for “always available” at Amazon scale
• CouchDB — Multi-master replication, resolves conflicts later

Best for: Social media feeds, shopping carts, analytics — where showing slightly old data is fine but downtime is not.

How to Decide: CP or AP?

Ask ourselves these questions:

1. Is stale data dangerous? If a user sees an old bank balance and overdrafts, that’s bad → CP
2. Is downtime unacceptable? If our e-commerce site goes down during Black Friday, we lose millions → AP
3. Can we fix inconsistency later? If we can reconcile data after the partition heals → AP is fine

Most real-world systems aren’t purely CP or AP. They’re tunable. Cassandra, for example, lets us choose consistency level per query. We can read with QUORUM for important data (more consistent) and ONE for less critical data (more available).

Common Misconceptions

“CAP means we pick exactly two.” Not really. In normal operation (no partition), we get all three. CAP only forces a choice during a partition event.

“CA systems exist in distributed setups.” Nope. If we’re distributed, partitions will happen. CA only applies to single-node databases.

“CP means zero availability.” No. CP systems are available most of the time. They only sacrifice availability during actual partition events, which are (hopefully) rare.

Key Takeaway

In simple language, CAP theorem tells us that in a distributed system, when the network breaks, we have to choose: do we serve possibly-stale data (AP), or do we refuse requests until we’re sure the data is correct (CP)? Neither choice is wrong — it depends on what our system is doing. Banks pick consistency. Social media picks availability. The key is knowing which trade-off we’re making and why.

When we have data spread across multiple servers, the big question is: how “fresh” does the data need to be when someone reads it? The answer depends on the consistency model we choose. Different models give different guarantees, and there’s always a trade-off between correctness and performance.

The Consistency Spectrum

The further left we go, the more “correct” our reads are — but the slower and harder to scale. The further right, the faster and more available — but we might read stale data.

Strong Consistency (Linearizability)

Every read returns the most recent write. Period. If User A writes “balance = $500” and User B reads right after, User B sees $500. There’s no window where stale data is visible.

Think of it like a single database with one copy of the data. Even if there are multiple replicas behind the scenes, the system behaves as if there’s only one copy.

How it works: The system coordinates between all replicas before confirming a write. This takes time — every write waits for a majority of nodes to agree.

Examples:

• Google Spanner (uses synchronized clocks across data centers)
• CockroachDB
• Single-node PostgreSQL (trivially strong — there’s only one copy)

Best for: Bank accounts, stock trading, inventory counts — anywhere wrong data causes real damage.

Trade-off: Higher latency on writes. More coordination overhead. Less availability during partitions.

Eventual Consistency

If we stop writing, eventually all replicas will converge to the same value. But right after a write, different replicas might return different values. There’s no guarantee on when they’ll catch up — could be milliseconds, could be seconds.

Think of it like updating a Google Doc. If two people edit at the same time, each person sees their own version briefly. Eventually it all syncs up.

Examples:

• Amazon DynamoDB (default mode)
• Cassandra (with consistency level ONE)
• DNS propagation — we update a DNS record and it takes hours to propagate everywhere

Best for: Social media likes, view counts, recommendation feeds — where showing a count of 4,823 instead of 4,825 for a few seconds is perfectly fine.

Trade-off: Fast writes and high availability, but we might read stale data.

Causal Consistency

This one is clever. It preserves cause-and-effect ordering. If event A caused event B, everyone sees A before B. But unrelated events can appear in any order.

Example: I post a message “Hey, who wants pizza?” (event A), then someone replies “I do!” (event B). Causal consistency guarantees everyone sees my question before the reply. But two unrelated posts from different users can appear in any order.

Examples:

• MongoDB (with causal sessions)
• Some configurations of Cassandra

Best for: Comment threads, chat applications — where ordering within a conversation matters but global ordering doesn’t.

Read-Your-Writes Consistency

After we write something, we are guaranteed to see our own write. Other users might see stale data temporarily, but the person who made the change always sees it reflected.

This solves the classic problem: a user updates their profile name, refreshes the page, and sees the old name. Infuriating. With read-your-writes, that doesn’t happen.

How to implement it:

• Route the user’s reads to the same replica that processed their write
• After a write, read from the primary for a few seconds
• Include a timestamp token — only accept reads from replicas that are at least that fresh

Best for: User profile updates, settings changes — anywhere the “I just changed this, why doesn’t it show?” experience is bad.

Monotonic Reads

Once we’ve read a value, we never see an older value on subsequent reads. Our reads only move forward in time, never backward.

Without this, a user might refresh a page and see their feed go backward — posts disappear and reappear. That happens when successive reads hit different replicas at different replication states.

How to implement: Use sticky sessions — route a user to the same replica consistently.

How Real Systems Handle This

Most databases let us tune consistency per operation. We don’t have to pick one model for everything.

Picking the Right Model

The decision comes down to two questions:

1. What happens if a user reads stale data?

• Nothing bad → eventual consistency is fine
• Confusion or bad UX → read-your-writes
• Financial loss or data corruption → strong consistency

2. How much latency can we tolerate?

• Milliseconds matter → eventual consistency (fastest)
• Some latency is OK → causal or read-your-writes
• Correctness over speed → strong consistency

A Practical Example

Imagine we’re building an e-commerce platform:

• Product catalog (price, description) → Eventual consistency. If a price update takes a second to propagate, nobody gets hurt.
• User sessions → Read-your-writes. A user logs in and should immediately see their cart.
• Inventory count → Strong consistency. We can’t sell 5 items when only 3 are in stock.
• Review counts → Eventual consistency. Showing “142 reviews” when it’s actually 143 for a few seconds is harmless.

Different parts of the same system can (and should) use different consistency models.

Key Takeaway

In simple language, consistency models are about how fresh our data needs to be when someone reads it. Strong consistency means always fresh — but slow. Eventual consistency means sometimes stale — but fast. Most real systems mix and match: strong where it matters (money, inventory), eventual where it doesn’t (likes, views). The skill is knowing which parts of our system need which guarantee.

Things break. Servers crash, disks fail, networks go down, entire data centers lose power. The question isn’t if something will fail — it’s when. Failover and redundancy are how we design systems that keep working even when parts of them die.

What Is Redundancy?

Redundancy means having backup copies of everything critical. If one server dies, another takes over. If one database crashes, a replica is ready to go. We eliminate single points of failure.

Think of it like having a spare tire in our car. We don’t plan to get a flat, but when it happens, we’re not stuck on the side of the road.

What Is Failover?

Failover is the process of switching from a failed component to a healthy backup. It’s the mechanism that actually makes redundancy useful.

Without failover, having a backup database is like having a spare tire but no jack — the backup exists but we can’t switch to it.

Active-Passive Failover

The most common pattern. One server (the active/primary) handles all traffic. One or more backup servers (the passive/standby) sit idle, receiving replicated data but serving no traffic.

When the primary dies, a standby gets promoted to primary and starts handling traffic.

Pros:

• Simple to set up
• No data conflict issues (only one node writes at a time)
• Standby can also serve read queries (read replicas)

Cons:

• The standby is wasted capacity — it sits idle most of the time
• Failover isn’t instant — there’s a brief downtime during switchover
• Risk of data loss if the primary dies before replicating its latest writes

Active-Active Failover

All servers handle traffic simultaneously. A load balancer splits requests across them. If one dies, the others absorb the extra load.

Pros:

• No wasted capacity — every server is doing useful work
• Better performance — load is distributed
• Faster failover — no promotion step, traffic just routes around the dead node

Cons:

• More complex — we need to handle data conflicts (what if two servers accept conflicting writes?)
• Need a load balancer in front
• Data synchronization is harder

Best for: Stateless application servers (easy), databases with multi-master replication (hard but possible with Cassandra, CockroachDB).

Redundancy at Every Layer

A chain is only as strong as its weakest link. We need redundancy at every layer of our stack:

The most overlooked one? The load balancer itself. If we put all our servers behind a single load balancer and that LB dies, everything is down. Always have redundant LBs.

The Nines of Availability

When we talk about uptime, we use “nines”:

Going from 99.9% to 99.99% is exponentially harder and more expensive. Most web apps aim for three nines (99.9%). Banks and critical infra aim for four or five nines.

SLA vs SLO vs SLI

These terms come up in interviews:

• SLI (Service Level Indicator) — The actual metric. “Our p99 latency is 200ms.”
• SLO (Service Level Objective) — Our internal target. “We aim for 99.9% uptime.”
• SLA (Service Level Agreement) — A legal promise to customers. “If we drop below 99.9%, we give you credits.”

The SLA is always looser than the SLO. We target 99.95% internally (SLO) so we don’t breach our 99.9% customer promise (SLA).

Health Checks and Heartbeats

How does the system know something is dead? Two main approaches:

Health checks (pull-based): A monitor pings each server periodically. “Are you alive? Return 200 OK.” If a server doesn’t respond after several retries, it’s considered dead.

Heartbeats (push-based): Each server periodically sends a “I’m alive” signal. If the monitor doesn’t hear from a server within a timeout, it’s considered dead.

Most load balancers use health checks. Most cluster managers (like Kubernetes) use a combination of both.

Health Check: Monitor → "GET /health" → Server → "200 OK"
             Monitor → "GET /health" → Server → (no response)
             Monitor → "GET /health" → Server → (no response)
             Monitor: "Server is dead, removing from pool"

Failover Strategies for Databases

Database failover is the trickiest because data is involved:

1. Cold standby — Standby is off, brought online from backups. Slowest recovery (minutes to hours).
2. Warm standby — Standby receives replicated data but doesn’t serve traffic. Moderate recovery (seconds to minutes).
3. Hot standby — Standby is fully synced and ready to take over immediately. Fastest recovery (seconds).

Most cloud-managed databases (RDS, Cloud SQL) use warm or hot standby with automatic failover.

Key Takeaway

In simple language, redundancy is having backups of everything, and failover is the process of switching to those backups when something breaks. Active-passive is simpler (one server works, one waits). Active-active is more efficient (all servers work, survivors absorb the load). Build redundancy at every layer, understand our uptime target in nines, and make sure our health checks actually work. The goal is making our system boring — it keeps running no matter what breaks.

In a distributed system, one failing service can take down everything else. Imagine our payment service is slow. Every request to it hangs for 30 seconds. Our order service calls payment and waits. Its threads get consumed. Now the order service can’t handle any requests either. Soon, the whole system is frozen. This is a cascading failure, and it’s one of the scariest things in distributed systems.

Circuit breakers and bulkheads are patterns that prevent this.

Circuit Breaker Pattern

Think of it like an electrical circuit breaker in our house. When there’s a power surge, the breaker trips and cuts the circuit — protecting everything from frying. Same idea in software.

A circuit breaker wraps calls to an external service and monitors for failures. When failures cross a threshold, the breaker “trips” and stops making calls entirely. Instead of waiting 30 seconds for a timeout, we fail instantly. This gives the downstream service time to recover without being hammered by requests.

The Three States

Closed — Everything is normal. Requests pass through to the downstream service. The breaker counts failures. If failures stay below the threshold, life is good.

Open — Too many failures happened. The breaker trips. All requests immediately fail with a fallback response. No calls are made to the downstream service at all. This state lasts for a configured timeout (e.g., 30 seconds).

Half-Open — After the timeout, the breaker lets one request through as a test. If it succeeds, the breaker closes (back to normal). If it fails, the breaker opens again for another timeout period.

What Happens When the Circuit Is Open?

We don’t just show errors. We use fallbacks:

• Return cached data (slightly stale but better than nothing)
• Return a default value (“0 items in cart” instead of crashing)
• Queue the request for later processing
• Show a degraded experience (“Recommendations unavailable”)

# Pseudocode for a circuit breaker
def get_recommendations(user_id):
    if circuit_breaker.is_open():
        return cached_recommendations(user_id)  # fallback

    try:
        result = recommendation_service.get(user_id)
        circuit_breaker.record_success()
        return result
    except TimeoutError:
        circuit_breaker.record_failure()
        return cached_recommendations(user_id)  # fallback

Configuration

Typical circuit breaker settings:

• Failure threshold: 5 failures in 60 seconds → trip
• Open duration: 30 seconds before trying half-open
• Half-open test: Allow 1 request through
• Success threshold: 3 consecutive successes in half-open → close

These numbers aren’t magic. We tune them based on our service’s behavior.

Bulkhead Pattern

The bulkhead pattern isolates different parts of our system so a failure in one doesn’t sink the rest. The name comes from ships — a ship’s hull is divided into watertight compartments (bulkheads). If one compartment floods, the others stay dry and the ship stays afloat.

In software, we create isolation boundaries:

Thread pool bulkhead: Each downstream service gets its own thread pool. If the payment service is slow and consumes all its threads, the order-processing thread pool is untouched.

┌─────────────────────────────────────┐
│         Application Server          │
│                                     │
│  ┌──────────────┐ ┌──────────────┐  │
│  │ Payment Pool │ │  Order Pool  │  │
│  │  10 threads  │ │  20 threads  │  │
│  │  ⚠️ 10/10    │ │  ✅ 5/20    │  │
│  │  (saturated) │ │  (healthy)   │  │
│  └──────────────┘ └──────────────┘  │
│                                     │
│  Payment is broken but orders       │
│  keep processing normally.          │
└─────────────────────────────────────┘

Connection pool bulkhead: Same idea but with database connections. One slow query pattern doesn’t exhaust connections needed by other queries.

Process bulkhead: Run different features as separate services or containers. If the image processing service OOMs, the authentication service keeps running.

Retry with Exponential Backoff

When a request fails, our first instinct is to retry immediately. Bad idea. If the service is overloaded, 1,000 clients all retrying instantly makes it worse. That’s a retry storm.

Instead, we use exponential backoff:

Attempt 1: wait 1 second
Attempt 2: wait 2 seconds
Attempt 3: wait 4 seconds
Attempt 4: wait 8 seconds
(give up after max retries)

We also add jitter — a random offset so all clients don’t retry at the exact same time:

Attempt 1: wait 1s + random(0-500ms)
Attempt 2: wait 2s + random(0-500ms)
Attempt 3: wait 4s + random(0-500ms)

When NOT to Retry

• 4xx errors (400, 401, 403, 404) — The request is bad. Retrying won’t fix it.
• Non-idempotent operations — If we POST a payment and it timed out, retrying might charge twice. Only retry operations that are safe to repeat.

Combining All Three

In practice, we use circuit breakers, bulkheads, and retries together:

1. Bulkhead isolates the failure so it doesn’t spread
2. Retry with backoff handles transient errors
3. Circuit breaker stops us from hammering a dead service

Request
  → Bulkhead (uses dedicated thread pool)
    → Circuit Breaker (checks if service is healthy)
      → Retry with backoff (handles transient failures)
        → Actual service call

Libraries like Resilience4j (Java), Polly (.NET), and Hystrix (Java, deprecated but influential) implement all three patterns.

Key Takeaway

In simple language, circuit breakers stop us from calling a broken service (fail fast instead of waiting forever). Bulkheads isolate failures so one broken thing doesn’t sink everything. Retries with exponential backoff handle temporary hiccups without overwhelming the system. Together, these three patterns are how we build systems that degrade gracefully instead of collapsing entirely when things go wrong.

We can design the most elegant system in the world, but if we can’t see what’s happening inside it, we’re flying blind. When something breaks at 3 AM, the question is: do we have enough information to figure out what went wrong, or are we guessing?

Monitoring, logging, and alerting are how we answer that question. In system design interviews, they’re often the final piece that shows we think about real production systems, not just architecture diagrams.

The Three Pillars of Observability

Each pillar answers a different question. We need all three for full observability.

Metrics — The Numbers

Metrics are numeric values measured over time. They tell us the health of our system at a glance.

The Four Golden Signals (from Google SRE)

1. Latency — How long requests take. Not just the average — we care about percentiles.

   • p50 (median): Half the requests are faster than this
   • p95: 95% of requests are faster — this catches the slow tail
   • p99: 99% are faster — the really unlucky users
   • A p50 of 50ms with a p99 of 5 seconds means most users are happy but 1% are miserable

2. Traffic — How many requests per second we’re handling. Helps us know when to scale.

3. Error rate — Percentage of requests that fail (5xx errors, timeouts). A sudden spike means something broke.

4. Saturation — How full our resources are. CPU at 90%? Memory almost maxed? Disk filling up? This tells us how close we are to falling over.

How Metrics Work

Most metrics systems use a pull model. Our application exposes metrics at an endpoint (like /metrics), and a collector (Prometheus) scrapes it periodically.

App Server → exposes /metrics
Prometheus → scrapes every 15 seconds
Grafana   → queries Prometheus, shows dashboards

Logs — The Details

Logs are records of discrete events. When something breaks and metrics show a spike, logs tell us why.

Structured Logging

Plain text logs are hard to search:

[2024-03-15 14:23:01] ERROR: Payment failed for user 12345

Structured logs (JSON) are much better:

{
  "timestamp": "2024-03-15T14:23:01Z",
  "level": "error",
  "service": "payment",
  "user_id": 12345,
  "error": "card_declined",
  "amount": 49.99,
  "request_id": "abc-123"
}

Structured logs let us filter and search: “show me all errors from the payment service where amount > $100 in the last hour.”

Log Aggregation

With 50 servers, we can’t SSH into each one to read logs. We ship all logs to a central system:

• ELK Stack — Elasticsearch (store/search) + Logstash (process) + Kibana (visualize). Self-hosted, powerful, complex.
• Loki + Grafana — Lightweight log aggregation. Pairs perfectly with Prometheus.
• Splunk — Enterprise log management. Expensive but very powerful.
• Datadog Logs — SaaS. Easy setup, good integration with metrics and traces.

Log Levels

Use them correctly:

• DEBUG — Verbose detail for development. Never in production.
• INFO — Normal operations. “User signed up,” “Order placed.”
• WARN — Something unexpected but not broken. “Cache miss rate high.”
• ERROR — Something failed. “Payment processing failed.”
• FATAL — The system is going down. “Database connection lost.”

Distributed Tracing — The Journey

In a microservices architecture, a single user request might touch 10 different services. When that request is slow, which service is the bottleneck?

Distributed tracing follows a request across all services by propagating a trace ID:

User Request (trace-id: abc-123)
  → API Gateway     [12ms]
    → Auth Service   [5ms]
    → Order Service  [350ms]  ← bottleneck!
      → Inventory DB [300ms]  ← root cause!
      → Payment Svc  [45ms]
    → Notification   [8ms]

Each step is a span. All spans with the same trace ID form a complete picture of the request’s journey. We can see that the order service is slow because the inventory database query took 300ms.

Alerting — The Wake-Up Call

Monitoring is useless if nobody looks at the dashboards. Alerting bridges that gap — it tells us when something needs attention.

What to Alert On

Good alerts are actionable. Every alert should require a human to do something. If we get an alert and the response is “meh, it’ll fix itself,” that’s a bad alert.

Alert on symptoms, not causes:

• Good: “Error rate exceeded 5% for 5 minutes” (symptom)
• Bad: “CPU usage above 80%” (cause — might be normal during a deploy)

Alert on SLO breaches:

• “p99 latency exceeded 500ms for the last 10 minutes”
• “Availability dropped below 99.9% for this billing period”

Alert Fatigue

The worst thing we can do is alert on everything. When teams get 200 alerts a day, they start ignoring all of them. And then they miss the real one.

Rules for good alerting:

• Every alert should have a clear owner
• Every alert should have a runbook (what to do when it fires)
• Review alerts monthly — if we keep dismissing one, delete it or fix the root cause
• Use severity levels: critical (page someone at 3 AM) vs warning (check it tomorrow)

The Monitoring Stack

A typical production setup:

In System Design Interviews

When wrapping up a system design answer, mentioning monitoring shows maturity:

• “We’d track p99 latency and error rate per service with Prometheus and Grafana”
• “Structured JSON logs shipped to ELK for debugging”
• “Distributed tracing with Jaeger so we can find bottlenecks across services”
• “Alerting on SLO breaches — if p99 exceeds 500ms for 5 minutes, page the on-call”

We don’t need to go deep. Just showing we think about operational concerns sets us apart.

Key Takeaway

In simple language, metrics tell us what’s happening (numbers over time), logs tell us why it happened (detailed events), and traces show us where it happened (a request’s journey across services). Together, they give us observability — the ability to understand our system’s behavior from the outside. Without them, debugging production issues is just guessing.

REST is the most common way to build web APIs. When we open a mobile app and it loads our feed, it’s almost certainly making REST API calls behind the scenes. Understanding REST API design is fundamental for system design interviews because every system we design will have an API layer.

What Is REST?

REST stands for Representational State Transfer. In simple language, it’s a set of conventions for building APIs over HTTP. We use URLs to identify resources, HTTP methods to define actions, and JSON to send data back and forth.

REST isn’t a protocol or a standard — it’s an architectural style. That means there’s no strict spec, just widely accepted conventions.

Core Principles

1. Resource-based — Everything is a resource (user, post, order) with a URL
2. Stateless — Each request contains everything the server needs. No session state stored between requests.
3. Uniform interface — Consistent URL patterns and HTTP methods
4. Client-server — Frontend and backend are separate and can evolve independently

HTTP Methods

Each method maps to a CRUD operation:

Idempotent means calling it multiple times has the same effect as calling it once. GET, PUT, PATCH, and DELETE are idempotent. POST is not — calling POST /orders three times creates three orders.

Status Codes

We don’t just return data — we return a status code that tells the client what happened:

Success (2xx)

• 200 OK — Request succeeded. Used for GET, PUT, PATCH.
• 201 Created — Resource created. Used for POST.
• 204 No Content — Success but nothing to return. Used for DELETE.

Client Error (4xx)

• 400 Bad Request — The request body is malformed or invalid.
• 401 Unauthorized — Not authenticated. “Who are you?”
• 403 Forbidden — Authenticated but not allowed. “I know who you are, but you can’t do this.”
• 404 Not Found — Resource doesn’t exist.
• 409 Conflict — Conflicts with current state (e.g., duplicate email).
• 429 Too Many Requests — Rate limit exceeded. Slow down.

Server Error (5xx)

• 500 Internal Server Error — Something broke on our end.
• 502 Bad Gateway — Our server got a bad response from an upstream service.
• 503 Service Unavailable — Server is overloaded or down for maintenance.

A good rule: 4xx means the client did something wrong. 5xx means we did something wrong.

URL Design

Good REST URLs are intuitive and consistent:

# Resources are nouns, not verbs
✅ GET  /users/123
❌ GET  /getUser?id=123

# Plural nouns for collections
✅ GET  /users
❌ GET  /user

# Nested resources for relationships
✅ GET  /users/123/posts        (posts by user 123)
✅ GET  /users/123/posts/456    (specific post by user 123)

# Query params for filtering, sorting, searching
✅ GET  /posts?status=published&sort=created_at&order=desc
✅ GET  /users?search=manish

# Actions that don't fit CRUD → use sub-resources
✅ POST /orders/123/cancel
✅ POST /users/123/verify-email

API Design Example: Blog Platform

# Posts
GET    /posts                    → List all posts
GET    /posts/123                → Get post 123
POST   /posts                    → Create a new post
PUT    /posts/123                → Replace post 123
PATCH  /posts/123                → Update post 123 partially
DELETE /posts/123                → Delete post 123

# Comments on a post
GET    /posts/123/comments       → List comments on post 123
POST   /posts/123/comments       → Add comment to post 123
DELETE /posts/123/comments/456   → Delete comment 456

# User's posts
GET    /users/789/posts          → All posts by user 789

A response might look like:

{
  "id": 123,
  "title": "REST API Design Guide",
  "content": "...",
  "author": {
    "id": 789,
    "name": "Manish"
  },
  "created_at": "2024-03-15T10:30:00Z",
  "tags": ["api", "design"]
}

Pagination

When a collection has thousands of items, we can’t return them all at once. Two main approaches:

Offset-Based Pagination

GET /posts?page=2&limit=20

Simple but has problems:

• If items are added/deleted between pages, we might skip or duplicate items
• OFFSET 10000 in SQL is slow — the DB still scans 10,000 rows to skip them

Cursor-Based Pagination

GET /posts?after=abc123&limit=20

The cursor (abc123) is an opaque token pointing to the last item we saw. The server fetches the next 20 items after that cursor.

• No skipping or duplicates even if data changes
• Fast — uses an index scan instead of OFFSET
• Used by Twitter, Facebook, Slack — basically everyone at scale

Response includes the next cursor:

{
  "data": [...],
  "pagination": {
    "next_cursor": "def456",
    "has_more": true
  }
}

API Versioning

APIs evolve. We can’t break existing clients when we change things.

URL versioning (most common):

GET /v1/users/123
GET /v2/users/123

Header versioning:

GET /users/123
Accept: application/vnd.myapi.v2+json

Query parameter:

GET /users/123?version=2

URL versioning is the most explicit and easiest to understand. It’s what most APIs use.

Other Best Practices

• Use JSON for request and response bodies (set Content-Type: application/json)
• Use ISO 8601 for dates — 2024-03-15T10:30:00Z, not “March 15, 2024”
• Return meaningful errors — Include an error code and message
• Rate limiting — Protect our API from abuse with 429 Too Many Requests
• HTTPS everywhere — Never serve an API over plain HTTP
• Authentication — Use Bearer tokens (JWT) or API keys in the Authorization header

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Email is required",
    "field": "email"
  }
}

Key Takeaway

In simple language, REST is about using URLs to identify things (resources), HTTP methods to do things to them (GET, POST, PUT, DELETE), and status codes to say what happened. Keep URLs clean and consistent, use cursor-based pagination for large datasets, version our API from day one, and always return helpful error messages. REST isn’t perfect for every use case, but it’s the default for good reason — it’s simple, well-understood, and works great for most APIs.

GraphQL was created by Facebook in 2012 to solve a very specific problem: their mobile app needed data from many different resources, and REST APIs were either sending too much data or requiring too many requests. Instead of the server deciding what data to return, GraphQL lets the client decide.

The Problem GraphQL Solves

Imagine we’re building a social media app. We want to show a user’s profile with their name, avatar, and last 3 posts (each with title and like count).

With REST (the problem)

Over-fetching: We call GET /users/123 and get back 30 fields — name, email, phone, address, bio, settings — when we only need name and avatar. We’re downloading data we don’t use.

Under-fetching: We need the user AND their posts. That’s two separate API calls: GET /users/123 then GET /users/123/posts. On a slow mobile connection, each round trip hurts.

The N+1 problem: We fetch 3 posts, and each post needs the author info. That’s 1 call for posts + 3 calls for authors = 4 requests total.

With GraphQL (the solution)

One request. We ask for exactly what we need:

query {
  user(id: 123) {
    name
    avatarUrl
    posts(last: 3) {
      title
      likeCount
    }
  }
}

Response — nothing more, nothing less:

{
  "data": {
    "user": {
      "name": "Manish",
      "avatarUrl": "https://...",
      "posts": [
        { "title": "System Design Basics", "likeCount": 42 },
        { "title": "CAP Theorem Explained", "likeCount": 28 },
        { "title": "REST vs GraphQL", "likeCount": 35 }
      ]
    }
  }
}

Core Concepts

Schema and Types

GraphQL has a strongly-typed schema. The schema defines what data exists and how it’s related:

type User {
  id: ID!
  name: String!
  email: String!
  avatarUrl: String
  posts: [Post!]!
}

type Post {
  id: ID!
  title: String!
  content: String!
  likeCount: Int!
  author: User!
}

type Query {
  user(id: ID!): User
  posts(limit: Int): [Post!]!
}

The ! means non-nullable. [Post!]! means a non-null list of non-null posts. The schema is the contract between frontend and backend.

Queries (Reading Data)

Queries are how we read data. The client specifies exactly which fields it wants:

# Just the name
query {
  user(id: 123) {
    name
  }
}

# Name + email + all posts with comments
query {
  user(id: 123) {
    name
    email
    posts {
      title
      comments {
        text
        author { name }
      }
    }
  }
}

Same endpoint, different shapes of data. The client controls it.

Mutations (Writing Data)

Mutations change data — create, update, delete:

mutation {
  createPost(input: {
    title: "New Post"
    content: "Hello world"
  }) {
    id
    title
    createdAt
  }
}

We can also ask for specific fields back in the response — so we don’t need a separate GET call after creating something.

Subscriptions (Real-Time Data)

Subscriptions let the client listen for real-time updates over WebSocket:

subscription {
  newMessage(chatId: "abc") {
    text
    sender { name }
    timestamp
  }
}

Whenever a new message is sent to that chat, the server pushes it to all subscribers.

REST vs GraphQL Comparison

When to Use GraphQL

GraphQL shines when:

• Our frontend needs data from multiple related resources in one call
• Different clients (web, mobile, watch) need different data shapes
• We’re tired of creating custom REST endpoints for each screen
• Rapid frontend iteration — frontend can change data requirements without backend changes

Stick with REST when:

• Our API is simple CRUD operations
• We need HTTP caching heavily (GraphQL makes this harder)
• We’re doing file uploads/downloads
• Our team is small and the overhead of a GraphQL schema isn’t worth it
• We’re building a public API (REST is more universally understood)

The Downsides

GraphQL isn’t perfect:

Complexity: We need a schema, resolvers, and a query engine. More moving parts than REST.

Caching is harder: REST uses URLs as cache keys. GraphQL uses a single endpoint with POST requests, so HTTP caching doesn’t work out of the box. We need client-side caching (Apollo Client, urql).

N+1 on the server: If a query asks for 100 posts with their authors, the resolver might make 100 separate database calls for authors. Solution: use DataLoader to batch and deduplicate.

Security: Clients can craft deeply nested queries that overload the server. We need query depth limiting and complexity analysis.

# A malicious query
query {
  user(id: 1) {
    posts {
      comments {
        author {
          posts {
            comments {
              author {
                # ...infinite nesting
              }
            }
          }
        }
      }
    }
  }
}

Key Takeaway

In simple language, GraphQL lets the client ask for exactly the data it needs in a single request — no over-fetching, no under-fetching. It’s great when we have complex, interconnected data and multiple client types. But it adds complexity compared to REST, especially around caching and security. For most simple APIs, REST is still the better choice. For data-rich applications with lots of relationships (social networks, dashboards, e-commerce), GraphQL really shines.

HTTP was built for a simple pattern: the client asks, the server answers. But what if the server needs to push data to the client without being asked? Think chat messages, live notifications, stock price tickers, or multiplayer games. The server can’t wait for the client to ask “any new messages?” every second. We need a way for both sides to send data whenever they want. That’s what WebSockets are for.

HTTP vs WebSocket

With HTTP, the client always initiates. The server can only respond — it can’t proactively send data. Every exchange opens a new connection, sends data, and closes.

With WebSocket, we open a connection once and both sides can send data anytime. The connection stays open for as long as needed — minutes, hours, even days.

The WebSocket Handshake

WebSocket connections start as a regular HTTP request, then “upgrade” to a WebSocket connection:

Client → Server:
GET /chat HTTP/1.1
Host: example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13

Server → Client:
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

After the 101 Switching Protocols response, the connection is upgraded. From here on, both sides communicate using WebSocket frames — lightweight binary packets, not HTTP.

Common Use Cases

• Chat applications — Slack, Discord, WhatsApp Web. Messages need to arrive instantly.
• Live notifications — Facebook notifications, Gmail new email alerts.
• Real-time dashboards — Stock tickers, monitoring dashboards, live sports scores.
• Collaborative editing — Google Docs, Figma. Multiple users editing the same document.
• Multiplayer gaming — Player positions, actions, game state updates.
• Live streaming — Chat alongside a video stream (Twitch chat).

The common thread: all of these need the server to push data to the client, and latency matters.

A Simple WebSocket Example

Client-side (JavaScript):

const ws = new WebSocket('wss://chat.example.com/ws');

ws.onopen = () => {
  console.log('Connected!');
  ws.send(JSON.stringify({ type: 'join', room: 'general' }));
};

ws.onmessage = (event) => {
  const message = JSON.parse(event.data);
  console.log('New message:', message);
};

ws.onclose = () => {
  console.log('Disconnected. Reconnecting...');
  // reconnect logic here
};

Server-side (Node.js):

const WebSocket = require('ws');
const wss = new WebSocket.Server({ port: 8080 });

wss.on('connection', (ws) => {
  ws.on('message', (data) => {
    const msg = JSON.parse(data);
    // broadcast to all connected clients
    wss.clients.forEach(client => {
      if (client.readyState === WebSocket.OPEN) {
        client.send(JSON.stringify(msg));
      }
    });
  });
});

Scaling WebSocket Connections

Here’s where it gets tricky. Each WebSocket connection is a persistent TCP connection held in memory. A single server might handle 10,000-100,000 connections. But what happens when we have multiple servers?

The Problem

User A connects to Server 1. User B connects to Server 2. User A sends a message to User B. But Server 1 doesn’t know about User B — that connection lives on Server 2.

Solution 1: Sticky Sessions

Route each user to the same server every time using a load balancer. Simple but limits our ability to distribute load evenly.

Solution 2: Pub/Sub Backbone

All servers subscribe to a shared message bus (Redis Pub/Sub, Kafka, NATS). When Server 1 receives a message, it publishes to the bus. Server 2 picks it up and delivers to User B.

User A → Server 1 → Redis Pub/Sub → Server 2 → User B
                   → Server 3 → User C
                   → Server 1 → User D

This is the standard pattern. Redis Pub/Sub is the most common choice for chat-scale applications.

Solution 3: Managed WebSocket Services

Skip the complexity and use a managed service:

• AWS API Gateway WebSocket — Serverless WebSocket support
• Pusher / Ably — Real-time messaging as a service
• Socket.IO — Library that handles reconnection, fallbacks, and rooms

Connection Management

WebSocket connections are long-lived, so we need to handle:

Heartbeats (ping/pong): The server sends periodic pings. If the client doesn’t respond with a pong, the connection is dead — close it and free resources.

Reconnection: Connections drop all the time (network changes, phone going to sleep). Clients must implement auto-reconnect with exponential backoff.

Authentication: We can’t send auth headers on every message like HTTP. Authenticate during the handshake or in the first message after connecting.

Connection limits: Each connection uses memory. Set a max per server and reject new connections gracefully when full.

When NOT to Use WebSockets

WebSockets aren’t always the answer:

• Occasional updates (new email every few minutes) → Server-Sent Events or long polling are simpler
• One-directional server push → SSE is lighter weight
• Standard CRUD operations → REST is simpler and cacheable
• Firewalls and proxies — Some corporate networks block WebSocket connections

Key Takeaway

In simple language, WebSockets give us a persistent, two-way connection between client and server. Instead of the client constantly asking “anything new?”, the server can push data whenever it wants. They’re essential for real-time features like chat, live notifications, and collaborative editing. The main challenge is scaling — each connection lives on a specific server, so we need a pub/sub layer (like Redis) to bridge messages across servers.

REST uses JSON over HTTP. It’s human-readable, widely supported, and great for public APIs. But when two microservices are talking to each other thousands of times per second inside our data center, we don’t need human-readability. We need speed. That’s where gRPC comes in.

What Is gRPC?

gRPC (gRPC Remote Procedure Calls) is an open-source framework created by Google. It lets services call methods on other services as if they were local function calls. Under the hood, it uses HTTP/2 for transport and Protocol Buffers (protobuf) for serialization.

In simple language, instead of crafting HTTP requests with URLs and JSON, we just call orderService.createOrder(orderData) and gRPC handles the networking.

Protocol Buffers (Protobuf)

Protobuf is a binary serialization format. We define our data structure in a .proto file, and protobuf generates code in any language.

// user.proto
syntax = "proto3";

package userservice;

message User {
  int32 id = 1;
  string name = 2;
  string email = 3;
  repeated string roles = 4;
}

message GetUserRequest {
  int32 id = 1;
}

message GetUserResponse {
  User user = 1;
}

service UserService {
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
  rpc CreateUser(User) returns (User);
  rpc ListUsers(ListUsersRequest) returns (stream User);
}

The numbers (= 1, = 2) are field tags — they identify fields in the binary format and should never be changed once deployed.

Why Binary?

JSON is text-based. A number like 12345 is 5 characters (5 bytes). In protobuf, it’s a variable-length integer — just 2 bytes. At millions of requests per second, those savings add up.

How gRPC Works

1. We define our service and messages in a .proto file
2. The protobuf compiler (protoc) generates client and server code in our language
3. The server implements the service methods
4. The client calls those methods like regular functions
5. gRPC handles serialization, HTTP/2 transport, and error handling

Client                           Server
  │                                │
  │  orderService.CreateOrder()    │
  │  ──── protobuf bytes ────→    │
  │        over HTTP/2             │
  │                                │  → executes CreateOrder()
  │  ←──── protobuf bytes ────    │
  │       (response)               │

We never manually construct HTTP requests or parse JSON. The generated code handles everything.

Four Communication Patterns

1. Unary RPC (Request-Response)

Like a regular function call. Client sends one request, server sends one response.

rpc GetUser(GetUserRequest) returns (GetUserResponse);

This is the most common pattern. Same as a REST API call but faster.

2. Server Streaming

Client sends one request, server sends back a stream of responses.

rpc ListUsers(ListUsersRequest) returns (stream User);

Great for: fetching large datasets incrementally, real-time event feeds, live logs.

3. Client Streaming

Client sends a stream of requests, server sends one response when the client is done.

rpc UploadChunks(stream FileChunk) returns (UploadResult);

Great for: file uploads, sending sensor data, batch operations.

4. Bidirectional Streaming

Both sides send streams of messages simultaneously. Like a WebSocket conversation but with typed messages.

rpc Chat(stream ChatMessage) returns (stream ChatMessage);

Great for: chat systems, collaborative editing, real-time gaming.

gRPC vs REST

When to Use gRPC

Use gRPC for:

• Microservice-to-microservice communication inside our infrastructure
• High-throughput services where performance matters (10,000+ RPS)
• Polyglot environments — proto files generate code for Go, Java, Python, C++, etc.
• Streaming use cases — server streaming, bidirectional streaming
• Mobile clients — smaller payloads = less bandwidth on cellular networks

Stick with REST for:

• Public APIs — REST is universally understood, gRPC needs protobuf knowledge
• Browser clients — browsers can’t make native gRPC calls (need grpc-web proxy)
• Simple CRUD — the overhead of proto files isn’t worth it for basic operations
• Debugging — we can’t just curl a gRPC endpoint and read the response

HTTP/2 Advantages

gRPC uses HTTP/2, which gives us:

• Multiplexing — Multiple RPC calls over a single TCP connection. No head-of-line blocking.
• Header compression — HTTP/2 compresses headers with HPACK, reducing overhead on repeated calls.
• Binary framing — More efficient than HTTP/1.1 text parsing.

In REST over HTTP/1.1, each request needs its own TCP connection (or waits in a queue). In gRPC over HTTP/2, we multiplex hundreds of concurrent calls on one connection.

Error Handling

gRPC uses its own status codes (similar to HTTP but different):

gRPC also has built-in deadlines. Every call can specify a timeout, and if the server doesn’t respond in time, the call fails with DEADLINE_EXCEEDED. This prevents hanging calls.

Key Takeaway

In simple language, gRPC is a faster alternative to REST that uses binary data (protobuf) instead of JSON and HTTP/2 instead of HTTP/1.1. It’s ideal for communication between microservices where speed and efficiency matter more than human-readability. The proto file serves as both the API contract and the source for auto-generated client/server code. Use gRPC internally between services, REST externally for public APIs. Many large systems (Google, Netflix, Square) use both — gRPC inside, REST outside.

Not every real-time feature needs WebSockets. Sometimes we just need the server to push updates to the client — new notifications, live scores, order status changes. Before reaching for WebSockets, we should consider simpler alternatives: polling, long polling, and Server-Sent Events. Each has its sweet spot.

Short Polling

The simplest approach. The client asks the server “any updates?” at regular intervals.

// Client polls every 5 seconds
setInterval(async () => {
  const response = await fetch('/api/notifications');
  const data = await response.json();
  if (data.length > 0) {
    showNotifications(data);
  }
}, 5000);

Think of it like checking our mailbox every 5 minutes. Most of the time, there’s nothing new. But we keep walking to the mailbox and back anyway.

Pros:

• Dead simple to implement
• Works everywhere (it’s just regular HTTP)
• Stateless — server doesn’t track connections

Cons:

• Wasteful — most requests return “nothing new”
• Delay between events — if something happens right after we poll, we don’t know for 5 seconds
• Scales poorly — 10,000 clients polling every 5 seconds = 2,000 requests per second for nothing

Long Polling

A smarter version of polling. The client sends a request, and the server holds it open until there’s new data. When data arrives, the server responds. The client immediately sends another request.

// Client long-polls
async function longPoll() {
  try {
    const response = await fetch('/api/notifications/subscribe');
    const data = await response.json();
    showNotifications(data);
  } catch (err) {
    // wait a bit before retrying on error
    await new Promise(r => setTimeout(r, 3000));
  }
  // immediately poll again
  longPoll();
}

longPoll();

Think of it like calling a restaurant and asking “is my table ready?” Instead of them saying “no” and hanging up (short polling), they say “hold on, I’ll let you know when it’s ready” and keep us on the line.

Pros:

• Near real-time — data arrives as soon as the server has it
• No wasted requests — each request results in data
• Works through firewalls and proxies (it’s still HTTP)

Cons:

• Server holds open connections — each waiting client ties up server resources
• Not truly bidirectional — client can’t push data while waiting
• Timeout handling — if the server holds too long, proxies/load balancers may kill the connection (typically 30-60 second timeout)

Server-Sent Events (SSE)

SSE is a standardized way for the server to push updates to the client over a single, long-lived HTTP connection. The client opens the connection once, and the server sends events whenever it wants.

// Client — SSE is built into browsers
const eventSource = new EventSource('/api/notifications/stream');

eventSource.onmessage = (event) => {
  const data = JSON.parse(event.data);
  showNotification(data);
};

eventSource.onerror = () => {
  console.log('Connection lost, browser will auto-reconnect');
};

Server side (Node.js):

app.get('/api/notifications/stream', (req, res) => {
  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    'Connection': 'keep-alive',
  });

  // Send a notification whenever we have one
  const sendEvent = (data) => {
    res.write(`data: ${JSON.stringify(data)}\n\n`);
  };

  // Subscribe to notification events
  notificationEmitter.on('notify', sendEvent);

  req.on('close', () => {
    notificationEmitter.off('notify', sendEvent);
  });
});

The server sends data in a specific format:

data: {"type": "notification", "message": "New follower!"}

data: {"type": "notification", "message": "Your order shipped!"}

event: heartbeat
data: ping

Pros:

• Built into browsers — EventSource API with automatic reconnection
• Simple — just HTTP with a special content type
• Efficient — single connection, server pushes when ready
• Auto-reconnect — the browser handles reconnection automatically

Cons:

• One-directional — server to client only. Client can’t send data over the same connection.
• Connection limits — browsers limit SSE connections per domain (6 in HTTP/1.1, more in HTTP/2)
• Text only — no binary data (but we can base64 encode if needed)

Comparison: All Four Approaches

When to Use Each

Short Polling — When updates are infrequent and slight delay is fine. Weather data, email checking. Dead simple to implement.

Long Polling — When we need near real-time but can’t use WebSocket or SSE. Good fallback. Used by older chat systems.

SSE — When we only need server-to-client push. Live notifications, news feeds, stock tickers, progress bars, live logs. This is the underrated option — it’s simpler than WebSocket and handles 90% of “real-time” use cases.

WebSocket — When we need full bidirectional communication. Chat, multiplayer games, collaborative editing. Worth the extra complexity only when both sides need to send data frequently.

A Real-World Decision Tree

Do we need real-time updates?
  ├─ No → Regular HTTP (REST)
  └─ Yes → Does the client need to send data too?
      ├─ No → SSE (simplest real-time option)
      └─ Yes → How often does the client send?
          ├─ Rarely → SSE + regular HTTP POST for client messages
          └─ Frequently → WebSocket

The often-overlooked option is SSE + REST. We use SSE for the server pushing updates and regular POST requests when the client needs to send something. This gives us 80% of WebSocket functionality with way less complexity.

Key Takeaway

In simple language, polling is the client repeatedly asking “anything new?” — simple but wasteful. Long polling improves this by having the server wait until there IS something new. SSE opens a one-way channel where the server pushes updates as they happen. WebSocket opens a two-way channel for full real-time communication. Start with the simplest option that meets our needs — often that’s SSE, not WebSocket.

Rate limiting is controlling how many requests a client can make to our system in a given time period. Think of it like a bouncer at a club — only so many people can get in per hour, no matter how eager they are.

Every production system needs it. Without rate limiting, one misbehaving client (or an attacker) can overwhelm our servers and ruin the experience for everyone else.

Why We Need It

• Prevent abuse — Stop DDoS attacks and brute-force login attempts
• Protect resources — Our servers, databases, and third-party APIs have finite capacity
• Fair usage — Make sure one noisy client doesn’t hog everything
• Cost control — If we’re paying per API call to a downstream service, we don’t want runaway costs
• Stability — Graceful degradation is better than a total crash

Rate Limiting Algorithms

There are five main algorithms. Each has tradeoffs.

1. Token Bucket

The most popular algorithm. Used by AWS, Stripe, and most API gateways.

Imagine a bucket that holds tokens. A refiller adds tokens at a fixed rate (say 10 per second). Each request takes one token. If the bucket is empty, the request is rejected. The bucket has a max capacity, so tokens don’t accumulate forever.

Why it’s popular: It allows bursts (up to the bucket size) while maintaining a steady average rate. Simple to implement and memory efficient.

2. Leaking Bucket

Similar to token bucket, but requests go INTO a bucket (a FIFO queue) and leak out at a fixed rate. If the bucket overflows, new requests are dropped.

The only difference from token bucket: leaking bucket enforces a perfectly smooth outflow rate. Token bucket allows bursts. Leaking bucket doesn’t.

Good for: When we need a perfectly steady processing rate (like sending API calls to a strict third-party rate limit).

3. Fixed Window Counter

Divide time into fixed windows (say 1-minute windows). Count requests in each window. If the count exceeds the limit, reject.

Window: 10:00-10:01 → 95 requests (limit: 100) ✓
Window: 10:01-10:02 → 45 requests (limit: 100) ✓

Problem: Burst at the boundary. If 90 requests come at 10:00:55 and 90 more at 10:01:05, that’s 180 requests in 10 seconds even though the limit is 100/minute. Both windows pass individually, but the actual rate is way over.

4. Sliding Window Log

Keep a timestamped log of every request. When a new request arrives, remove entries older than the window size, then count what’s left.

Fixes the boundary burst problem. But it’s memory-hungry — we’re storing a timestamp for every single request.

5. Sliding Window Counter

A hybrid of fixed window and sliding window log. We keep counts for the current and previous windows, then calculate a weighted count based on where we are in the current window.

Previous window: 84 requests
Current window (40% through): 36 requests
Weighted count: 84 × 0.6 + 36 = 86.4 → Under 100, allow!

Best of both worlds: Smooth like sliding window, memory-efficient like fixed window. This is what most real implementations use.

HTTP Headers

When we rate limit, we should tell clients what’s going on. Standard headers:

The HTTP status code for “you’ve been rate limited” is 429 Too Many Requests.

Where to Implement

API Gateway level — The most common place. Rate limit before requests even hit our services. Tools like Kong, NGINX, and AWS API Gateway have this built in.

Middleware level — Inside the application. More flexible — we can rate limit based on user tier, endpoint, or any custom logic.

Per-service level — Each microservice protects itself. Useful when different services have different capacity limits.

In practice, most systems do it at the API gateway with additional per-service limits for extra protection.

Client-Side vs Server-Side

Server-side rate limiting is what we’ve been discussing — the server decides when to reject requests. This is the important one. We never trust the client.

Client-side rate limiting is when the client self-throttles (e.g., backing off after receiving a 429). It’s a nice optimization — reduces wasted requests — but it’s not a security mechanism. A malicious client can just ignore it.

Good API clients implement exponential backoff: wait 1s, then 2s, then 4s, then 8s between retries. This prevents a stampede when the server recovers.

Rate Limiting in Distributed Systems

When we have multiple servers, we need a shared counter. Each server can’t just keep its own count — a client could hit different servers and bypass the limit.

The solution: use a centralized store like Redis. Redis is fast enough for this (single-digit millisecond lookups), and commands like INCR and EXPIRE make implementing rate limiters straightforward.

INCR user:123:rate_limit    → 1
EXPIRE user:123:rate_limit 60  → expires in 60s

In simple language, rate limiting is putting a speed limit on our API. Without it, one bad actor can crash the party for everyone. Token bucket is the go-to algorithm — it’s simple, allows bursts, and is easy to implement with Redis. Always return proper HTTP headers so good clients know where they stand.

We covered the basics of caching earlier. Now let’s go deeper into the different caching patterns and when to use each one. The difference between a well-cached system and a poorly-cached system isn’t whether we use a cache — it’s which pattern we pick.

The Five Caching Patterns

Cache-Aside (Lazy Loading)

The most common pattern. The application talks to the cache and the database directly. The cache doesn’t know the database exists.

Read path:

1. App checks the cache
2. Hit — return the cached value
3. Miss — app reads from DB, writes the value into cache, then returns it

Write path:

1. App writes to the database
2. App deletes (invalidates) the cache entry
3. Next read will re-populate the cache

Why it’s popular: Simple to implement. Cache only holds data that’s actually been requested. If the cache goes down, the app still works (just slower).

The catch: The first request for any key is always a cache miss. And there’s a small window between the DB write and cache invalidation where we serve stale data.

Read-Through

Looks similar to cache-aside from the app’s perspective, but the only difference is: the cache itself is responsible for loading data from the database on a miss. The app only ever talks to the cache.

Think of it like a library. We ask the librarian (cache) for a book. If they don’t have it, they go fetch it from the warehouse (DB) — we just wait.

Pros: Simpler application code. The data-loading logic lives in one place (the cache layer). Cons: Cache needs to know how to query the database. Cache failure means no data access at all.

Write-Through

Every write goes to the cache AND the database at the same time, synchronously. The cache acts as the primary interface for writes.

App writes "user:123" → Cache stores it → DB stores it → Done

Pros: Cache is always in sync with the DB. No stale data. Pairs perfectly with read-through — together they give us a fully consistent cache layer.

Cons: Higher write latency (two writes per operation). We end up caching data that might never be read, wasting memory.

Write-Behind (Write-Back)

Write to the cache immediately, then asynchronously flush to the database in the background. The app gets a fast response because it only waits for the cache write.

App writes "user:123" → Cache stores it → Returns immediately
                          ↓ (async, batched)
                          DB stores it later

Pros: Super fast writes. We can batch multiple writes into one DB operation, reducing database load.

Cons: If the cache crashes before flushing to DB, we lose data. This is a real risk. Use this only when speed matters more than durability.

Good for: Write-heavy workloads like analytics counters, view counts, or gaming leaderboards where losing a few seconds of data is acceptable.

Refresh-Ahead

The cache proactively refreshes entries that are about to expire. If an entry’s TTL is 60 seconds, the cache might refresh it at the 50-second mark — before anyone asks for it.

Pros: Hot data never experiences a cache miss. Users get consistently fast responses.

Cons: We’re refreshing data that might not be needed. If we predict wrong, we waste resources refreshing data nobody’s reading.

Good for: Data that’s read very frequently and is expensive to compute (dashboards, popular product pages, trending feeds).

The Cache Stampede Problem

Also called the thundering herd. Here’s the scenario:

1. A popular cache key expires
2. 1000 requests come in simultaneously for that key
3. All 1000 requests see a cache miss
4. All 1000 requests hit the database at the same time
5. The database collapses under the load

This is a real problem for hot keys with millions of reads.

Solutions

Locking (Mutex): When a cache miss happens, the first request acquires a lock, fetches from DB, and populates the cache. All other requests wait for the lock to release, then read from cache.

Early expiration (Staggered TTL): Add a random jitter to TTL values so not all keys expire at the same time. Instead of all keys expiring at exactly 60s, they expire between 55-65s.

Refresh-ahead: Proactively refresh before expiry, so the key never actually expires for readers.

Never expire + background refresh: Set keys to never expire. A background job periodically refreshes them. Readers always get a cache hit (possibly slightly stale).

Distributed Caching

When our app runs on multiple servers, a local in-memory cache (like a HashMap) on each server has problems — different servers have different data, and when a server restarts, its cache is gone.

The solution: a shared distributed cache that all servers read from and write to.

Redis Cluster

Redis is the go-to for distributed caching. Redis Cluster splits data across multiple Redis nodes using hash slots (16,384 slots total). Each node owns a range of slots. If one node fails, a replica takes over.

Memcached

Simpler than Redis. Pure key-value store. Uses consistent hashing to distribute keys across multiple servers. Slightly faster than Redis for simple get/set operations because it does less.

When to Use Which

In simple language, caching patterns are about who loads the data, who writes the data, and when. Cache-aside is the safe default — we manage everything ourselves. Read-through and write-through let the cache do more work. Write-behind is fast but risky. And always plan for the stampede — because when a hot key expires, the thundering herd is coming.

We’ve all used search. Type a few words, get results in milliseconds. But have we ever thought about how it works under the hood? Regular databases are terrible at this. Let’s see why, and what we use instead.

Why Databases Are Bad at Search

If we want to find all products containing the word “wireless” in their description, a SQL LIKE '%wireless%' query seems fine. But here’s the problem:

• It does a full table scan — checks every single row
• No index can help with a leading wildcard (%wireless%)
• Case sensitivity, typos, synonyms, relevance ranking — none of that works
• With millions of rows, this takes seconds. Users expect milliseconds.

Databases are built for structured queries (find user where id = 5). Search engines are built for unstructured text queries (find documents that best match “wireless bluetooth headphones”).

What Is an Inverted Index?

This is the core data structure behind every search engine. Think of it like the index at the back of a textbook — instead of reading every page to find “photosynthesis,” we look it up in the index and go straight to pages 47, 112, and 203.

An inverted index maps every word to the list of documents that contain it.

When someone searches “fast red,” we look up both words in the index, find the intersection of document lists, and return the results. No scanning required. This is why search engines are fast.

How Elasticsearch Works

Elasticsearch (ES) is the most popular search engine for backend systems. It’s built on Apache Lucene, which does the actual indexing and searching. ES wraps Lucene with a distributed system, REST API, and cluster management.

Key Concepts

Index — A collection of related documents. Think of it like a database table. An e-commerce site might have a products index and a reviews index.

Document — A single record in an index. It’s a JSON object. Each product is one document.

Shard — An index is split into shards for horizontal scaling. Each shard is a self-contained Lucene index. A products index with 5 shards distributes data across the cluster.

Replica — A copy of a shard on a different node. Replicas give us fault tolerance and allow read scaling (searches can hit replicas too).

How Data Gets Indexed

When we add a document to ES:

1. The text is passed through an analyzer
2. The analyzer tokenizes it — breaks text into individual terms (“Wireless Bluetooth Headphones” becomes [“wireless”, “bluetooth”, “headphones”])
3. Tokens are normalized — lowercased, stemmed (“running” becomes “run”), stop words removed (“the”, “is”, “a”)
4. The resulting terms go into the inverted index

This is why ES can find “running shoes” when we search for “run shoe” — the analyzer reduced both to the same root form.

Relevance Scoring

Not all results are equal. ES ranks them by relevance using a scoring algorithm (BM25 by default). The score depends on:

• Term Frequency (TF) — How often does the term appear in this document? More = more relevant.
• Inverse Document Frequency (IDF) — How rare is this term across all documents? Rare terms are more meaningful. “the” appears everywhere and tells us nothing. “Elasticsearch” is specific.
• Field length — A match in a short title is more significant than a match in a long description.

When to Use a Search Engine

Use a search engine when:

• We need full-text search across large amounts of text
• Users expect typo tolerance, synonyms, and relevance ranking
• We’re building product search, log analysis, or autocomplete
• We need aggregations on text data (faceted search, like filtering by category)

Stick with the database when:

• We’re doing exact lookups (get user by ID)
• The dataset is small (a few thousand rows)
• We only need simple prefix matching (PostgreSQL’s LIKE 'prefix%' uses indexes just fine)
• We don’t want to maintain another system

Common Use Cases

Product search — The classic. Users search “blue running shoes size 10” and we need to match across multiple fields (name, description, category, size) with relevance ranking. Elasticsearch shines here.

Log analysis — Centralize logs from hundreds of servers. The ELK stack (Elasticsearch, Logstash, Kibana) is the standard for log search and visualization.

Autocomplete — As the user types “wire…”, we suggest “wireless headphones”, “wireless charger”, etc. ES has dedicated completion and search_as_you_type field types for this.

Geosearch — Find restaurants within 5km. ES supports geo queries natively with geo_point fields and distance filters.

Keeping Search in Sync with the Database

The database is still the source of truth. ES is a secondary index. We need to keep them in sync.

Dual write — Write to both DB and ES. Simple but risky — if one write fails, they’re out of sync.

Change Data Capture (CDC) — Listen to database changes (like Debezium reading the Postgres WAL) and stream them to ES. More reliable. The database doesn’t even know ES exists.

Periodic sync — A background job re-indexes data from DB to ES every few minutes. Simple but introduces lag.

CDC is the best approach for production systems. It’s reliable, near real-time, and decoupled.

In simple language, search engines use inverted indexes — maps from words to documents — to find things fast. Regular databases scan through rows; search engines look up words directly. Elasticsearch gives us full-text search, relevance ranking, and typo tolerance out of the box. We use it alongside our database, not instead of it.

Most systems store the current state of things. A user’s balance is $500. An order status is “shipped.” But what if instead of storing the final answer, we stored every event that got us there? That’s event sourcing. And when we combine it with separate models for reading and writing, that’s CQRS.

Traditional CRUD vs Event Sourcing

In a traditional system, when a user deposits $100 into their account, we just update the balance:

UPDATE accounts SET balance = balance + 100 WHERE user_id = 123;

The old balance is gone. We overwrote it. If someone asks “what happened at 3pm yesterday?” — we can’t answer that from the database.

With event sourcing, we store the event instead:

{ event: "MoneyDeposited", userId: 123, amount: 100, timestamp: "..." }

The current balance isn’t stored. We calculate it by replaying all events. Deposit $500, withdraw $200, deposit $100 — replay them and we get $400.

Why Event Sourcing Is Powerful

Complete audit trail — Every change is recorded. We can answer “what was the balance at 3pm on Tuesday?” by replaying events up to that timestamp. For banking, healthcare, and compliance, this is huge.

Replay and rebuild — Found a bug in our billing logic? Fix it, replay all events, and recalculate. We can rebuild the entire state from scratch.

Temporal queries — “Show me the state of this order 2 days ago.” With CRUD, that’s impossible unless we built time travel. With event sourcing, we just replay events up to that point.

Event-driven architecture — Events become the backbone. Other services can subscribe to events and react. “OrderPlaced” can trigger inventory update, email notification, and analytics — all independently.

Debugging — When something goes wrong, we have a complete history of exactly what happened and in what order.

The Event Store

Events are stored in an event store — an append-only log. Events are never modified or deleted. This is important. The immutability is what makes the audit trail trustworthy.

Each event typically has:

• Event type — “OrderPlaced”, “ItemShipped”, “PaymentReceived”
• Aggregate ID — Which entity this event belongs to (order_id: 456)
• Payload — The event data (items, amounts, addresses)
• Timestamp — When it happened
• Version — Sequence number for ordering

Tools like EventStoreDB, Apache Kafka (as an event log), or even a regular database table with an append-only constraint can serve as an event store.

Projections (Materialised Views)

Replaying thousands of events for every read request would be way too slow. So we build projections — pre-computed read models that are kept up to date by processing events.

Think of it like this: the event store is the source of truth. Projections are pre-built answers to common questions.

Events:  OrderPlaced → ItemAdded → ItemAdded → PaymentReceived → OrderShipped

Projection 1 (Order Summary):  { orderId: 456, items: 2, status: "shipped", total: $89 }
Projection 2 (Revenue Report):  { date: "2025-03-15", revenue: $12,450 }

When a new event arrives, the relevant projections are updated. Reads are fast because they hit the projection directly — no replay needed.

What Is CQRS?

CQRS stands for Command Query Responsibility Segregation. In simple language, it means using a different model for reading data than we use for writing data.

In a typical app, the same database model handles both reads and writes. CQRS splits them apart:

• Command side (writes) — Receives commands like “PlaceOrder” or “UpdateProfile.” Validates business rules and produces events or state changes.
• Query side (reads) — Serves read requests using optimised read models (projections). Can use a completely different database or schema.

Why separate them? Because reads and writes have very different needs:

• Writes need strong consistency, validation, and business rules
• Reads need speed, and the shape of data we read is often different from how we store it

How Event Sourcing + CQRS Work Together

They’re a natural pair:

1. A command comes in (“PlaceOrder”)
2. The command handler validates it and produces events (“OrderPlaced”, “InventoryReserved”)
3. Events are saved to the event store (write side)
4. Event handlers update projections (read side)
5. Read requests query the projections

The write side doesn’t care about read performance. The read side doesn’t care about business rules. Each is optimised for its job.

Real-World Examples

Banking — Every deposit, withdrawal, and transfer is an event. The balance is a projection. Regulators love the complete audit trail.

Shopping cart — ItemAdded, ItemRemoved, QuantityChanged, CouponApplied. We can replay to see exactly how the cart evolved. If a bug miscalculated a discount, we fix the logic and replay.

Version control (Git) — Git is basically event sourcing. Commits are events. The current state of the code is a projection. We can go back to any point in time.

When to Use It

Good fit:

• Systems that need a complete audit trail (finance, healthcare, legal)
• Complex business domains with many state transitions
• Systems where replaying history has real value
• High-read, low-write scenarios where CQRS shines

Bad fit:

• Simple CRUD apps — event sourcing adds enormous complexity for little benefit
• Small projects or MVPs — we’re adding infrastructure overhead we don’t need yet
• Systems where eventual consistency between write and read models is unacceptable
• Teams unfamiliar with the pattern — the learning curve is steep

The Tradeoffs

Event sourcing and CQRS are powerful but come with real costs:

• Complexity — Significantly more moving parts than simple CRUD
• Eventual consistency — Read models lag behind writes. We have to be OK with that
• Event schema evolution — Events are immutable, but our schema will change over time. We need a versioning strategy
• Storage growth — We never delete events, so storage grows forever. Snapshots help (save the current state periodically so we don’t replay from the beginning)

In simple language, event sourcing says “don’t store the answer, store the math.” Instead of saving “balance is $400,” we save every deposit and withdrawal. CQRS says “use different models for reading and writing because they have different needs.” Together, they give us an auditable, replayable, flexible system — but only if the complexity is worth it.

Here’s a problem that sounds simple but is incredibly hard: how do a bunch of servers agree on something? Which node is the leader? Is this transaction committed? What’s the current configuration? When networks are unreliable and servers can crash at any moment, getting everyone on the same page is one of the hardest problems in computer science.

That’s what distributed consensus solves.

The Problem

Imagine we have 5 database replicas. A write comes in. We need all (or most) of them to agree that the write happened. Sounds easy — just tell all 5, right?

But what if:

• Node 3 is temporarily unreachable (network partition)
• Node 5 crashed and restarted with stale data
• Two nodes both think they’re the leader
• Messages arrive out of order

Without a consensus protocol, we’d end up with nodes disagreeing about the state of the world. One node thinks the value is “A,” another thinks it’s “B.” That’s data corruption.

Leader Election

Most consensus systems work by electing a leader. One node is in charge. It receives all writes, decides the order of operations, and tells followers what to do. If the leader dies, a new one is elected.

Why not just let any node accept writes? Because coordinating writes across multiple nodes simultaneously is much harder than funneling everything through one node that makes decisions.

The tricky part: electing a leader that everyone agrees on, especially when the network is flaky.

Raft Consensus Algorithm

Raft was designed to be understandable (unlike Paxos, the OG consensus algorithm that’s famously confusing). Most modern systems use Raft or something based on it.

The Three States

Every node in a Raft cluster is in one of three states:

How Election Works (Simplified)

1. All nodes start as followers
2. Followers expect regular heartbeats from the leader
3. If a follower doesn’t hear from the leader for a while (election timeout), it assumes the leader is dead
4. It becomes a candidate and starts a new term (election round)
5. It votes for itself and asks other nodes for their vote
6. If it gets votes from a majority (quorum), it becomes the new leader
7. The new leader starts sending heartbeats, and everyone falls in line

The election timeout is randomised (e.g., 150-300ms) so that not all followers become candidates at the same time. This avoids vote splitting.

How Writes Work

Once we have a leader:

1. Client sends a write to the leader
2. Leader appends it to its log
3. Leader sends the entry to all followers
4. Followers append it and acknowledge
5. Once a majority acknowledges, the entry is committed
6. Leader applies it and responds to the client

The key: writes are committed only when a majority of nodes have the data. Even if some nodes are down, as long as the majority is alive, the system works.

The Split-Brain Problem

This is the nightmare scenario. Two nodes both think they’re the leader. Both accept writes. Now we have conflicting data and no way to merge it.

How does this happen? Network partitions. If the network splits a 5-node cluster into a group of 3 and a group of 2, both groups might try to elect a leader.

Quorum Prevents Split-Brain

The solution is the quorum — a majority requirement. To do anything (elect a leader, commit a write), we need agreement from more than half the nodes.

In a 5-node cluster, the quorum is 3. If the network splits into groups of 3 and 2:

• The group of 3 can elect a leader (3 >= quorum)
• The group of 2 cannot elect a leader (2 < quorum)

There can only ever be one group with a majority. Math prevents split-brain.

This is why consensus clusters use odd numbers of nodes (3, 5, 7). With 4 nodes and a 2-2 split, neither side has a majority and the system is stuck.

Fault Tolerance

A Raft cluster of N nodes can tolerate (N-1)/2 failures:

Going from 3 to 5 nodes gives us tolerance for one more failure. Going from 5 to 7 gives another. But more nodes means more coordination overhead. Most production systems use 3 or 5 nodes.

Where Consensus Is Used

Distributed databases — CockroachDB, TiDB, and YugabyteDB use Raft to replicate data across nodes. Every write is committed through consensus.

Configuration management — etcd (used by Kubernetes) and ZooKeeper store cluster configuration. When config changes, all nodes must agree on the new value.

Leader election for other services — Services that need exactly one active instance (like a scheduler) use consensus to pick the leader. If the leader dies, a new one is elected.

Distributed locks — Only one service can hold a lock at a time. Consensus ensures everyone agrees on who holds it.

Tools That Implement Consensus

etcd

A distributed key-value store that uses Raft. It’s the backbone of Kubernetes — all cluster state (pods, services, configs) lives in etcd. Written in Go. Simple API. Very reliable.

ZooKeeper

The older, battle-tested option. Used by Kafka (older versions), Hadoop, and HBase. Uses a protocol called ZAB (ZooKeeper Atomic Broadcast), which is similar to Raft. Written in Java. More complex to operate than etcd.

Consul

By HashiCorp. Uses Raft. Focuses on service discovery and health checking in addition to key-value storage. Often used in microservice architectures.

Consensus vs Eventual Consistency

Consensus gives us strong consistency — all nodes agree before we respond. The tradeoff is latency. Every write requires a network round trip to the majority.

Eventual consistency (like DynamoDB or Cassandra) is the opposite — writes are fast because we don’t wait for agreement. Nodes will eventually converge. Faster, but we might read stale data.

Most systems mix both. Use consensus for the things that absolutely must be correct (leader election, config, financial transactions). Use eventual consistency for things where speed matters more (feeds, analytics, caches).

In simple language, distributed consensus is how servers vote on the truth. Raft elects a leader, the leader coordinates writes, and everything needs a majority to proceed. The quorum rule guarantees we can never have two conflicting leaders. It’s slower than just writing to one server, but it’s the price we pay for systems that survive failures without losing data.

This is probably the most classic system design interview question. We’re designing a URL shortening service like TinyURL or bit.ly — take a long URL, give back a short one, and when someone visits the short URL, redirect them to the original. Simple concept, but the details get interesting fast.

Let’s walk through it step by step, the way we’d do it in an actual interview.

Step 1: Requirements

Functional Requirements

• Given a long URL, generate a unique short URL
• When a user visits the short URL, redirect them to the original long URL
• Users can optionally set a custom short link
• Links expire after a default period (configurable)
• Analytics — track how many times a short URL was clicked

Non-Functional Requirements

• High availability — the redirect service can’t go down, or millions of links break
• Low latency — redirects should feel instant (< 50ms)
• Read-heavy — way more people click short links than create them (100:1 read/write ratio)
• Short URLs should be as short as possible
• URLs should not be predictable (so people can’t guess other URLs)

Step 2: Estimation

Let’s put some numbers on this.

Assumptions:

• 500M new URLs created per month
• 100:1 read to write ratio → 50B redirects per month

QPS:

Write QPS = 500M / (30 × 86,400) ≈ ~200 URLs/sec
Read QPS  = 200 × 100 = ~20,000 redirects/sec
Peak QPS  = ~40,000 redirects/sec (2x average)

Storage (5 years):

Each URL record ≈ 500 bytes (short URL + long URL + metadata)
500M × 12 months × 5 years = 30 Billion URLs
30B × 500 bytes = 15 TB

Cache:

Following the 80/20 rule — 20% of URLs generate 80% of traffic.
Daily read requests = 50B / 30 ≈ 1.7B/day
Cache 20% of daily data = 1.7B × 0.2 × 500 bytes ≈ 170 GB

170 GB fits comfortably in a few Redis instances. Nice.

Step 3: High-Level Design

How the redirect works:

1. User visits short.url/abc123
2. Load balancer routes to an app server
3. App server checks Redis cache for abc123
4. Cache hit → return the long URL. Cache miss → query DB, put in cache, return.
5. Server responds with a 301 (permanent redirect) or 302 (temporary redirect)

301 vs 302 — which do we pick?

• 301 (Moved Permanently) — The browser caches the redirect. Next time, it goes directly to the long URL without hitting our server. Better for the user, but we lose analytics visibility.
• 302 (Found/Temporary) — The browser always comes back to our server first. We can track every click. More load on our server, but we keep full analytics.

If analytics matter (and for a URL shortener, they do), we go with 302.

Step 4: API Design

POST /api/v1/shorten
Body: { "long_url": "https://example.com/very/long/path", "custom_alias": "my-link", "expires_at": "2026-12-31" }
Response: { "short_url": "https://short.url/abc123", "expires_at": "2026-12-31" }

GET /{short_url_key}
Response: HTTP 302 Redirect → Location: https://example.com/very/long/path

GET /api/v1/stats/{short_url_key}
Response: { "total_clicks": 15420, "created_at": "2025-01-15", "long_url": "..." }

Step 5: Data Model

-- Main URL table
CREATE TABLE urls (
    id          BIGINT PRIMARY KEY AUTO_INCREMENT,
    short_key   VARCHAR(7) UNIQUE NOT NULL,   -- the "abc123" part
    long_url    TEXT NOT NULL,
    user_id     BIGINT,                        -- nullable for anonymous users
    created_at  TIMESTAMP DEFAULT NOW(),
    expires_at  TIMESTAMP,
    INDEX idx_short_key (short_key)            -- fast lookups by short key
);

-- Analytics table (append-only, write-heavy)
CREATE TABLE click_events (
    id          BIGINT PRIMARY KEY AUTO_INCREMENT,
    short_key   VARCHAR(7) NOT NULL,
    clicked_at  TIMESTAMP DEFAULT NOW(),
    ip_address  VARCHAR(45),
    user_agent  TEXT,
    referrer    TEXT,
    country     VARCHAR(2),
    INDEX idx_short_key_time (short_key, clicked_at)
);

We keep the URL table lean for fast reads. The analytics table is append-only — we’re only ever inserting into it, never updating. This is a perfect candidate for a time-series approach or a message queue that writes asynchronously.

Step 6: Deep Dives

Deep Dive 1: Short URL Generation Strategies

This is the heart of the problem. How do we turn a long URL into a short, unique key? We have three main approaches.

Approach A: Hash + Collision Check

Take the long URL, hash it (MD5 or SHA-256), and take the first 7 characters.

MD5("https://example.com/long/path") = "a1b2c3d4e5f6..."
Short key = "a1b2c3d" (first 7 chars)

Problem: collisions. Two different URLs could produce the same first 7 characters. So we check the DB — if the key exists, we append a counter and rehash. This works but the collision checks add latency and complexity.

Approach B: Auto-Increment ID + Base62 Encoding

Use a database auto-increment ID and convert it to base62 (a-z, A-Z, 0-9 = 62 characters).

ID = 123456789
Base62 = "8M0kX"   (123456789 in base 62)

With 7 characters, base62 gives us 62^7 = 3.5 trillion possible URLs. That’s plenty. The problem? URLs are predictable. If someone gets abc123, they know abc124 probably exists too. Also, the auto-increment ID becomes a single point of failure in a distributed system.

Approach C: Pre-Generated Key Service (KGS)

A separate service pre-generates millions of unique keys and stores them in a database. When an app server needs a key, it grabs one from the pool.

KGS Database:
┌─────────────┬──────────┐
│ key         │ used     │
├─────────────┼──────────┤
│ "a7Bx2q"   │ false    │
│ "k9Mw3r"   │ false    │
│ "p2Lz8n"   │ true     │  ← already assigned
└─────────────┴──────────┘

Each app server fetches a batch of keys (say 1000) and keeps them in memory. No collision checking needed. No coordination between servers. This is the most scalable approach and the one most interviewers love.

The winner: Pre-Generated Key Service. It’s clean, fast, and eliminates the collision problem entirely.

Deep Dive 2: Caching Hot URLs

Our system is extremely read-heavy (100:1). Most traffic goes to a small percentage of popular URLs. This screams “cache me.”

We put Redis between the app servers and the database. The strategy:

1. On a redirect request, check Redis first
2. Cache hit → return immediately (sub-millisecond)
3. Cache miss → query DB → store in Redis with a TTL → return
4. Use LRU eviction — when cache is full, kick out the least recently used URL

With our 170 GB estimate, we can use a Redis cluster of 3-4 nodes with replication. The cache hit rate should be 90%+ since URL access follows a power law — a few URLs get the vast majority of clicks.

Cache invalidation: When a URL expires or gets deleted, we remove it from the cache. Simple because URLs are immutable — we never update a short URL to point to a different long URL.

Deep Dive 3: Analytics and Click Tracking

Every redirect is a potential analytics event. But we can’t let analytics slow down the redirect. The redirect must be fast — analytics can be eventual.

The approach: async processing with a message queue.

User clicks → App Server sends 302 redirect immediately
           → App Server pushes click event to Kafka/SQS
           → Analytics workers consume from queue
           → Workers batch-insert into click_events table

This way, the user gets their redirect in milliseconds. The analytics data flows through a queue and gets processed in the background. If the analytics system falls behind, clicks queue up but the redirect service stays fast.

For the dashboard, we can pre-aggregate hourly/daily counts in a summary table instead of running expensive COUNT queries on billions of rows.

Step 7: Scaling

Database scaling:

• The URL table is read-heavy → add read replicas
• As data grows past a single DB → shard by short_key hash
• The click_events table grows fast → partition by time (monthly partitions) and archive old data

App server scaling:

• Stateless servers behind a load balancer → horizontally scale by adding more instances
• Each server holds a batch of pre-generated keys in memory → no coordination needed between servers

Cache scaling:

• Start with a single Redis instance, then move to Redis Cluster
• Consistent hashing to distribute keys across cache nodes

Global distribution:

• Deploy app servers in multiple regions
• Use GeoDNS to route users to the nearest region
• Replicate the database across regions (or use a globally distributed DB like CockroachDB)

Handling 40K QPS at peak:

• Load balancer distributes across app servers
• Redis handles ~90% of reads (36K QPS in cache = easy for Redis)
• Only ~4K QPS actually hits the database
• Each DB read is a simple key lookup by indexed short_key — blazing fast

In simple language, a URL shortener is a giant key-value store with a smart key generation strategy. The short key is the hard part — we use a pre-generated key service to avoid collisions. The redirect is the hot path — we cache it aggressively. And analytics go through a queue so they never slow down the user. That’s the whole system.

A rate limiter controls how many requests a client can make in a given time window. Think of it like a bouncer at a club — only so many people get in per hour. If we exceed the limit, the request gets rejected with a 429 Too Many Requests.

Every major API has one. Without it, a single misbehaving client can take down our entire service.

Step 1: Requirements

Functional Requirements

• Limit the number of requests a client can make within a time window
• Support different rate limiting rules (e.g., 100 req/min for free tier, 1000 req/min for paid)
• Return clear error responses when rate limited (429 status + retry-after header)
• Support multiple limiting strategies (by user ID, IP address, API key)

Non-Functional Requirements

• Low latency — the rate limiter sits in the request path, so it must be fast (< 1ms overhead)
• Distributed — must work across multiple servers (not just in-memory on one box)
• Highly available — if the rate limiter goes down, we’d rather let all requests through than block everything
• Accurate — in a distributed setup, we shouldn’t let significantly more requests through than the limit

Step 2: Estimation

Assumptions:

• We’re protecting an API that handles 10,000 QPS across all clients
• 1 million unique clients (user IDs or API keys)
• Each rate limit check should add < 1ms of latency

Storage:

Each client needs a counter + timestamp ≈ 20 bytes
1M clients × 20 bytes = 20 MB

That’s nothing. The entire state fits easily in Redis.

QPS on the rate limiter:

Every incoming request triggers a rate limit check.
10,000 QPS → 10,000 Redis operations/sec

A single Redis instance handles 100K+ operations/sec. We’re well within limits. For higher scale, we shard by client ID.

Step 3: High-Level Design

Where does the rate limiter live?

We have a few options:

• API Gateway — Cloud providers (AWS API Gateway, Kong) have built-in rate limiting. Easiest to set up.
• Middleware — A thin layer in our application code, before the request hits the business logic.
• Sidecar — A separate process alongside our app (like Envoy proxy).

For most systems, putting it at the API Gateway level is the right call. It’s centralized, handles it before the request even reaches our servers, and most gateways support it out of the box.

Step 4: API Design

The rate limiter doesn’t have its own API per se — it’s middleware. But the response headers tell clients about their rate limit status:

# On every response (allowed or rejected):
X-RateLimit-Limit: 100        # max requests allowed in the window
X-RateLimit-Remaining: 42     # requests left in current window
X-RateLimit-Reset: 1735689600 # Unix timestamp when the window resets

# On rejection:
HTTP/1.1 429 Too Many Requests
Retry-After: 30               # seconds until client should retry
Content-Type: application/json
{ "error": "Rate limit exceeded. Try again in 30 seconds." }

For configuring rate limit rules, we’d have an internal admin API:

POST /admin/rate-rules
Body: {
  "client_type": "free_tier",
  "endpoint": "/api/v1/search",
  "max_requests": 100,
  "window_seconds": 60
}

Step 5: Data Model

Rate limiters usually don’t use a traditional database. Redis is the go-to because it’s in-memory and supports atomic operations.

# Redis key pattern for counters
rate_limit:{client_id}:{endpoint}:{window}

# Example: user 42, search endpoint, per-minute window
rate_limit:user_42:/api/search:202603301430

# Value: current request count (integer)
# TTL: set to expire when the window ends

For the rules configuration:

# Rate limit rules (stored in DB or config file)
rules:
  - client_type: "free_tier"
    limits:
      - endpoint: "/api/v1/*"
        max_requests: 100
        window: 60        # seconds

  - client_type: "paid_tier"
    limits:
      - endpoint: "/api/v1/*"
        max_requests: 1000
        window: 60

  - client_type: "default"
    limits:
      - endpoint: "*"
        max_requests: 50
        window: 60

Step 6: Deep Dives

Deep Dive 1: Rate Limiting Algorithms

There are four main algorithms. Each has different tradeoffs.

1. Token Bucket

Think of a bucket that holds tokens. It starts full. Each request takes one token. The bucket refills at a steady rate. If the bucket is empty, the request is rejected.

Bucket: max 10 tokens, refill 1 token/second

Time 0:  [10 tokens] → Request → [9 tokens] ✓
Time 0:  [9 tokens]  → Request → [8 tokens] ✓
...
Time 0:  [1 token]   → Request → [0 tokens] ✓
Time 0:  [0 tokens]  → Request → REJECTED    ✗
Time 1:  [1 token]   → Request → [0 tokens] ✓  (refilled)

Pros: Allows short bursts (up to bucket size). Smooth over time. Amazon and Stripe use this. Cons: Two parameters to tune (bucket size + refill rate).

2. Sliding Window Log

We store the timestamp of every request. To check the limit, we count how many timestamps fall within the last N seconds.

Window: 3 requests per 60 seconds

Timestamps: [1:00:15, 1:00:30, 1:00:45]
New request at 1:00:50 → 3 requests in window → REJECTED
New request at 1:01:20 → remove 1:00:15 (expired) → 2 in window → ALLOWED

Pros: Very accurate — no boundary issues. Cons: Stores every timestamp, uses more memory. Not great for high limits.

3. Sliding Window Counter

A clever hybrid. We keep counters for the current and previous windows, then calculate a weighted count based on where we are in the current window.

Previous window (1:00-1:01): 84 requests
Current window  (1:01-1:02): 36 requests
We're 25% into the current window.

Weighted count = 84 × 0.75 + 36 = 99
Limit = 100 → ALLOWED (barely!)

Pros: Memory-efficient (just two counters). Smooth enough for most use cases. Cons: Not perfectly accurate — it’s an approximation. But it’s close enough.

The winner for most systems: Token Bucket or Sliding Window Counter. Token Bucket if we want to allow bursts. Sliding Window Counter if we want simplicity.

Deep Dive 2: Distributed Rate Limiting with Redis

Here’s the tricky part. Our API has multiple servers behind a load balancer. If we just count requests in memory on each server, a client can send 100 requests to Server A and 100 to Server B, effectively getting 200 when the limit is 100.

We need a centralized counter, and Redis is perfect for this.

The basic approach with the Fixed Window Counter algorithm:

# Pseudocode for each incoming request:

key = f"rate:{client_id}:{current_minute}"
count = INCR(key)              # atomic increment in Redis
if count == 1:
    EXPIRE(key, 60)            # set TTL on first request

if count > limit:
    return 429                 # rejected
else:
    forward request            # allowed

But there’s a subtle race condition. The INCR and EXPIRE are two separate commands. If our server crashes between them, the key never expires and the client is blocked forever.

The fix: use a Lua script to make it atomic.

-- Atomic rate limit check in Redis (Lua script)
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])

local current = redis.call('INCR', key)
if current == 1 then
    redis.call('EXPIRE', key, window)
end

if current > limit then
    return 0  -- rejected
else
    return 1  -- allowed
end