Observability is how well we can understand what’s happening inside our system just by looking at what it emits from the outside. In simple language — when production breaks at 2am, observability is the difference between “I can see exactly why” and “let me add some logs and redeploy and pray”.
The why is easy. A backend with a dozen services has too many moving parts to reason about in our head. We need the system to tell us what it’s doing. It does that through three kinds of signals, and interviewers love asking us to name all three.
The three pillars
Each pillar answers a different question. That’s the whole point — we need all three, not one.
Logs — the events
A log is a timestamped record of something that happened. “User 42 logged in”, “payment failed with code 402”. They’re the detail layer.
The one upgrade that matters: structured logging. Instead of a plain sentence, we emit a JSON object with fields. Why? Because we can then search and filter by field instead of grepping strings.
// Bad — a human sentence, painful to query
console.log(`payment failed for user ${userId}, amount ${amount}`);
// Good — structured, machine-queryable
logger.error({ event: "payment_failed", userId, amount, code: 402 });
// now we can filter: event=payment_failed AND code=402Add a correlation id (more on that below) to every log line and suddenly we can pull every log for a single request across every service.
Metrics — the numbers
A metric is a number measured over time — request count, error rate, memory usage, queue depth. They’re cheap to store (just numbers + timestamps) so we keep them forever and graph them. This is what dashboards and alerts run on.
Two famous shortcuts for which metrics to track:
- RED — for request-driven services: Rate (requests/sec), Errors (failures/sec), Duration (latency). Think of it as the health of a thing that serves traffic.
- USE — for resources: Utilization, Saturation, Errors. The health of a thing that has capacity — CPU, disk, a connection pool.
Rule of thumb: RED for our API endpoints, USE for the machines and pools underneath them.
Traces — the request’s journey
A trace follows one single request as it hops across services. It’s made of spans — each span is one unit of work (an HTTP call, a DB query) with a start time, a duration, and a parent. Stitch the spans together and we get a waterfall showing exactly where the 800ms went.
The glue is a trace id (also called a correlation id). The first service generates it, then passes it along in a header to every downstream call. Every log and span tags itself with that id. That’s how “one request across ten services” becomes a single searchable story.
Think of it like a package tracking number — one id, and we can see every warehouse the parcel passed through.
Monitoring vs observability
People use these interchangeably but interviewers like the distinction.
- Monitoring = watching known problems. We decide in advance what to measure (“alert if error rate > 1%”) and watch those dashboards. Great for the failures we’ve already imagined.
- Observability = being able to ask new questions without shipping new code. When something novel breaks, rich logs/metrics/traces let us slice the data in ways we didn’t plan for.
In simple language — monitoring answers “is the thing I worried about happening?”, observability answers “why is this weird thing I’ve never seen happening?”.
A word on alerting
Metrics feed alerts, but a good alert has one property: it’s actionable and tied to user pain. Alerting on “CPU is 90%” wakes us up for nothing if users are fine. Alerting on symptoms — high error rate, high latency, SLO burn — wakes us up only when something actually hurts. The SRE mantra: page on symptoms, not causes.
Interview soundbite
Observability rests on three pillars: logs (discrete structured events — what happened), metrics (numbers over time — track RED for services, USE for resources), and traces (one request’s path across services, made of spans stitched by a shared trace/correlation id). Monitoring watches problems we already predicted; observability lets us investigate ones we didn’t. And we alert on user-facing symptoms, not raw resource numbers.