Gyaan

Health Checks & Graceful Shutdown

intermediate health-checks graceful-shutdown kubernetes sigterm reliability

A health check is a tiny endpoint our service exposes so an orchestrator can ask “are you okay?” without guessing. In simple language — it’s the service answering a doctor’s “how are you feeling?” so the platform knows whether to keep sending it traffic or restart it.

The why: in a world of containers, something else (Kubernetes, a load balancer) is constantly deciding where to route requests and when to reboot a sick instance. It can only make good decisions if each instance honestly reports its state. There are two different questions it needs answered, which is why we have two probes.

Liveness vs readiness

They sound similar and get confused constantly. Here’s the clean split:

  • Liveness = “are you alive, or should I kill and restart you?” A failed liveness probe means the process is wedged — deadlocked, stuck — and a restart is the fix.
  • Readiness = “are you ready to take traffic right now?” A failed readiness probe means “I’m alive but busy/warming up — don’t send me requests yet”, and the platform just stops routing to us. No restart.

Why both? Because “broken forever, restart me” and “temporarily not ready, just hold traffic” need completely different responses. Restarting a service that’s merely warming up its cache would be a disaster loop. Removing a deadlocked service from rotation but never restarting it would leave it dead forever.

Think of it like a shop: liveness is “is anyone in the building?” (if not, unlock and send staff in), readiness is “is the shop actually open for customers?” (the lights are on but they’re still restocking shelves).

# Kubernetes — same container, two very different checks
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }   # deadlocked? -> restart me
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }     # DB not connected yet? -> hold traffic
  periodSeconds: 5

A good /readyz actually checks dependencies — “can I reach the DB?” — while /healthz stays dumb and cheap so a slow DB doesn’t get our whole process killed.

How the platform uses them

The load balancer or orchestrator polls these on a schedule. On readiness fail, our instance is pulled out of the pool of endpoints receiving traffic — silently, no user sees an error. On liveness fail, our instance gets killed and a fresh one is started. This is the machinery that makes rolling deploys and self-healing work.

Graceful shutdown

Now the flip side. When we deploy a new version, the platform has to stop the old instances. If we just yank the plug, every request that was mid-flight gets dropped — users see 502s. Graceful shutdown is the polite exit that avoids that.

The trigger is SIGTERM — the platform sends this signal and gives us a grace period (Kubernetes default 30s) before it sends the un-ignorable SIGKILL. Our job is to use that window well.

Graceful shutdown sequence (on SIGTERM)
1fail readiness — flip /readyz to unhealthy so no NEW traffic arrives
↓
2stop accepting — close the listener, refuse new connections
↓
3drain in-flight — let requests already running finish (up to a deadline)
↓
4close resources — DB pool, message queue, open files
↓
5exit(0) — clean exit before the grace period runs out

The order matters. We fail readiness first so the platform stops routing to us, then close the listener, then drain. If we closed the listener before flipping readiness, requests could still be in transit and get refused.

// Node/Express — a minimal graceful shutdown handler
let ready = true;
app.get("/readyz", (_req, res) => res.status(ready ? 200 : 503).end());

process.on("SIGTERM", async () => {
  ready = false;                         // 1. stop new traffic
  server.close(async () => {             // 2+3. stop accepting, drain existing
    await db.end();                      // 4. close the DB pool
    process.exit(0);                     // 5. clean exit
  });
  // safety net: force-exit if draining hangs past the grace period
  setTimeout(() => process.exit(1), 25_000).unref();
});

That setTimeout safety net matters — if one request hangs forever, we still exit before SIGKILL yanks us harder.

Practical takeaway

Expose two probes: a dumb /healthz for liveness (restart me if wedged) and a dependency-aware /readyz for readiness (hold traffic if I’m not ready). On SIGTERM, fail readiness first, stop accepting new work, drain what’s in flight, close the DB pool, then exit — all inside the grace period, with a timeout as backstop. Do this and rolling deploys drop zero requests.

References