Written by Nimisha GJ
on 

10 Things Every Service Should Do Before Production

The incidents that hurt the most are almost never about a missing big feature. They are about a missing small one — a timeout never set, a retry without backoff, a health check that lied.

This is the checklist I run through before any service goes to production. For each item: what it is, and why it is needed.

1. Graceful Shutdown

On SIGTERM: stop accepting new requests, finish what is in flight, deregister from the load balancer, then exit.

Why it is needed: every deploy sends SIGTERM to a pod that is mid-request. Without a handler, those requests die as 502s — quietly, on every single deploy.

SIGTERM load balancer deregistered ✓ service exit 0 draining in-flight… ✕ new requests refused responses complete ✓
SIGTERM arrives → stop accepting new work → finish what is in flight → deregister → exit 0. Nobody gets a 502.

2. Health Checks (Liveness, Readiness, Startup)

Three probes, three jobs. Liveness: “are you alive” — fail and you get restarted. Readiness: “should I get traffic” — fail and traffic stops, but you keep running. Startup: “done booting” — until it passes, the other two wait.

Why it is needed: wiring a dependency check to liveness turns a database hiccup into a cascading restart of every pod. Readiness sheds traffic; liveness kills. Mixing them up is the most common Kubernetes misconfiguration there is.

liveness “are you alive?” ✕ fails → pod restarted readiness “send traffic?” ✕ fails → traffic stops, pod stays up startup “done booting?” ✓ only now do the other probes fire
Three probes, three different consequences. Wire “can I reach the database” to readiness, never to liveness.

3. Timeouts at Every Layer

A timeout on every network call, database query, and HTTP client — always lower than the timeout of whatever is calling you.

Why it is needed: the default in most libraries is “no timeout”. One downstream slows to 30s, your workers pile up waiting on it, and soon requests that have nothing to do with it are stuck too. Your service is down without anything having crashed. This is the single highest-leverage item on the list.

no timeout timeout: 2s service — worker pool downstream replying in 30s… every worker stuck waiting — unrelated requests queue too calls give up at 2s — workers free, the pool keeps breathing ✓
One slow downstream. Without a timeout it silently owns your whole worker pool; with one, it only owns two seconds of it.

4. Retries With Backoff (And Limits)

Four properties: exponential backoff between attempts, jitter so clients do not retry in sync, a cap on total attempts (three is usually enough), and ideally a circuit breaker around it all.

Why it is needed: naïve retries make outages worse. A small hiccup turns into a sustained flood — every client retries, fails, retries again — and now the dependency cannot recover even after the original problem is gone.

no backoff sustained flood backoff + jitter stop — cap hit, breaker takes over t=0 time →
Same failure, two clients. The top one is why the dependency cannot recover; the bottom one waits 1s, 2s, 4s — a little jittered — then gives up.

5. Idempotency on Write Endpoints

An idempotency key: the caller sends a unique ID per logical action, the server stores it, and a duplicate ID gets the original result back instead of the work being done twice.

Why it is needed: the moment callers retry — and by the previous section, they will — a lost response means the same valid-looking request arrives twice. Without the key, that is a user charged twice.

client timeout — retry got result ✓ server seen keys ab12 charges: 0 charges: 1 2? POST /pay key=ab12 200 ✓ ✕ response lost retry key=ab12 key already seen → replay stored result, do not charge again 200 ✓ same result
The response gets lost, the client retries, and the user is still charged exactly once. That is the whole point of the key.

6. Context Propagation

Generate a request ID at the edge, include it in every log line, pass it in headers to every downstream call.

Why it is needed: without it, a slow request is a mystery — “service A took 800ms” in one file, “service B took 600ms” in another, and no way to know they are the same request. One propagated ID makes the problem obvious.

edge checkout payments ledger 7f3a checkout req= 7f3a 120ms payments req= 7f3a 644ms ← slow ledger req= 7f3a 38ms one ID, one grep, and the 800ms mystery is solved
Generate the ID at the edge, log it everywhere, pass it in headers. Three log files become one story.

7. Structured Logging

JSON or key-value logs with consistent field names across services. One line of config in most logging libraries. No DEBUG in production, no PII or secrets.

Why it is needed: it turns “grep for an hour across six services” into “filter by user_id in thirty seconds” — and converting later means redoing every grep, dashboard, and alert built on the unstructured logs.

plain text grep 12345 *.log …forty minutes in, still grepping structured user_id=12345 {"user_id":12345,"svc":"pay"} {"user_id":98801,"svc":"pay"} {"user_id":12345,"svc":"cart"} {"user_id":55102,"svc":"auth"} {"user_id":12345,"svc":"auth"} {"user_id":77216,"svc":"cart"} 3 hits, thirty seconds ✓
The same six log lines, twice. The only difference is whether “find everything for this user” is a filter or an archaeology project.

The cost of doing it on day one is nothing. The cost of converting later is enormous, because every grep, every dashboard, every alert built on the unstructured logs has to be redone.

8. Rate Limiting and Backpressure

A per-client rate limit on every public endpoint, plus backpressure: notice when the queue grows faster than it drains, and shed work early instead of falling over.

Why it is needed: one client with a buggy retry loop can hit you ten thousand times a second and take the service down for everyone. And “what to do when overloaded” is much easier to design before you are overloaded.

bursty inflow steady drain queue limit 429 shedding load before the queue explodes
The drain rate never changes — that is the point. When inflow outruns it, saying 429 early beats falling over later.

9. Circuit Breakers on Dependencies

Count recent failures. Past a threshold, “open” the circuit and stop calling for a while. Then let one trial request through — success closes the circuit, failure keeps it open.

Why it is needed: failing fast is almost always better than timing out — your service stays responsive while a dependency is down, and the dependency gets room to recover instead of being hammered.

CLOSED OPEN HALF-OPEN service dependency failure threshold crossed → open the circuit failing fast — no thread waits, dependency gets a break one trial request succeeds → close the circuit
Closed → open → half-open → closed. The switch in the middle is the entire idea: when things are bad, stop touching them.

10. Safe Database Migrations

Additive first: add the column, then write to it, then read from it, then drop the old one. No long locks. Always a rollback plan. Every step safe with the previous version of the code still running.

Why it is needed: migrations are how most “the deploy broke prod” incidents actually happen — the code is fine, but the migration locked a table for two minutes or removed a column the old version still reads.

1 · add new_col 2 · write to both 3 · read from new 4 · drop old_col app code orders table old_col dropped — nothing reads it new_col write write read read ✓ every phase is safe with the previous version of the code still running
Expand, migrate, contract. At no point does the running old version lose a column it depends on.

What NOT to Build Too Early

The other half of production-readiness is not over-engineering. Almost certainly not needed yet: dynamic config reload, unsampled tracing on every span, custom retry frameworks, feature flags on every line, multi-region active-active before your first incident.

Why it matters: each of these costs complexity today for a failure you cannot see yet. Build for the failures you can already see.

Closing

None of this is glamorous. But the gap between services that run quietly for years and services that page somebody every other week is almost entirely in this list.

Build for boring. Boring is what survives.