14:03
⏸ paused · click to resume
⚠ payment-service p95 > SLO · sev2
payment-service · p95 latency
~14:00
trace · one slow request
auth 120ms
fraud-check 80ms
pricing 2.4s ←
db 40ms
diagnosed ✓ 6 min
CI deploys
feature flags
config
migrations
"did anyone deploy pricing around 14:00?"
…cross-referencing timestamps by hand
#incident-payment
who owns the api-gateway?
try Ravi?
Ravi left the company last year
…anyone?
11 minutes to find a volunteer
ROLLBACK.md
last updated: 2 quarters ago
a migration is in flight.
is rolling back even safe?
20 minutes reading migration files
to confirm what design should have guaranteed
planning to roll back, ok?
team lead
in a meeting
waiting
.
.
.
12 minutes lost to a question the on-call already knew the answer to
who's running this?
I'm also debugging the gateway
…me too??
is anyone updating status page?
I don't have prod access
who can grant it? …the lead. in the meeting
none of this shows on a dashboard, but all of it shows in the MTTR
$ rollback pricing --to-release 13:50
✓ gateway config restored
latency back under SLO
understanding:
6 min
· resolution:
91 min
that Tuesday
with the machinery
diagnose · 6m
what changed? · 14m
find an owner · 11m
rollback = wiki page · 25m
permission · 12m
coordination chaos · 29m
total: 97 minutes
deploy annotation on the graph: one glance
catalog: api-gateway → platform team ✓
ROLL BACK
one rehearsed command
"the on-call may roll back without asking"
total: ~15 minutes
Observability buys you the first six minutes.
The rest you have to build.
click to replay from the top