Skip to content
TriageAI
All posts

How a Senior SRE Investigates an Outage, Step by Step

The pager goes off. Checkout is failing, the alert says error rates are climbing, and you have a few minutes before the questions start coming. What you do next decides whether this is a 15-minute blip or a two-hour outage.

This is the playbook experienced on-call engineers follow, written as the questions to ask at each step. It works whatever your stack is, and it's the same playbook TriageAI's outage analysis runs.

Three rules hold the whole way through:

  • Mitigate before you diagnose. Customers care that it works again, not why it broke. If a rollback is safe and likely to help, do it before you fully understand the cause.
  • Evidence over hunches. Every claim ("it's the database") gets checked against a graph, a log line or a change record before anyone acts on it.
  • Write it down as you go. A timestamped incident log costs seconds during the outage and saves hours in the postmortem.

1. The first five minutes

The goal is to decide how big this is and get the right people moving, not to find the cause.

  • Is it real, and is it affecting users right now? Do error rates, synthetic checks and support tickets agree?
  • Is it getting worse, steady or recovering on its own?
  • Are other alerts firing at the same time? Is our monitoring itself healthy?
  • Is there already an open incident, a maintenance window or a runbook linked to this alert?
  • What severity is it, and does it need an incident commander?

Acknowledge the page, open an incident channel, start the log, and escalate early if it's serious or spans teams. Escalating late is the most common mistake.

2. Establish the facts

A vague symptom leads to a vague investigation. "The site is slow" points nowhere. "Checkout POST returns 503 for 14% of requests since 14:04" points at the cause.

  • What exactly fails, and what does the user see: errors, slowness, wrong data?
  • When was the first bad data point? Not the alert time; alerts lag.
  • Did it start as a step change or a ramp? A step points to a change or a failure. A ramp points to growth, a leak or exhaustion.
  • Who is affected, and who isn't? One region, tenant, platform or client version?
  • Is any data lost or corrupted? That changes every decision after this.

3. Ask what changed

Most outages follow a change, so this is the most valuable question in the whole investigation. Line every change up against the start time.

  • What deployed, changed config or flipped a feature flag just before it started?
  • Did infrastructure change: node upgrades, autoscaling, DNS, network?
  • Did traffic change: a marketing push, a bot, a retry storm, a big customer batch job?
  • Did a dependency degrade: a cloud service, a third-party API, a shared internal service?
  • Did a certificate, token or credential expire or rotate?
  • Is it a scheduled job or a time boundary: a cron, month-end, a daylight-saving change?

Ask it out loud in the incident channel too: people often know about changes no system recorded. A change that lines up to the minute is strong evidence. Ten minutes apart is a coincidence until proven otherwise.

4. Read the signals

Read from the user inward. Metrics tell you where and when, traces tell you which hop, logs tell you why.

  • Golden signals for every service on the request path: did latency, traffic, errors or saturation move first?
  • Resources: what's running out? CPU, memory, connection pools, thread pools, queue depth, disk, database connections.
  • Traces: in a few failing requests, which span takes the time or returns the error?
  • Logs: compare the outage window with an hour earlier. Find the first new error, not the loudest one.
  • Platform events: crash loops, failed probes, evictions, zone or instance problems.

5. Narrow the blast radius

Split the failing traffic along every dimension you can until the bad part separates from the good part.

  • Is it every region, zone, host, version, endpoint, tenant and client, or only some?
  • Failures only on the new version or the canary? That's almost always the answer.
  • Which component is unhealthy without a sick dependency of its own? That's where the problem starts.
  • Which service went bad first? Is this service failing, or passing on a failure it received?

6. Mitigate first

As soon as an action is likely to help and unlikely to make things worse, take it. Keep investigating in parallel.

MitigationUse it when
Roll back the deployThe outage started at or just after a rollout
Turn off a feature flagA flag change lines up with the start
Revert a config changeA config or secret change lines up with the start
Shift traffic awayOne region, zone or cluster is bad
Scale up or outSomething is saturated, and the bottleneck isn't downstream
Restart or replaceA few bad pods or hosts, after capturing evidence
Shed or limit loadA traffic surge, a retry storm or one abusive client

Before you act: what do you expect to happen, how will you know within minutes, and can you undo it? Afterwards, check the same user-facing signal that told you it was broken. A quiet alert alone doesn't mean it's fixed.

7. Confirm the root cause

Treat every explanation as a hypothesis until the evidence closes it.

  • Does it explain every symptom, including the timing and who was not affected?
  • What would you expect to see if it were false? Do you see that instead?
  • Keep asking why until you reach something you can change. "The database ran out of connections" is a symptom. "A config change raised the pool size per pod, and nothing checked the total against the database limit" is a cause.

Patterns that come up again and again: a bad deploy or config, resource exhaustion, retry storms, cascading failures, expired certificates, DNS, a dependency outage, a cold cache, bad data, and time-based triggers.

8. Communicate

Silence during an outage is read as "nobody is on it". Post an update every 15 to 30 minutes for a serious incident, even when nothing has changed. Each one answers five questions: what's affected, what we know, what we're doing, whether there's a workaround, and when the next update is.

9. Resolve and review

An incident is resolved when users are no longer affected and the system is stable, not when the alert stops.

  • Have user-facing signals been at baseline for a sustained period?
  • Is the mitigation permanent or temporary? Who owns the follow-up?
  • Is there a backlog or data to repair?
  • Are the graphs and logs saved before retention expires?

Then hold a blameless postmortem within a few days. Turn each finding into an action with an owner and a date, and ask how you could have detected it sooner and mitigated it faster.

Where an AI agent helps

Most of this playbook is gathering evidence and lining it up: checking signals, pulling change history, segmenting failures, walking dependencies. An agent can do that in parallel the moment the alert fires. The decisions should stay with people.

That's how TriageAI's outage analysis is designed. It runs steps 1 to 5 and the evidence-gathering parts of 7 to 9, then suggests a mitigation with its risks. Severity, escalation, the mitigation itself and the final call on the cause stay with the on-call engineer. TriageAI has read-only access and never changes production.

See the playbook, step by step, in outage analysis.

Watch TriageAI work a sample outage

Follow it from the alert to the change behind it and the exact line of code, then get the fix and a regression test. You review every step.