Outage analysis
Your best on‑call engineer's playbook, run the moment the page fires
TriageAI works an outage the way a senior SRE does. It scopes the impact, finds what changed, reads the signals and narrows the blast radius, all at once, then tells you how to mitigate. When the cause is in code, it goes on to the fix. You make the calls.
Outage, checkout failing
- 14:02 Deploy payments-api v3.8.0Argo CD
- 14:04 checkout-api p99 latency > 2sPrometheus
- 14:05 payments-db connections at 100%Datadog
- 14:06 checkout 5xx rate 14%, on-call pagedPagerDuty
Probable root cause
payments-api v3.8.0 raised its database pool from 10 to 40 connections per pod. Across 12 pods that passes the database limit of 300. Connections climbed within 2 minutes of the deploy while traffic stayed flat.
Blast radius
checkout-api, cart-service and orders-worker. 14% of checkout requests failing.
Suggested remediation
Roll back payments-api to v3.7.2. Needs approval.
The first half hour of an outage is mostly looking things up
The same questions, every time
Is it real? Who's affected? What changed? The first half hour of most outages goes on answering them by hand.
The evidence is in many tools
Alerts in one place, graphs in another, logs and deploy history somewhere else. You switch tabs while customers wait.
Experience lives in a few heads
Senior engineers know where to look first. Everyone else learns it the hard way, at 3 a.m.
Nine steps senior SREs follow. TriageAI runs them for you.
Every step pairs the questions an experienced on-call engineer asks with what TriageAI does to answer them. Pick a step to see what it found in the example outage.
Step 1 of 9
Triage the page
A senior SRE asks
- Is it real, and is it hitting users right now?
- Is it getting worse, steady or recovering?
- Is there already an open incident or a runbook?
TriageAI does
Checks the alert against error rates and synthetic checks, then finds related alerts, open incidents and the runbook linked to the alert.
You decide
Set the severity and decide whether to escalate.
In the example checkout outage
Real: 14% of checkout requests failing and rising. 3 related alerts. No open incident.
Want the full playbook, with every question? Read how a senior SRE investigates an outage
“What changed?” answered first
Most outages follow a change, so it's the first question a senior SRE asks. TriageAI pulls every deploy, config change, flag flip and infrastructure event in the window, and ranks them by how well they fit the start time and the services that are failing.
A match to the minute is strong evidence. A change that doesn't explain who's affected is ruled out, with the reason shown.
- 1Strong match
Deploy payments-api v3.8.0
14:02, 2 min before onset
Timing lines up, and every failing path goes through payments-api.
- 2Weak
Flag new-checkout-ui ramped to 50%
13:10, 54 min before onset
Too early, and failures hit users with the flag off too.
- 3Ruled out
Node pool upgrade, pool-b
11:40, 2 h 24 min before onset
Failures span every node pool, not just pool-b.
What it reads
Alerts and paging
Prometheus Alertmanager, Datadog, CloudWatch, PagerDuty or Opsgenie. The page that woke you up is where the analysis starts.
Metrics
Latency, traffic, errors and saturation for every service on the request path, before and during the outage.
Logs and traces
OpenSearch, Loki or OpenTelemetry, filtered to the affected services and the outage window.
Changes and deploys
Deploys, config changes, feature flags and Kubernetes events, lined up against the moment things broke.
What you get back
Timeline and blast radius
When it started, how it spread, and which services and users it hit.
Probable root cause
The change or failure that explains every symptom, with the evidence attached.
Remediation steps
The safest mitigation that fits the evidence, its risks, and the signal to watch.
Status updates and postmortem
Plain-language update drafts during the incident, and a postmortem draft after it.
You stay in command
TriageAI has read-only access and never changes production. It does the looking; you do the deciding.
| TriageAI does | You decide |
|---|---|
| Checks whether the alert reflects real user impact | Set the severity and escalate |
| Ranks every recent change by how well it fits | Add the changes no system recorded |
| Reads the signals and finds where the failure starts | Confirm it with the service owners |
| Suggests a mitigation with its risks | Approve it and run it |
| Drafts status updates and the postmortem | Edit, send and own the follow-ups |
Common questions
Does it replace the on-call engineer?
No. It does the evidence gathering an experienced engineer would do first, in parallel and without switching tools. The decisions stay with you: severity, escalation, mitigation and the final call on the cause.
Does TriageAI roll back or restart anything?
No. Remediation steps are suggestions. TriageAI has read-only access to your observability and deploy tools, and an engineer approves and runs every change.
What if the cause isn't clear?
It says so. You get the timeline and the strongest candidates with their evidence for and against, plus what it would check next. A senior engineer would rather say "we don't know yet" than guess, and so does TriageAI.
How is this different from bug triage?
Outage analysis starts from a live incident across many services and ends in a mitigation and a postmortem. When the cause is in code, it hands off to bug triage, which goes to the file and line and returns a fix and a regression test.
Run the playbook on your next outage
Try outage analysis on a sample incident, or ask for early access and connect your own alerts, metrics, logs and deploys.