Skip to content
TriageAI

Outage analysis

Your best on‑call engineer's playbook, run the moment the page fires

TriageAI works an outage the way a senior SRE does. It scopes the impact, finds what changed, reads the signals and narrows the blast radius, all at once, then tells you how to mitigate. When the cause is in code, it goes on to the fix. You make the calls.

Outage, checkout failing

  1. 14:02 Deploy payments-api v3.8.0Argo CD
  2. 14:04 checkout-api p99 latency > 2sPrometheus
  3. 14:05 payments-db connections at 100%Datadog
  4. 14:06 checkout 5xx rate 14%, on-call pagedPagerDuty

Probable root cause

payments-api v3.8.0 raised its database pool from 10 to 40 connections per pod. Across 12 pods that passes the database limit of 300. Connections climbed within 2 minutes of the deploy while traffic stayed flat.

Blast radius

checkout-api, cart-service and orders-worker. 14% of checkout requests failing.

Suggested remediation

Roll back payments-api to v3.7.2. Needs approval.

Example analysis. A postmortem draft is attached when the outage resolves.

The first half hour of an outage is mostly looking things up

  • The same questions, every time

    Is it real? Who's affected? What changed? The first half hour of most outages goes on answering them by hand.

  • The evidence is in many tools

    Alerts in one place, graphs in another, logs and deploy history somewhere else. You switch tabs while customers wait.

  • Experience lives in a few heads

    Senior engineers know where to look first. Everyone else learns it the hard way, at 3 a.m.

Nine steps senior SREs follow. TriageAI runs them for you.

Every step pairs the questions an experienced on-call engineer asks with what TriageAI does to answer them. Pick a step to see what it found in the example outage.

Step 1 of 9

Triage the page

A senior SRE asks

  • Is it real, and is it hitting users right now?
  • Is it getting worse, steady or recovering?
  • Is there already an open incident or a runbook?

TriageAI does

Checks the alert against error rates and synthetic checks, then finds related alerts, open incidents and the runbook linked to the alert.

You decide

Set the severity and decide whether to escalate.

In the example checkout outage

Real: 14% of checkout requests failing and rising. 3 related alerts. No open incident.

Want the full playbook, with every question? Read how a senior SRE investigates an outage

“What changed?” answered first

Most outages follow a change, so it's the first question a senior SRE asks. TriageAI pulls every deploy, config change, flag flip and infrastructure event in the window, and ranks them by how well they fit the start time and the services that are failing.

A match to the minute is strong evidence. A change that doesn't explain who's affected is ruled out, with the reason shown.

  1. 1

    Deploy payments-api v3.8.0

    14:02, 2 min before onset

    Timing lines up, and every failing path goes through payments-api.

    Strong match
  2. 2

    Flag new-checkout-ui ramped to 50%

    13:10, 54 min before onset

    Too early, and failures hit users with the flag off too.

    Weak
  3. 3

    Node pool upgrade, pool-b

    11:40, 2 h 24 min before onset

    Failures span every node pool, not just pool-b.

    Ruled out
Example: changes in the 3 hours before the checkout outage.

What it reads

  • Alerts and paging

    Prometheus Alertmanager, Datadog, CloudWatch, PagerDuty or Opsgenie. The page that woke you up is where the analysis starts.

  • Metrics

    Latency, traffic, errors and saturation for every service on the request path, before and during the outage.

  • Logs and traces

    OpenSearch, Loki or OpenTelemetry, filtered to the affected services and the outage window.

  • Changes and deploys

    Deploys, config changes, feature flags and Kubernetes events, lined up against the moment things broke.

What you get back

  • Timeline and blast radius

    When it started, how it spread, and which services and users it hit.

  • Probable root cause

    The change or failure that explains every symptom, with the evidence attached.

  • Remediation steps

    The safest mitigation that fits the evidence, its risks, and the signal to watch.

  • Status updates and postmortem

    Plain-language update drafts during the incident, and a postmortem draft after it.

You stay in command

TriageAI has read-only access and never changes production. It does the looking; you do the deciding.

What TriageAI does and what the on-call engineer decides
TriageAI doesYou decide
Checks whether the alert reflects real user impactSet the severity and escalate
Ranks every recent change by how well it fitsAdd the changes no system recorded
Reads the signals and finds where the failure startsConfirm it with the service owners
Suggests a mitigation with its risksApprove it and run it
Drafts status updates and the postmortemEdit, send and own the follow-ups

Common questions

Does it replace the on-call engineer?

No. It does the evidence gathering an experienced engineer would do first, in parallel and without switching tools. The decisions stay with you: severity, escalation, mitigation and the final call on the cause.

Does TriageAI roll back or restart anything?

No. Remediation steps are suggestions. TriageAI has read-only access to your observability and deploy tools, and an engineer approves and runs every change.

What if the cause isn't clear?

It says so. You get the timeline and the strongest candidates with their evidence for and against, plus what it would check next. A senior engineer would rather say "we don't know yet" than guess, and so does TriageAI.

How is this different from bug triage?

Outage analysis starts from a live incident across many services and ends in a mitigation and a postmortem. When the cause is in code, it hands off to bug triage, which goes to the file and line and returns a fix and a regression test.

Run the playbook on your next outage

Try outage analysis on a sample incident, or ask for early access and connect your own alerts, metrics, logs and deploys.