An SRE guide to AI-assisted incident triage

What AI correlation surfaces

When the SLO burn alarm fires at 03:00 UTC, the on-call is waking up fifteen seconds into the incident but reading about one hundred thousand signals from the last hour. The shift the desk makes for them is not "summarise the alert" — it is correlation. The desk reads the last hour of commit diffs, the matching error spike window, and the trace topology in one pass, and asks: which changed call path did the failing traffic first see, and who last deployed it. Per-file blame, last-deployed-by, and SLO burn rate against the affected path get surfaced onto a single page, so the on-call is responding to a hypothesis, not chasing the data to form one. Time-to-root-cause drops from "the on-call reads for ten minutes and then escalates" to "the on-call reviews a hypothesis the desk already proposed."

The correlation is not magic. It is pattern-matching across three streams the desk already has: structured logs with stable trace IDs, distributed traces with a parent span that consistently names the entry path, and a deploy index that ties commit SHAs to time windows. The desk reads the same data the on-call would read, just without the sleep inertia.

Signals that matter

Logs earn correlation weight when they have three things at once: an error class that is specific enough to group by, a frequency above the rolling baseline, and a first-seen timestamp inside the deploy window that the desk is investigating. Logs that have only one or two of those three end up in the noisy tail and the desk drops them on purpose. The on-call's first instinct, when they wake up, is to look at every error — the desk's first move is to rank errors by specificity × frequency × first-seen, and to surface only the top three.

Traces earn weight when the parent span's sat time has moved in a way that explains the symptom and when the downstream fan-out has actually widened. A trace where every child span looks healthy is, by definition, not the failing path — leave it alone. A trace where the parent sat for two seconds and the children all exited fast is a downstream stall, and the desk highlights the boundary. The signal is not "is this span slow" but "is this span slow in a way that explains the customer-visible symptom."

Deployment diffs earn weight when a call path changed, a config flag flipped, or a library bumped inside the same commit window. Cosmetic diffs — comment changes, test-only refactors, doc updates — do not enter the correlation even though they sit in the same commit. The diff filter is the difference between "what changed at all" and "what changed in a way that could produce this incident."

Where human judgement wins

The first place judgement wins is confirming a hypothesis the desk surfaced. The desk's correlation is high-precision and high-recall against the surface signals, but it is not customer-aware. The on-call knows that the tenant in this region is mid-migration, that the partner on the failing endpoint has been flaky for three days, that the on-call engineer who shipped the diff is on vacation — none of which is in any signal the desk can read. The on-call signs off on the hypothesis or rewrites it before the page goes out. The desk is the radar; the on-call is the operator.

The second place is declining a confident-but-wrong correlation. Correlation is high-confidence when the match is tight: same file, same deploy window, same failing path. Correlation is low-confidence when it is the only viable-looking failure mode in a noisy incident: the next-stream diff beats it on specificity, but the desk still surfaces it as a candidate. The on-call's job is to recognise the second case and route around it rather than act on it. Following a wrong-but-confident suggestion is worse than following no suggestion at all.

The third place is choosing a mitigation over a rollback. A rollback is the safe option and the one the desk reaches for first. But sometimes the right answer is a partial mitigation that keeps tenancies intact while the on-call investigates — a feature-flag flip that holds the new path open for a single tenant, a rate-limit that absorbs the burst, a slow-roll cancel that lets the deploy finish safely. The on-call owns the call because the on-call owns the customer context: this merchant has a customer-facing demo in forty minutes, that one is in the middle of a billing cycle, the third is mid-migration and a rollback would lose them their day. None of this is in the signals; all of it is in the on-call.

What to instrument next

The correlation the desk does today is bounded by what shipped. Three things, instrumented consistently, make next quarter's incidents cheaper than this quarter's. Consistent trace IDs across services, so the desk can stitch logs and traces without guessing. Structured logs with a fixed error-class field, so the desk can rank errors by specificity rather than regex. A deploy → trace linkage, so the desk can ask "which deploy did this trace first see" without joining on commit SHAs by hand. Each one is small; together they shorten time-to-root-cause by enough minutes that the on-call wakes up reading instead of reading to wake up. The rest of the rotation — the hypothesis confirmation, the degradation call, the mitigation-over-rollback judgement — keeps staying human.