First postmortem from the on-call desk

Timeline

SLO breach fired at INC-2419 on checkout-api at 03:12 UTC, with error_budget_burn at 4.2x against payment.charge. The desk read the last hour of commits, matched the failing path to feat/checkout-retry-with-jitter (commit a91f3c, deployed 38 minutes prior), scored the incident SEV-2, routed the page to #sre-checkout, and opened PR #1841 with the rollback pre-wired behind the merge button — all within fifteen seconds of detect. The on-call woke, reviewed the diff, and merged at 03:14 UTC. Customer-facing error rate dropped below SLO by 03:16 UTC. The incident closed at 03:21 UTC.

Root cause

The new retry path doubled the connection pool pressure on payment.charge during a deploy window that already carried elevated traffic from a partner migration. The original PR introduced per-attempt jitter but did not bound the burst rate when retries stacked against a slow upstream; the second-order effect was a thundering herd that exhausted the connection pool and stalled the downstream charge call. The desk identified the failing path against the diff in eleven seconds, and the proposed fix capped the per-key retry fan-out to the upstream's available headroom.

Customer impact

Approximately 4.1% of charge attempts in the affected four-minute window returned a 502 to the merchant. No successful charge was double-billed, and no PII was exposed — retries are idempotent on the upstream side and the desk's evidence pack ties every dropped request back to its merchant and reference. The most affected vertical was EU retail, where the partner migration was concentrated. Postmortem draft was generated in the incident channel at 03:21 UTC and committed to the repo at 03:24 UTC.

Resolution

PR #1841 added a configurable retry fan-out cap (default 4) keyed off the upstream's reported active connections, surfaced as a service-level knob to calibration later. The PR also introduced a unit test that replays the production retry trace against the new cap and fails if the fan-out would have exceeded the upstream's headroom. The fix was reviewed by the checkout rotation, merged in under three minutes, and rolled out behind the existing gradual-deploy flag with the rollback pre-wired for one-click revert. The deploy completed at 03:18 UTC, two minutes after first merge.

Rollback

The rollback path was staged behind a feature flag rather than a full revert, so reverting required flipping one toggle and rolling the deploy forward. The flag was armed for seven days as a precaution, then permanently enabled once the deploy passed the seven-day soak; no customer-visible incidents have hit the retry path since. The same flag pattern is now in the desk's default rollout template, so every subsequent fix on checkout-api ships with the same one-click revert pre-wired — and the rotation reviews a change instead of writing one under duress.