Case 01 · Software

Their on-call team was buried in alerts. Now most of them never reach a person.

A software company with a large engineering team, running across a couple of cloud providers.

Outcome · hours back every week
  • On-call
  • Alert sorting
  • Audit trail
  • Always on
Scroll

The issue

Where it hurt.

Their on-call rotation was getting paged round the clock, and most of it was noise — the same alert firing twice, a brief blip in one region, two scheduled jobs stepping on each other. None of it was an emergency, but every page still pulled someone out of what they were doing. The engineering lead told us this rotation had been the team's biggest source of frustration for months. They'd already tried a couple of off-the-shelf tools and a contractor before us, and none of it really moved things. By the time we talked, they were ready to rebuild it properly, starting from where the alerts actually come from.

Discovery

What five days of looking found.

  1. 01

    Three months of their alert history was the starting point — grouped by what fired, where it came from, when it happened, and how it actually got resolved.

  2. 02

    Most of the pages turned out to be the same handful of harmless patterns: an alert that cleared itself in a few minutes with nothing behind it.

  3. 03

    Then the on-call playbooks, line by line with the team lead. A few hadn't been touched in years, and those were the ones throwing the loudest, most useless alerts.

How we worked

Side by side, not over tickets.

  1. 01

    We sat with their on-call lead a few mornings a week while we tuned the system, working through it live rather than over tickets.

  2. 02

    A shared chat channel let anyone flag a bad call the moment it happened — each one became an example the system learned from the next day.

  3. 03

    Every couple of weeks there was a short review with the people actually on the rotation; they rewrote the playbooks the system now points to.

The build

What runs now.

  1. 01

    When an alert fires, it goes to 3node first. If it matches one of the harmless patterns, it gets closed on its own with a full record kept. If it's real, the person on call gets the page along with the background they need — the recent related incidents and the code changes that touched that part of the system in the last day.

  2. 02

    Anything the system isn't sure about goes into a short review lane. The on-call engineer confirms or corrects it, and that decision feeds back so it gets better over time.

  3. 03

    Every auto-closed alert leaves a permanent record, which happens to be exactly the paper trail their auditors ask for — so nobody scrambles to assemble it later.

Outcome

What it's worth.

hours back every week
  • The team's own survey on how the rotation feels went from rough to genuinely good.
  • Nobody has left the on-call team since — which, for them, was new.
Saving
Several on-call slots' worth of engineering time given back every week. Over a year that adds up to a meaningful six-figure saving — but mostly it's people not getting woken up for nothing.
Speed
Real incidents now get picked up in under two minutes instead of a quarter of an hour, and the noise pages dropped by roughly two-thirds.
Reliability
No serious incident has slipped through in over a year of running. A couple of close calls were actually caught earlier than the old rotation would have.

Where it is now

Still running, and we're still on it. Every quarter we retune as their systems change. Currently being scoped: flagging risky code changes before they ship, based on which parts of the system tend to cause trouble.

Your next step

Your version of this story.

Thirty-minute call. We'll find three flows worth automating before we hang up. No deck. No "discovery proposal." Just the work.