A software company with a large engineering team, running across a couple of cloud providers.
Their on-call team was buried in alerts. Now most of them never reach a person.
The issue
Their on-call rotation was getting paged round the clock, and most of it was noise — the same alert firing twice, a brief blip in one region, two scheduled jobs stepping on each other. None of it was an emergency, but every page still pulled someone out of what they were doing. The engineering lead told us this rotation had been the team's biggest source of frustration for months. They'd already tried a couple of off-the-shelf tools and a contractor before us, and none of it really moved things. By the time we talked, they were ready to rebuild it properly, starting from where the alerts actually come from.
Discovery & analysis
- Three months of their alert history was the starting point — grouped by what fired, where it came from, when it happened, and how it actually got resolved.
- Most of the pages turned out to be the same handful of harmless patterns: an alert that cleared itself in a few minutes with nothing behind it.
- Then the on-call playbooks, line by line with the team lead. A few hadn't been touched in years, and those were the ones throwing the loudest, most useless alerts.
How we worked with the team
- We sat with their on-call lead a few mornings a week while we tuned the system, working through it live rather than over tickets.
- A shared chat channel let anyone flag a bad call the moment it happened — each one became an example the system learned from the next day.
- Every couple of weeks there was a short review with the people actually on the rotation; they rewrote the playbooks the system now points to.
What we built & shipped
- When an alert fires, it goes to 3node first. If it matches one of the harmless patterns, it gets closed on its own with a full record kept. If it's real, the person on call gets the page along with the background they need — the recent related incidents and the code changes that touched that part of the system in the last day.
- Anything the system isn't sure about goes into a short review lane. The on-call engineer confirms or corrects it, and that decision feeds back so it gets better over time.
- Every auto-closed alert leaves a permanent record, which happens to be exactly the paper trail their auditors ask for — so nobody scrambles to assemble it later.
Outcome
Several on-call slots' worth of engineering time given back every week. Over a year that adds up to a meaningful six-figure saving — but mostly it's people not getting woken up for nothing.
Real incidents now get picked up in under two minutes instead of a quarter of an hour, and the noise pages dropped by roughly two-thirds.
No serious incident has slipped through in over a year of running. A couple of close calls were actually caught earlier than the old rotation would have.
- The team's own survey on how the rotation feels went from rough to genuinely good.
- Nobody has left the on-call team since — which, for them, was new.
Where it stands now
Still running, and we're still on it. Every quarter we retune as their systems change. Currently being scoped: flagging risky code changes before they ship, based on which parts of the system tend to cause trouble.
Got a process like this?
Talk to a founder.