Your monitoring doesn't have an alert problem, it has a context problem
A thousand alerts a night are not a thousand problems. How a two-line rule turned the noise of an FTTH network into incidents you can act on.
The night shift in a small NOC always looks the same: a screen with hundreds of red lines and one person deciding, by eye, which ones matter. It is not that monitoring is missing. There is too much of it.
For nine years I was on the other side of that screen. What I learned is that the problem is almost never a shortage of alerts: it is that every alert arrives on its own, with nothing around it to explain what it means.
An alert is not an incident
A device reporting loss of signal may be broken, or someone may have cut the power on that block, or it may have rebooted on its own and come back. Three very different situations producing exactly the same red line.
The operator resolves that by cross-referencing in their head: did this device report something else a while ago, are the neighbours complaining too, is there scheduled work. That cross-referencing is real work, it is done by hand, and it is done badly when you are tired.
The question worth asking is not “how do I send fewer alerts?” but “what has to happen for this to be worth waking someone up?”.
The rule I ended up using
In FTTH-Copilot, the diagnostics agent I wrote for fibre networks, the answer turned out to be fairly boring, which is exactly why it works:
Group alerts by device. Open an incident only when the same device reports at least two distinct kinds of alert within a time window.
That is all. No model, no magic threshold, no moving average.
Why two distinct kinds? Because noise tends to repeat itself identically. A flapping link sends the same loss-of-signal alert a hundred times. A device that is genuinely dying sends different things: first the optical power drops, then the temperature rises, then it goes away. Two different signals in the same place are far harder to explain by coincidence than a hundred identical ones.
The code that does this fits on one screen. It is published in the repository, in packages/alerts/src/correlate.ts, and the constant that sets the minimum is called DEFAULT_MIN_KINDS. It is 2.
What nobody tells you about time windows
The hard part is not the rule, it is deciding how long the window lasts and when the incident closes.
Too short and you lose real correlations: temperature takes time to climb. Too long and you start grouping things that have nothing to do with each other, and you are back to noise — except now the noise comes bundled and looks more serious.
There is no correct number. There is a method: look at the real incidents of the past few months, measure how long the second signal took to appear, and use that percentile. If you have no such history, start short and stretch. It is cheaper to miss a correlation than to invent one.
How to know it is working
Two metrics are enough, and neither of them is “number of alerts”:
- Incidents per shift. If it drops and resolution time does not get worse, you won.
- Incidents the operator closed without doing anything. If this number grows, your rule is grouping noise and promoting it to incident. That is worse than not grouping at all.
The second one is what matters. A system that lies to you with confidence does more damage than one that shouts.
What to take away
If your monitoring is drowning you, before buying a new tool try this: group by entity — device, service, customer, whatever makes sense in your domain — and require two distinct signals before you bother a person. It is an afternoon’s work and it is usually enough.
The expensive tool comes later, if it is really needed. It almost never is.