Hardware Monitoring: The Alerts That Catch Silent Failures
Most estates monitor whether machines are up. Far fewer monitor the things that predict failure, which is where monitoring actually earns its cost. This covers what to watch and why the alert path matters more than the sensor.
Up or down is the least useful thing to monitor
By the time a machine is down, monitoring has told you nothing you would not have learned from a user within minutes.
The value is in the indicators that change before something fails, which give you the chance to act during working hours instead of at 3am.
The indicators worth alerting on
Drive health values changing. Reallocated and pending sector counts rising, uncorrectable errors appearing. Alert on the change rather than an absolute threshold — a count that moved is the signal. Our drive failure guide covers the values.
SSD endurance consumed. Predictable and plannable, unlike mechanical failure. Alert well before the rated limit. Our SSD health guide covers it.
Array status. Degraded or rebuilding, and how long a rebuild has been running.
Controller cache battery status. Its failure silently collapses write performance with no other symptom, and it is one of the highest-value alerts you can configure. Our slow server guide covers why.
Power supply status in redundant pairs. A failed supply produces no visible symptom until the second one goes. Our power supply guide covers this.
Temperature at equipment intake, and fan status. Our server room guide covers where to measure.
Memory correctable error counts. Correctable errors rising on a module frequently precede an uncorrectable one, and the module can be replaced during a planned window.
UPS battery status and runtime. Our UPS guide covers why self-tests mislead.
The alert path matters more than the sensor
An alert that arrives somewhere nobody reads is not monitoring. It is logging.
Four things to get right.
Alerts reach a person, by a route that person actually checks, at the time it matters.
Someone is responsible for each alert type. An alert going to a shared mailbox everyone assumes someone else reads is the classic failure.
Alert volume is low enough to be read. This is the one that breaks most monitoring. An estate generating dozens of alerts daily trains everyone to ignore them, and the important one arrives into that habit.
There is a documented action for each alert. An alert nobody knows what to do with gets acknowledged and forgotten.
Tuning: the work that makes it usable
Monitoring fails by noise far more often than by missing something.
Three practices.
Remove alerts nobody acts on. If an alert has fired fifty times and nobody has ever done anything, it is not an alert.
Set thresholds against your environment, not defaults. A temperature normal for your room should not alert.
Suppress expected noise during maintenance windows and rebuilds, deliberately and temporarily.
The test: when an alert arrives, does someone believe it? If not, the monitoring is not working regardless of how much it covers.
Out-of-band monitoring
Monitoring a machine through the operating system tells you nothing when the machine is down or the OS has failed.
Management controllers have their own network connection and their own power, and they report hardware health independently. That means they still answer when the machine does not, which is exactly when you need the information.
Two practical points. Put management controllers on a network you can reach when the production network is affected. And test access before you need it, because credentials nobody has verified are credentials nobody has. Our out-of-band guide covers this.
Where to start
If you are starting from nothing, five alerts deliver most of the value.
Drive health changes. Array degraded. Controller cache battery. Power supply failed in a redundant pair. Intake temperature.
All five are silent failures — things that have already gone wrong and that nobody would notice. Everything else can follow. Our inherited estate guide covers establishing what you have first.
Frequently asked questions
What should I monitor on servers?
The indicators that change before failure: drive health values, SSD endurance, array status, controller cache battery, power supply status in redundant pairs, intake temperature and fans, memory correctable errors, and UPS battery status.
Which five alerts should I set up first?
Drive health changes, array degraded, controller cache battery, power supply failed in a redundant pair, and intake temperature. All five are silent failures that nobody would otherwise notice.
Why does my monitoring get ignored?
Alert volume. An estate generating dozens of alerts daily trains everyone to ignore them, and the important one arrives into that habit. Remove alerts nobody acts on and set thresholds against your environment rather than defaults.
Is monitoring that a server is up enough?
No. By the time a machine is down, monitoring has told you nothing a user would not have reported within minutes. The value is in indicators that change before failure.
Why monitor through the management controller?
It has its own network connection and power, so it still answers when the machine does not, which is exactly when you need the information. Put it on a network you can reach and test access before you need it.
How do I know my monitoring is working?
The test is whether someone believes an alert when it arrives. If alerts are routinely acknowledged and ignored, the monitoring is not working regardless of how much it covers.
Tell us what you run and we will help you work out which failures would currently go unnoticed.
