The incident in summary
A 4TB SATA HDD running the transaction log volume for a PostgreSQL replica went offline at 03:17. The on-call SRE received a pagerduty alert about replication lag at 03:19. By 03:47, the situation had been classified as a hardware failure. The drive was physically unresponsive. Replacement hardware was sourced from a spare shelf. The replica was rebuilt from the primary. The SRE signed off at 06:52.
Three hours and thirty-five minutes. That is the human cost, measured in hours, without accounting for what it means to be woken at 3am and expected to diagnose a database problem that turns out to be a hardware problem that turns out to be a problem that was visible 11 days earlier.
We reconstructed this incident from the telemetry log of the failed drive. We want to walk through what the signals looked like and why they were not acted on.
The telemetry log 11 days before failure
This drive had SMART monitoring enabled. It passed the standard SMART health check on every scheduled poll in the 30 days before it failed. "PASSED" every time. The binary health check was not lying exactly, but it was not telling the useful story.
Here is what the raw telemetry showed during the same period:
Date SMART-5 SMART-197 SMART-187 SMART-188 Read Err Rate
2026-03-07 0 0 0 0 nominal
2026-03-08 2 0 0 0 nominal
2026-03-09 2 1 0 0 nominal
2026-03-10 2 3 0 0 nominal
2026-03-11 2 4 0 2 elevated
2026-03-12 4 7 0 3 elevated
2026-03-13 4 11 1 3 elevated
2026-03-14 6 14 2 4 high
2026-03-17 6 18 2 7 high
2026-03-18 -- -- -- -- device removedSMART attribute 197 (current pending sector count) went from 0 to 18 over 10 days. Attribute 5 (reallocated sectors) moved from 0 to 6. Attribute 188 (command timeout) moved from 0 to 7. These are not ambiguous noise. These are the textbook trajectory of a SATA HDD head entering the final wear phase.
Why the SMART health check missed it
The SMART "PASSED" result is evaluated against vendor-defined threshold values for each attribute. Most vendors set the threshold for attribute 197 somewhere between 0 and 100, with the failure threshold representing a level of sector instability that almost certainly guarantees imminent failure. Reaching a pending sector count of 18 was within the threshold band: not zero, but nowhere near the point that triggers the binary FAILED output.
This is the structural problem with threshold-based SMART health checks. They are calibrated to be conservative to minimize false positives. They tell you a drive has already failed, or is definitively about to fail. They do not tell you which drive in your fleet is three weeks into a degradation trajectory that ends in failure. The signal exists. The threshold-based check just discards it.
The on-call team had no visibility into the attribute trends. Their monitoring setup polled SMART health status (pass/fail) daily. It never stored or analyzed the attribute time series. Every day showed PASSED, because PASSED was technically accurate every day until the drive stopped responding.
What the failure signature looks like in hindsight
We are not saying SMART attribute trending eliminates false positives. A drive with stable non-zero attribute values, say, a reallocated sector count that has been at 4 for 18 months without movement, is not a replacement candidate. The signal is the trend, not the absolute value. Pending sector count increasing 3 to 5 per day is different from pending sector count stable at 14 for 6 weeks.
The clearest signal in this incident was attribute 197 moving consistently upward across 10 daily polls. A sector becomes "pending" when the drive attempts to read it during a routine scan or during an I/O operation and gets an error. The drive queues that sector for reallocation on the next successful write. When pending sectors are accumulating at a steady rate, it means the drive's surface is continuing to produce new read errors faster than the reallocate-on-write mechanism can clear them.
A drive accumulating 1 to 3 new pending sectors per day is one you want to be watching closely with a replacement timeline in mind. By day 10, this drive had 18 pending sectors and a rising command timeout count. Both are correlated with the physical degradation of the magnetic surface or the read/write head assembly entering instability.
The operational gap this exposes
Most SMART monitoring setups do one of two things: run smartctl -H and alert on FAILED, or run smartctl -a and dump raw attribute values to a log file nobody reads. Neither approach captures the useful signal.
What this incident needed was a system that:
- Stored SMART attribute values as time-series data, not just the current snapshot
- Computed rate-of-change on key failure-indicator attributes (5, 197, 187, 188, 196)
- Surfaced drives where those attributes were moving upward, even if they had not crossed vendor-defined failure thresholds
- Routed that information into the team's existing incident workflow before the drive became a 3am problem
The replacement could have been scheduled during a Tuesday morning maintenance window. The replica rebuild would have been planned, not emergency. The on-call SRE would have slept through the night. None of that required hardware changes or a fundamental shift in how the infrastructure was designed. It required treating SMART data as a time series rather than a binary health check.
What we changed after reconstructing this
After walking through this incident timeline, we added attribute delta tracking to Crest's scoring model with specific treatment for the three-attribute combination of rising 197 plus rising 188 plus any movement in 187. This combination appeared in this incident and in several others in our historical dataset. The combination of pending sectors accumulating, command timeouts increasing, and any reported uncorrectable errors being non-zero is a high-confidence signal for near-term SATA HDD failure.
The model now weights this combination as one of the strongest precursor patterns for drives in the 2 to 5 year age range. A drive showing this pattern gets a predicted failure window under 21 days and routes to a PagerDuty alert with medium severity, regardless of what its binary SMART health check reports.
The 11-day lead time in this case was enough to have scheduled and completed a replacement before the failure window. That is what this whole tool is built around: converting the gap between "technically still SMART PASSED" and "actively failing" into scheduled maintenance work.