Why we collected this data
When we started building Crest's NVMe failure model, we could not find published telemetry data that covered the failure-precursor period with enough granularity to validate feature selection. Most publicly available drive reliability studies report failures as point events. What we needed was the signal trajectory across the 30 to 90 day window before failure.
Over 90 days in 2025, we instrumented a mixed fleet of 340 NVMe drives across three host environments: a write-heavy database workload, a read-intensive object storage tier, and a mixed compute cluster. Drives ranged from consumer-grade M.2 units to enterprise data center NVMe in the U.2 form factor. This post describes what the failure-precursor signal distribution actually looked like, and which attributes did the most predictive work.
Fleet composition and observed failure rate
The 340 drives broke down roughly as follows:
- Enterprise U.2 NVMe (Samsung PM983, Seagate Nytro XM7): 180 drives, average age 26 months at observation start
- Enterprise M.2 (WD PC SN730, Kingston DC1000M): 92 drives, average age 14 months
- Consumer-grade M.2 (Samsung 970 EVO, WD Blue SN570): 68 drives, average age 19 months
Over the 90-day window, 14 drives reached a failure state: unrecoverable error, kernel-level device removal, or read-only failover. That is a failure rate of approximately 4.1% across the observation period. Annualized and adjusted for age distribution, this is consistent with published JEDEC endurance profiles for the enterprise units and somewhat higher than rated for the consumer-grade units, which is expected given the actual write workloads exceeded the consumer spec.
Where the early signals showed up
Of the 14 drives that failed, 11 showed detectable precursor signals more than 14 days before the failure event. The three that showed no early signals were sudden firmware-level failures with no prior telemetry movement. Those we currently cannot predict from health log data alone.
For the 11 drives with precursor signals, the primary observable changes were:
Available spare percentage declining past 20%: This was the most consistent early signal. Available spare is reported in the NVMe SMART/Health Information log (per NVMe 1.4 specification, section 5.14.1.2). For enterprise drives, the point where this value starts declining meaningfully is a more reliable precursor than it is for consumer drives, where spare pool sizing is less consistent across firmware versions.
Media and data integrity errors increasing: This log field counts reads from cache that required error correction from flash media. Nine of the 11 predictable failures showed this counter moving before the available spare percentage changed, typically by 3 to 7 days. The count increase was often small in absolute terms, 10 to 80 additional errors over the observation window, but statistically significant against the flat baseline the same drive showed in the prior 30 days.
Percentage used crossing 85%: For enterprise drives with known TBW ratings, this endurance indicator is straightforward. For consumer drives, the calculation is less standardized across vendors, and we found variance of roughly plus or minus 12 percentage points for the same physical endurance state across different firmware implementations. We track it but weight it lower for consumer drives in the model.
The consumer vs. enterprise signal gap
The most practically useful finding from this period was how differently consumer and enterprise NVMe drives report their internal state. Enterprise drives had consistent, predictable telemetry. The available spare counter decremented at a rate that correlated well with actual NAND wear. The media error counter incremented before failures in a pattern we could model reliably.
Consumer drives were significantly noisier. The same available spare percentage value did not represent the same physical wear state across different consumer models. Two Samsung 970 EVO drives at 30% available spare showed meaningfully different failure probabilities based on their total bytes written history and the specific firmware revision's spare management algorithm.
This variance is why a single-threshold approach to NVMe health monitoring underperforms in mixed fleets. "Replace when available spare falls below 10%" is a reasonable rule of thumb for a homogeneous enterprise fleet. It generates excessive replacements in a mixed environment because consumer drives report spare depletion on different curves.
Three drives worth walking through in detail
We are not claiming this 90-day sample is representative of all NVMe fleets. The workloads here were write-intensive, which accelerates wear-based failure modes and may underrepresent failure distributions you see in read-heavy or archival environments. The signal patterns hold directionally, but the timing relationships will differ.
Drive A (Samsung PM983, U.2): Media errors rose 22 days before failure. Available spare dropped from 48% to 31% over the following 8 days, then from 31% to 9% in 4 days. The rate of change on spare depletion was the useful signal, not the absolute threshold crossing.
Day -22: media_errors: 0 available_spare: 48%
Day -14: media_errors: 34 available_spare: 44%
Day -8: media_errors: 91 available_spare: 31%
Day -4: media_errors: 147 available_spare: 9%
Day 0: device removed (kernel: nvme0n1 I/O error, reset failed)Drive B (WD PC SN730, M.2): Percentage used crossed 91% with no simultaneous spare change. Critical warning bit set 6 days before device removal, with no precursor in the other fields. This was the hardest to model: a single-field predictor with no multi-signal confirmation available until very late in the degradation window.
Drive C (Samsung 970 EVO, consumer M.2): Available spare dropped from 60% to 44% over 11 days, media errors were flat throughout. This turned out to be aggressive spare block reclamation during a garbage collection cycle, not a precursor to failure. The drive was still running at the end of the observation window. This is the false-positive candidate a threshold model would have acted on unnecessarily.
What this shaped in the Crest model
The raw observations from this period shaped several decisions in how we handle NVMe-specific telemetry. We do not treat NVMe and SATA drives with the same feature set or the same thresholds. We maintain separate per-model-family wear curves rather than a universal NVMe profile. We weight rate-of-change signals more heavily than absolute-value thresholds for the available spare field, because Drive C above is not an isolated case.
The practical consequence for a team running the Crest agent: the false-positive rate on consumer NVMe drives is lower than threshold-based monitoring at the cost of a slightly wider predicted failure window. For teams running enterprise drives primarily, the model produces tighter windows because the telemetry is more consistent. The tradeoff is explicit in how the model's confidence thresholds are tuned per tier.
We are still collecting. The 90-day snapshot gave us enough to validate the approach and ship. The long-term goal is a per-model-family failure pattern library covering enough end-of-life examples for each drive model to produce prediction windows with high confidence across the full range from consumer to data-center NVMe.