Drive failure prediction for Kubernetes persistent storage

The Kubernetes storage problem is different from bare-metal

On a bare-metal host, the relationship between a physical drive and the workload running on it is direct. Drive /dev/sdb is your Postgres data volume. If /dev/sdb is degrading, you know exactly what is at risk.

In a Kubernetes cluster with dynamic persistent volume provisioning, that relationship is abstracted. A StatefulSet pod consumes a PersistentVolumeClaim. That PVC is backed by a PersistentVolume. That PV was provisioned by a storage class that may use local-path, Rook/Ceph, Longhorn, or a cloud-provider CSI driver. The physical drive underneath may be on any of a dozen storage nodes. If you are thinking about drive health, you need to traverse that full path to understand what is physically at risk when a PVC starts behaving badly.

This is the core challenge we had to solve to make Crest useful in Kubernetes environments. The agent cannot just report per-drive risk scores in a vacuum. It needs to surface that risk in terms that map to Kubernetes objects the SRE team is already monitoring.

How the Crest Kubernetes operator works

Crest runs as a Kubernetes operator in your cluster, deployed via a Helm chart. The operator has two components: a DaemonSet agent pod that runs on each storage node and reads physical drive telemetry, and a controller pod that reconciles drive risk state with Kubernetes API objects.

The DaemonSet pod runs with a hostPath volume mount to access block device interfaces and reads SMART and NVMe telemetry using the same methods as the bare-metal agent. The key difference from bare-metal deployment is what happens with the risk scores after collection.

The controller maps physical drive identifiers to PersistentVolume node affinity rules to determine which PVs have storage on which physical drives. It then annotates at-risk PVs with Crest risk metadata:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-a83f7c20-4b12-11ee-be56-0242ac120002
  annotations:
    crest.gocrest.org/health-score: "72"
    crest.gocrest.org/risk-level: "watch"
    crest.gocrest.org/predicted-failure-window: "18-28 days"
    crest.gocrest.org/physical-drive: "storage-node-04:/dev/sdc"
    crest.gocrest.org/last-updated: "2026-02-11T08:43:00Z"
spec:
  ...

These annotations are readable by anything that can call the Kubernetes API: Prometheus via kube-state-metrics custom resource annotations, your existing alerting rules, or your own tooling that watches PV state.

The metrics API surface

Beyond PV annotations, the Crest operator exposes a Prometheus-compatible metrics endpoint. The primary gauge is crest_drive_risk_score, labeled with the physical drive identifier, the host node, and any bound PV name:

# HELP crest_drive_risk_score Predicted failure risk score (0-100, higher = more risk)
# TYPE crest_drive_risk_score gauge
crest_drive_risk_score{drive="/dev/sdc",node="storage-node-04",pv="pvc-a83f7c20"} 28
crest_drive_risk_score{drive="/dev/sdd",node="storage-node-04",pv="pvc-b91e2d13"} 72
crest_drive_risk_score{drive="/dev/sde",node="storage-node-02",pv=""} 12

The empty pv label on the third line indicates a drive with no bound PVC, which may be a spare, an unprovisioned node disk, or a drive whose PV was recently deleted. All three are still tracked for health state regardless of PVC binding.

A typical Prometheus alerting rule for this setup:

groups:
- name: crest_storage
  rules:
  - alert: DriveHighRisk
    expr: crest_drive_risk_score > 70
    for: 1h
    labels:
      severity: warning
    annotations:
      summary: "Drive {{ $labels.drive }} on {{ $labels.node }} is at high risk"
      description: "PV {{ $labels.pv }} may need migration. Predicted window: check PV annotation."

StatefulSet workloads and the migration question

For most Kubernetes storage use cases, the operational response to a high-risk drive is: migrate the data off the affected PV before the drive fails. For Deployments using ReadWriteMany volumes, this is straightforward. For StatefulSets, it requires more care.

StatefulSet pods have stable network identities and stable PVC bindings. Pod postgres-0 always uses PVC data-postgres-0. That PVC is not designed to be migrated dynamically. The practical options when the drive backing data-postgres-0 is at risk are: manual PV migration with a maintenance window, a Velero-based backup-and-restore to a new PVC, or (for Rook/Ceph or Longhorn) triggering a replica rebalance away from the failing node.

Which path is right depends on your storage class, replication factor, and tolerance for application downtime. The Crest operator does not make this decision for you. It gives you the lead time to make it deliberately rather than reactively.

What the annotation model does not cover

We are not saying the PV annotation approach is a complete solution for all Kubernetes storage architectures. Distributed storage systems like Ceph add a layer of indirection where a "physical drive at risk" maps to degraded capacity in a storage pool, not a direct 1:1 PV failure. The annotation model is most directly useful for local-path PV provisioners and for bare metal Kubernetes where PVs map predictably to physical devices.

For Rook/Ceph deployments, the Crest operator annotates the OSD pod rather than the PV directly, because the Ceph OSD is the correct unit of failure mapping. A degraded OSD on a failing drive should trigger an OSD reweighting or removal, not a PV migration. The annotations surface the same risk data but with labels appropriate for a Ceph admin to act on, rather than PV-level labels that map to Ceph in a misleading way.

For cloud-provider persistent disks (GCE PD, EBS), drive telemetry is not accessible at the Kubernetes layer because the physical device is fully managed by the cloud provider. Crest does not instrument cloud-managed disks. That scope is explicitly out of bounds for the current agent design, and we do not pretend otherwise.

Installation and prerequisites

The Crest Kubernetes operator requires cluster-admin role for initial CRD and ClusterRoleBinding creation. The DaemonSet pods need privileged: true or specific capabilities (SYS_RAWIO) to access block device interfaces. In environments with PodSecurityAdmission in Restricted mode, the privileged DaemonSet will need an exemption in the relevant namespace.

Installation via Helm:

helm repo add crest https://charts.gocrest.org
helm repo update
helm install crest-operator crest/crest-operator \
  --namespace crest-system \
  --create-namespace \
  --set agent.apiKey=$CREST_API_KEY \
  --set agent.storageNodes.selector="node-role.kubernetes.io/storage=true"

After installation, drive health data appears in the Crest dashboard within 90 seconds for nodes that have the DaemonSet pod running. PV annotations are populated within 5 minutes of the controller reconciliation loop completing its first full pass.

What happens when a high-risk PV is identified

Once the operator is running and annotating PVs, the operational response depends on your storage class and replication setup. The Crest operator does not trigger automated pod migrations. That decision belongs to the operator on-call. What the annotations enable is an explicit, non-emergency window to make that decision.

For a StatefulSet pod backed by a local-path PV on a node with a high-risk drive, the recommended path is to schedule a maintenance window, use a snapshot or backup to create a new PV on a healthy node, update the StatefulSet's PVC reference (which requires a rollout), and verify data integrity before decommissioning the old PV. The lead time from a Crest "Watch" alert to actual failure is typically 14 to 30 days. That is sufficient for a coordinated migration without application downtime in most configurations.

For teams using Longhorn or Rook/Ceph with replication factor 3, the response is different: trigger a replica rebuild onto a healthy storage node before the failing drive drops below the replication minimum. Crest surfaces the at-risk node and drive so the storage admin can initiate the rebuild proactively rather than waiting for Longhorn or Ceph to detect the failure and begin automatic degraded-state recovery. The difference is a controlled rebuild at 3am versus an emergency rebuild at 3am.