Prove the Restore Before You Call It a Backup

A backup job that completes is not evidence that you can recover. The backup pipeline has a finish line the moment the job reports success, but recovery has a different finish line, and reaching it takes longer and fails more often. The two are routinely confused, which is why the 2026 reports on data resilience keep landing on the same uncomfortable number: most teams believe they can recover, and far fewer actually do ^{1 2}.

This is an infrastructure problem before it is a process problem. Stateful workloads now live in Kubernetes clusters where a backup has to move both application definitions and volume bytes, and the failure modes live in the seams between those pieces ². Databases now offer point-in-time recovery measured in seconds, and object storage offers snapshot retention measured in years, and choosing wrong on either sets an RPO you only discover during an incident ³. The goal is not a backup strategy. The goal is a restore you have proven, on a schedule, against a defined target. This article walks through the numbers, the architecture choices, and the concrete test that converts a backup from an assumption into a capability.

Confidence is not capability

The opening figure from the Veeam Data Trust and Resilience Report 2026 captures the gap in two figures. Among more than 900 senior IT, security, and risk leaders surveyed, 90% said they were confident they could recover from a cyber incident within their recovery time objectives, yet among organizations that actually suffered a ransomware attack affecting data or operations, only 28% fully recovered all of their affected data ¹. The average organization recovered 72% of the data it lost, and 44% recovered less than three quarters of it ¹.

That is not a story about a few unprepared shops. The report frames the disconnect as structural: confidence in recovery and proof of recovery have become different capabilities, and the gap is widening as AI workloads multiply the number of places data lives ¹. The organizations that did better shared four practices rather than one lucky tool: clear visibility into where data and AI risk live, enforced security controls rather than policy documents, proven recovery through realistic testing and validation, and executive agreement on what counts as recovered ¹. Nothing on that list is exotic, and none of it is about buying a bigger backup appliance.

The same pattern shows up inside Kubernetes. The CNCF's September 2026 disaster recovery guidance states the ecosystem's open gap plainly: "Backup success is treated as recovery proof," and while backup completion metrics are widely monitored, the results of restore rehearsals rarely are ². The two statements are the same observation from different angles. Teams instrument the job that produces a file and neglect the rehearsal that proves the file does anything.

Two numbers decide the architecture

Everything downstream of "we should be resilient" hangs on two objectives. The recovery time objective (RTO) is the longest acceptable outage before a service being down causes real harm, and the recovery point objective (RPO) is the largest acceptable gap, measured in time, since the last point you could recover from ⁴. RTO drives how fast your failover and restore tooling must run. RPO drives how often you must replicate or snapshot, because it bounds how much data you tolerate losing ⁴.

Those two numbers sort the standard recovery strategies on one axis. Backup and restore, the cheapest option, restores from periodic backups and gives the highest RTO and RPO, so it suits dev and test workloads. A pilot light keeps a minimal core of infrastructure running in a secondary region with data continuously replicated, and brings the rest up on failover, giving a medium RTO at moderate cost. Warm standby maintains a scaled-down but fully working environment in the secondary region, trading higher cost for a low RTO that fits business-critical applications. Multi-site active-active runs two complete environments simultaneously, distributing traffic in real time and reaching near-zero RTO and RPO at the highest operational cost ⁴.

The practical rule is to classify workloads before choosing a strategy. A mission-critical financial application might demand an RTO of minutes and an RPO of seconds, which points at multi-region active-active. An internal reporting tool can tolerate losing a day of data and being down for a day, which makes plain backup and restore the rational, cheaper choice ⁴. Choosing the expensive option for everything wastes money. Choosing the cheap one for everything guarantees a miss on RPO or RTO when it matters. Neither number belongs in the architecture decision for all workloads.

Low RPO does not have to mean constant full snapshots. AWS Backup offers two complementary modes with different tradeoffs, and the guidance is to use both. Continuous backup creates one full copy and then constantly records transaction logs, letting you rewind to a chosen point in time within one second of precision, back up to 35 days ³. Snapshot backups can run as often as hourly and be retained for up to a century, and copies circulate for full or incremental protection ³. Continuous backup gives you the precise RPO for recent data, and snapshots give you the retention depth for the long tail. Force either one to do both jobs and you give something up: continuous backup cannot go back further than 35 days and does not support on-demand backups, while hourly snapshots leave a multi-hour RPO window for the period people actually hit ³.

The restore windows confirm the same theme. Amazon RDS activity restores to within the most recent five minutes, S3 to within fifteen, and Aurora to the latest restorable time, which is typically under five minutes ³. Those are real RPO numbers, not aspirational ones, and they are achievable only because the transaction log is being replayed continuously rather than snapshotted on a timer.

Where Kubernetes backups break at the seams

The CNCF guidance is deliberately reproducible rather than advisory. It documents three failure scenarios, each recreated on a laptop, and each one separates having a backup from being able to recover ². The first is a backup that reports Completed while the volume data never reached the recovery store. A Kubernetes backup is two different things: the resource definitions and the persistent volume bytes, and a tool can report the operation finished while only the definitions moved ². The fix is to look past the backup phase and confirm the volume bytes actually landed, the way Velero's data mover reports a byte count for each backup ².

The second scenario is the GitOps trap, and it is the one that most often shows up as a green dashboard with an empty database. Git stores intent, the declarations, while the backup store holds the state. A GitOps controller in a recovery cluster syncs the application from Git perfectly, the StatefulSet rolls out, the pod is running and ready, every status is green, and the database is empty, because nothing ever restored the stored data ². In the lab, syncing from Git produced a running database and then relation "attendees" does not exist on the first query ². Recovery required bringing back the application from the backup store, not just the sync. The general lesson is that declarations and state are separate things and carry separate tools, and the knock-on effect is that infrastructure as code recovers the cluster while the backup recovers the data. A disaster recovery plan that starts with "restore the backup" has to say what the backup gets restored into, which is the question IaC answers ².

The third scenario is consistency across multiple volumes. The lab writes matched pairs of records, an order and a payment, five times per second, and the invariant is that every payment has its order. Snapshotting two volumes individually, five seconds apart, produced two snapshots that were each individually perfect, and restoring both produced twenty five payments with no matching order ². No component failed and every operation reported success. The combined recovery point simply described a moment in time that never existed. The fix is the volume group snapshot, which reached general availability in Kubernetes 1.36: one object selects volumes by label and the CSI driver takes one coordinated, crash-consistent recovery point across all of them ².

Velero is the reference implementation the CNCF lab runs, and it is the Kubernetes-native answer to the backup side of these scenarios. It backs up and restores cluster resources and persistent volumes, schedules recurring backups, runs pre and post backup hooks, and has been a CNCF Sandbox project ⁵. It added volume group snapshot support across multiple volumes in version 1.17 ⁶ and concurrent backup processing in 1.18 ⁵, which is the direction the gap analysis points at. But the CNCF guidance is careful to say the tool is not the point: the failure modes apply to any tool occupying the same role, because the lesson is about the seams between resources, not the quality of a particular binary ².

The recovery test is the product

The CNCF guidance defines what a recovery test actually is, because the term gets used loosely. A recovery test is not deleting a pod and watching it come back, which tests workload reconciliation. A real recovery test restores a complete stateful application into a clean target that has never run it before, validates the data and the user path against expected contents rather than against resource status, and measures the whole process with a clock ². The lab's own measurement makes the distinction memorable: powering off production to validated data took about four minutes live and just under two minutes in a rehearsed rerun, and the general lesson was that the moment the dashboards turned green was not the recovery. The recovery happened when the data came back and was checked ².

That clock is what turns the exercise from a demo into a gate. RTO is a number, and the only way to know whether your tooling actually meets it is to run it and time it, then run it again after the next infrastructure change moves the ground underneath it. The common guidance is to test at least quarterly, and after any major infrastructure change, on the assumption that runbooks and failover scripts rot quietly between rehearsals ⁴.

The same discipline extends the classic backup rule. The 3-2-1 rule asks for three copies, on two different types of media, with at least one off-site, and it remains a sound foundation but not a complete one, because it was designed before ransomware and immutable storage existed ⁷. The 3-2-1-1-0 extension adds the two components modern threats demand: one immutable copy that cannot be altered or deleted within its retention window, typically via write-once storage such as S3 Object Lock, and zero recovery errors, meaning each backup is actually proven recoverable rather than merely present ⁷. Backup vendors push automated verification for a reason: recovery behavior only surfaces on a restore, and a backup that has never been booted or restored tells you nothing about whether the restore will work ⁷. That is the gap between the 90% and the 28% in the opening numbers.

The AI workload changes what is in the recovery set

The cross pillar of the directive is here to make this concrete. Twenty years ago the recovery set was databases and file servers. Now it includes vector stores, model weights, embeddings, and the training and fine-tuning data that produced them, and those assets carry the same restore obligation as a financial table. The Veeam report ties AI directly to the confidence gap: 43% of leaders say AI adoption is outpacing their ability to secure data and models, 42% report limited visibility into the AI tools and models in use, and 40% say security policies have not yet been updated for AI-specific risk ¹.

The already-flimsy assumption that a backup is recoverable gets worse when the data is new. Nobody has a rehearsed restore for an embedding index the way mature teams have one for a database, because the index is often treated as regenerable and therefore exempt from the backup conversation. Regenerable is a true property and a dangerous assumption at the same time: rebuilding an embedding index is not free, and the training data behind it is not on the Git branch that restores it. Teams already lean on checkpoint-based restart to make scale-to-zero fast on model servers, and that checkpoint discipline is itself a restore, one that deserves the same rehearsal as a database. The discipline from the earlier sections applies unchanged. Decide the vector store's RPO, choose a strategy that meets it, and rehearse a restore into a clean target before the incident decides the RPO for you.

Make proving the restore a habit

The difference between a system that recovers and one that only claims to is whether the restore has been run, timed, and validated more than once. The pieces are all available: point-in-time recovery with second-level precision, snapshot retention measured in decades, group snapshots that hold a multivolume application to one consistent moment, and tools that will run the rehearsal for you. None of them changes the outcome by existing. Infrastructure teams that confide in the completion status of a backup job are predicting their own 28% outcome, and the teams that get out of it schedule the restore test, point the same monitoring rig at it you point at production, and treat a failed rehearsal as the valuable signal, because it surfaced the problem before the incident did.

Apply the same standard to the work in the room. In our own publishing pipeline, nothing counts as done because a file exists, and a draft carries a mechanical verification gate that has to pass before an editor looks at it, on every draft, not as a yearly chore. Completion and proof are different verbs there, and we treat them that way, for the same reason the CNCF lab points a clock at its restore ². If you already treat IaC as the thing that rebuilds infrastructure, read why the control-plane model is what keeps that converged, and pair it with the restore run that proves the data returns with it. The cluster is recovered by code, and the data is recovered by a backup you have actually restored.