Here is a question that has ended careers: when did you last restore from backup? Not "when did the backup job last run green." When did you last take a backup, restore it somewhere, and confirm the data came back whole and usable?

For an uncomfortable number of organizations, the honest answer is "never," or "during the incident, which is how we found out it didn't work." The backup job runs every night and reports success. The reports are filed. Everyone assumes the recovery works because the backup works. Those are two completely different claims, and the gap between them is where disasters live.

Why a green backup means less than you think

A successful backup job proves one thing: data was written to the backup target without an error the software noticed. It does not prove:

  • That the data is restorable — the backup format is intact, the catalog isn't corrupt, the chain of incrementals is unbroken.
  • That the restore completes in acceptable time — a backup that takes a week to restore is useless for a system that needs to be back in hours.
  • That the restored system actually runs — application-consistent state, correct permissions, working dependencies, valid licenses.
  • That you know how — the one person who understood the restore process left, and the runbook is three versions out of date.
  • That the backup covered what matters — the job backs up the database files but not the config, or the VM but not the external secrets, so the restore is technically successful and functionally incomplete.

Every one of those gaps is invisible until you attempt a real restore. The backup report cannot see them. Only the drill can.

The failure modes a drill catches

Teams that start running restore drills are routinely surprised by what breaks. The common findings:

  • The chain is broken. An incremental somewhere in the chain is corrupt or missing, so you can restore to last month but not last night.
  • Restore time is a shock. The RTO on paper is four hours; the actual restore of the full dataset takes thirty. Nobody measured it until now.
  • The restore depends on the thing that's down. The backup catalog lives on the same infrastructure the disaster took out. The encryption keys are in the vault that's offline. The restore needs DNS or authentication that isn't up yet.
  • It restores but won't start. Database files restored without application consistency, so the service won't come up. Or the VM boots but can't reach a dependency that the backup didn't capture.
  • Nobody is sure of the steps. The process is tribal knowledge, half-documented, and the half that's written is stale.

None of these are exotic. They're the normal state of an untested recovery process. The drill is what converts them from a 2 a.m. discovery into a Tuesday-afternoon fix.

What a real drill looks like

A restore drill doesn't have to be elaborate. It has to be real. The minimum useful version:

  1. Pick a system and a recovery point. Rotate through your important systems over time so each gets exercised.
  2. Restore it to isolated infrastructure — a test environment, a spare host, a cloud sandbox. Not production. The point is to prove recovery without risking the live system.
  3. Time it. Record how long the restore actually takes end to end. Compare against your stated RTO. The gap is a finding.
  4. Verify it works, not just that it completed. Boot the system. Open the application. Check the data is current and whole. Run whatever smoke test proves the thing is genuinely usable.
  5. Write down what broke and fix it. Every drill produces findings early on. That's the drill working.

Do that once a quarter for your important systems and you will never again be in the position of discovering during an incident that your recovery doesn't recover.

Why it doesn't happen

Restore drills don't get run for entirely human reasons. They take time nobody budgeted. They're nobody's explicit job. The backup is green so the risk feels addressed. And there's a quiet reluctance to go looking for a problem you'd then have to own.

That last one is the real enemy. The problem exists whether or not you look. The only choice is whether you find it on a scheduled afternoon with coffee and a test environment, or at the worst possible moment with the business down and everyone watching.

An afternoon a quarter. That's the price of knowing — actually knowing, not hoping — that the thing you're counting on will work. Against the cost of finding out the other way, it is the cheapest insurance in all of operations. Run the drill.