The Restore Drill: Prove Your restic Backups Are Real Before Ransomware Does It For You

You have a restic repo. You probably run restic backup on a schedule. You may have even run restic check once and been happy. What you haven't done — unless you've actually restored — is prove the data comes back intact.

This drill turns that proof into a weekly automated event: check the repo, restore the latest snapshot to a scratch directory, and byte-compare the result against the live files. A green run means your backups are real. A red run means you found the problem this week, not on the morning of the ransom note.

Before you start//

• restic 0.19.x: every command below was verified against the 0.19.1 stable docs (working with repos, restore, backing up, scripting).
• One existing repository — local path, SFTP, or S3/MinIO/B2-compatible. If it's offsite (S3/MinIO/B2), that's better: the drill then validates your offsite copy, which is the "2" in 3-2-1.
• A scratch directory with room for a full restore of your latest snapshot (same order of magnitude as your repo's unique data; for a 50GB repo, budget ~50GB of scratch space on a different disk than the repo, see failure mode #3).
• cron or a systemd timer on the box running the drill (or a separate box that can reach the repo).
• Roughly an hour of your time to set up. The drill itself runs unattended; on a big repo the --read-data check and the full restore are the slow parts (tens of minutes, not hours).
• Blast radius: the drill is read-mostly. It reads the repo (the --read-data check downloads pack data, so remote repos incur bandwidth), writes a scratch directory, and touches nothing in your live filesystem. The only thing that could "break" is your patience with a slow first run, and the scratch dir, which you can rm -rf at any time. No snapshot needed, but know where the scratch dir lives.

Set these once in the script below and everywhere it runs:

REPO=/srv/restic/repo          # or sftp:/backup@nas:./restic, s3:minio:bucket, etc.
DRILL=/mnt/scratch/restore-drill   # on a DIFFERENT disk than the repo

Step 1: Integrity first — restic check, then check --read-data//

restic -r "$REPO" check

The plain check verifies the repository's internal structure — snapshots, trees, and pack file metadata — without reading the data. Fast. It answers "is the repo organized correctly?"

That's not enough. An attacker (or a silently dying disk) can keep the structure pristine and still rot the data. The docs' framing for the stronger check is worth quoting, because it's exactly the threat model: the repository is "saved on a server that has a faulty hard drive, or even worse, attackers get privileged access and modify the files in your repository" to make you restore malicious data. The answer is --read-data:

restic -r "$REPO" check --read-data

This reads and re-hashes every pack file, the real data. The docs warn it "has to download all pack files in the repository," so it's slower and, on a remote repo, costs bandwidth. That's a weekly job, not a per-backup job.

Why both: check passing but --read-data failing is the "structure is fine, data is corrupted" case, the exact case a plain check misses. A pack-level tamper is invisible to anything that doesn't read the bytes.

Step 2: The actual restore — into a scratch dir, never in-place//

rm -rf "$DRILL" && mkdir -p "$DRILL"
restic -r "$REPO" restore latest --target "$DRILL"

restore latest restores the newest snapshot; --target puts it in the scratch directory instead of overwriting your live files. The reason for the scratch dir comes straight from the docs:

Restoring data in-place can leave files in a partially restored state if the restore operation is interrupted.

If the drill (or you, in an actual incident) gets interrupted mid-restore (disk full, network drop, OOM), an in-place restore leaves your real files half from the backup, half from before. Restoring to a fresh scratch dir means an interrupted drill just wastes an hour and a scratch dir, nothing else.

If you ever do want in-place restores, the --overwrite options are always (default), if-changed, if-newer, and never — but for a drill, the scratch dir is the default.

Spot-check a single file without restoring the world:

restic -r "$REPO" restore latest --include /home/user/work/important-file.txt --target "$DRILL"

(Paths to --include must be the absolute paths as stored in the snapshot — restic ls latest or restic find <name> shows them.)

Step 3: Verify it, byte-for-byte, on sampled files//

A successful restore with no error output is not the same as correct data. Compare hashes:

# pick a few real files that exist both live and in the restore
head -c 200 /home/user/work/important-file.txt | sha256sum
head -c 200 "$DRILL/home/user/work/important-file.txt" | sha256sum

Match = the restored bytes are the live bytes. For a broader check without restoring everything, compare the snapshot's file list against what you have:

restic ls latest | wc -l
find /home/user/work -type f | wc -l

They won't be identical (the snapshot excludes what you excluded at backup time), but a restore that's wildly short is a red flag worth investigating.

And then do the dumb, beautiful thing: open a restored file. head it, open it in a text editor, run it if it's a script you trust. A file you've looked at from a restore is worth a thousand green exit codes.

Step 4: Put it on a timer — the script that makes it a drill//

A drill that can't fail isn't a drill. Here's the shape of the weekly job (bash; the same logic maps to whatever scheduler you run the repo on):

#!/usr/bin/env bash
# restore-drill.sh — weekly restic backup verification
set -uo pipefail
REPO=/srv/restic/repo
DRILL=/mnt/scratch/restore-drill

# cron doesn't inherit your shell's env: this is where backups silently die
export RESTIC_PASSWORD_FILE=/etc/restic/.password   # or RESTIC_PASSWORD / a keychain cmd

fail() { echo "DRILL FAIL: $1" >&2; exit 1; }

# 1. structural check
restic -r "$REPO" check || fail "restic check (structure)"

# 2. data check — the "someone modified your packs" check
restic -r "$REPO" check --read-data || fail "restic check --read-data (data)"

# 3. test restore to scratch
rm -rf "$DRILL"; mkdir -p "$DRILL"
restic -r "$REPO" restore latest --target "$DRILL" || fail "restore latest"

# 4. byte-compare a sample
f="/home/user/work/important-file.txt"
[ -f "$DRILL$f" ] || fail "restored sample missing: $f"
[ "$(head -c 200 "$f" | sha256sum)" = "$(head -c 200 "$DRILL$f" | sha256sum)" ] \
  || fail "sha256 mismatch: $f"

# 5. record success — the "last good drill" timestamp, queryable
restic -r "$REPO" snapshots --json > /var/lib/restore-drill/last-good.json

rm -rf "$DRILL"
echo "DRILL PASS: $(date -Is)"

Wire it to cron (0 3 * * 0 /usr/local/bin/restore-drill.sh) or a systemd timer with the same weekly cadence, and alert on any non-zero exit (mail, ntfy, whatever). The snapshots --json line (documented in the scripting section) gives you a machine-readable record of the snapshots the drill ran against, including the time of each, so "when did we last prove our backups work?" has an exact answer.

Two deliberate traps to build in, so the alerting is real:

1. The drill must be able to fail. Run it once against a deliberately wrong repo (point REPO at a scratch dir that isn't a restic repo) and confirm your alert fires. If the alert never fires, the drill is just a green light.
2. The RESTIC_PASSWORD line is the whole game. Cron runs with a bare environment; if the password isn't available there, restic dies at "enter password for repository" and every backup — and every drill — fails silently. This is the classic "the backup's been failing for four months and nobody noticed" bug. The drill's alert is what catches it.

Step 5: The offsite leg//

If your repo lives on S3/MinIO/B2, the drill is already validating your offsite copy — check --read-data over the network is the strongest signal you can get that the copy you can actually use in a disaster is good. That's the "2" in 3-2-1 (three copies, two media, one offsite) finally being a fact instead of a claim. If your repo is purely local, the drill proves the local copy — and the honest follow-up is a second, offsite repo with its own drill.

When it breaks//

The three common failure modes, and how to read them:

1. check passes, --read-data fails: structure is intact, data is corrupt or tampered. This is the attacker-modified-pack case the docs describe. Don't rebuild from that repo; go to the offsite copy and verify that one. If both fail, you've found out the backups were lying, which, again, is the point of finding it on your schedule.
2. Restore fails or is incomplete — usually disk (scratch dir full mid-restore), network (remote repo, bandwidth), or memory on a very large restore. Because the drill restores to scratch, your live files are untouched; fix the resource, re-run. If an in-place restore is ever interrupted (real incident, not drill), that's exactly the "partially restored state" warning — re-run with --overwrite always (the default) so mismatching files get re-verified and completed.
3. RESTIC_PASSWORD missing in the scheduler's environment — restic exits at the password prompt, drill and backup both fail, and nobody notices because nothing emailed. Symptom: your alert firing on "restore" or "check" steps that used to pass. Fix the env var (a RESTIC_PASSWORD_FILE or RESTIC_PASSWORD_COMMAND beats a literal password in a cron line) and re-run.

Rollback is trivial by design: the drill writes only to the scratch dir and its log file. Kill it, rm -rf "$DRILL", and you're exactly where you started.