Immutable + Offsite Backups on a Budget: restic + rclone, So Ransomware Can't Delete Its Own Insurance

You have a restic repo. You run restic backup on a schedule, and you've even done the restore drill — so you know your backups come back. But here's the hole: if ransomware gets root on that host, it also has your RESTIC_PASSWORD_FILE, your SSH key, and restic itself. It can run restic forget --prune (or rm -rf) and burn every snapshot in the repo while you watch. That's why the restore drill is part one: this tutorial makes the offsite copy append-only, so a compromised host keeps its backup credentials but only gets add rights on history it can't delete or rewrite.

Before you start//

• restic ≥ 0.16, tested here against 0.19.x. Every restic command below was verified against the 0.19.1 stable docs — specifically the SFTP section (init syntax and URL forms) and the "Other Services via rclone" section (rclone.program / rclone.args).
• rclone on the remote side. Verified against the current rclone serve restic reference — that's where --append-only ("Disallow deletion of repository data") and --stdio ("Run an HTTP2 server on stdin/stdout") are documented. rclone does not need to be installed on the backup client (see Step 3).
• One offsite target: a $5 VPS with SSH, or object storage (B2/S3/MinIO) if you'd rather skip the VPS — the object-storage variant is in Step 4. A Hetzner Storage Box works too; its restricted shell runs rclone serve restic --stdio as a forced command, which community writeups document end to end.
• Skill floor: comfortable with SSH and env vars. Roughly 2 hours for first setup, ~30 minutes per additional host (new key + new authorized_keys line).
• Blast radius: this touches one remote user account you're creating from scratch, plus the scheduling on your backup client. Nothing in your live data is modified — backups still run from where they already run. Snapshot nothing; just don't point backup at a user who has sudo.

Set these once and use them everywhere:

[email protected]        # dedicated remote user, created in Step 1
REPO_PATH=/srv/restic-repo            # where the repo lives on the VPS
PASSFILE=/etc/restic/.password        # RESTIC_PASSWORD_FILE on the client (0600)

Step 1: A dedicated backup user with no sudo and no shell//

On the VPS, create a single-purpose account. No sudo means a compromised key can't escalate; a restricted or absent shell means it can't do much else either.

sudo useradd -m -d /home/backup -s /usr/sbin/nologin backup
sudo mkdir -p "$REPO_PATH" && sudo chown backup:backup "$REPO_PATH"

Why: the whole design is "the attacker can reach this account but not your system." One user, one home dir, one repo path, nothing else. If you're on a distro where nologin fights with forced commands (it shouldn't — the forced command runs instead of the shell), keep nologin anyway; it's still the correct default posture for an account that only ever speaks restic.

Step 2: Pin its authorized_keys to exactly one command//

This is the heart of the tutorial. On the VPS, in /home/backup/.ssh/authorized_keys, put one line — a forced command= plus three forwardings killed (the ArchWiki OpenSSH article shows the forced-command pattern and points at sshd(8)'s authorized-keys file format for the per-key options):

Generate the keypair on your backup client first (ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_backup), and make sure this is a separate key from anything else that logs into the VPS. Set permissions: chmod 700 /home/backup/.ssh && chmod 600 /home/backup/.ssh/authorized_keys.

Why each piece:

• command="..." — any login with this key runs exactly this command, no matter what the client asked for. The attacker's ssh backup@vps ls still gets a restic REST server on stdio.
• --stdio — serves the restic REST API over stdin/stdout instead of opening a listening port (rclone reference). No new firewall hole on the VPS; SSH is the only door.
• --append-only — "Disallow deletion of repository data" (rclone reference). Deletes and re-uploads of existing files get rejected.
• The three no-*-forwarding flags close the side channels that would otherwise let one key do more than run a command.

Install rclone on the VPS if it isn't there. The quick path is curl https://rclone.org/install.sh | sudo bash — be careful, that pipes remote code straight to root; prefer your distro's package when it ships one. Either way pin the version you test with (that's what the docs and reference above were verified against).

Step 3: Initialize + back up through the rclone backend (no local rclone needed)//

The subtle part, and the reason this tutorial exists instead of being a copy-paste of "use sftp:": you must init the repo over plain SFTP first, because an append-only key can't create anything. Then day-to-day backups go through restic's built-in rclone backend, which spawns the forced command for you — so rclone only ever runs on the VPS, never on the client (the restic 0.19.1 docs cover both halves: SFTP init syntax, and rclone.program / rclone.args).

First, the one-time init with a normal (unrestricted) key — e.g. your own user on the VPS, or a temporary second key for backup:

export RESTIC_PASSWORD_FILE="$PASSFILE"   # create it: 0600, holds the repo password once
restic -r sftp:[email protected]:/srv/restic-repo init

That's the exact URL form from the restic SFTP section (relative paths work too; for a non-default port or IPv6 host use sftp://user@[::1]:2222//path — note the double slash). Then back up through the append-only forced command:

restic -o rclone.program="ssh -i ~/.ssh/id_ed25519_backup [email protected]" \
       -r rclone:/srv/restic-repo backup /path/to/data

Why this works: restic's rclone backend builds its command as rclone.program + rclone.args (default serve restic --stdio ...) + the repo path after rclone:. Here rclone.program is literally "ssh to the VPS," and because the key has a forced command, everything after it gets discarded and replaced by the pinned rclone serve restic --stdio --append-only /srv/restic-repo. The client never needs rclone installed; the append-only flag lives on the server side where it can't be fudged.

Schedule it with your normal mechanism (systemd timer or cron), keeping RESTIC_PASSWORD_FILE in the environment — the restore drill's failure mode #3 applies verbatim: a scheduler without the password file fails silently.

Step 4: Object storage instead of a VPS? Use an append-only application key//

If you'd rather have B2/S3/MinIO than manage a box, the same two-key idea maps onto storage permissions: create a dedicated app key with write but no delete (on B2 that means picking capabilities without deleteFiles — list/read/write only; on MinIO/S3, a policy without s3:DeleteObject), and point restic at it — for B2, the restic 0.19.1 docs explicitly recommend the S3-compatible API over the native B2 backend.

export AWS_ACCESS_KEY_ID=<append-only-key-id>
export AWS_SECRET_ACCESS_KEY=<append-only-secret>
restic -r s3:https://<endpoint>/<bucket>/restic backup /path/to/data   # or rclone:<remote>:<path> for services restic lacks

Why: the storage provider enforces immutability server-side, no SSH involved. The trade-off is that "no delete" policy enforcement varies more between providers than --append-only does — test it (Step 5) before you trust it with your only offsite copy.

Step 5: Verify the wall actually holds (non-negotiable)//

An append-only config you haven't tested is a rumor. Two checks, in order:

1. Try to delete through the append-only key — watch it fail. With the restricted key active, attempt something that needs write+delete rights and confirm rejection:

restic -o rclone.program="ssh -i ~/.ssh/id_ed25519_backup [email protected]" \
       -r rclone:/srv/restic-repo forget --keep-last 0 --prune

If your append-only setup is correct, this fails (a delete/overwrite gets rejected by --append-only). If it succeeds, your wall isn't there — usually because the key you're using has no forced command, or rclone on the VPS wasn't started with --append-only. Then prove the master path still works: with the unrestricted key (plain SFTP), run a real prune and confirm delete rights exist where you want them.

2. Prove reads still work. Run the restore drill against this repo: restic check --read-data, then restore latest to scratch and byte-compare. That one proved you can read backups; together they prove you can read them and nobody-but-you-can-erase-them.

When it breaks//

1. Password loss = unrecoverable data. The restic docs warn about this twice (init output and the SFTP section): "Losing your password means that your data is irrecoverably lost." Keep RESTIC_PASSWORD_FILE in a secrets manager or keychain, not in a compose file — and know that both keys plus the password must survive for a restore. The append-only key protects you from attackers; it does nothing for you forgetting the password.
2. The #8958 seam: rclone's append-only is stricter than restic's official REST server. Issue #8958 (open, filed Nov 10 2025 by a long-time rclone contributor who works on the serve/restic code) documents the divergence precisely: rclone serve restic --append-only rejects all DELETEs and re-uploads of existing files, while the official REST server allows overwriting a file if its checksum matches — which is how restic safely retries after a partial upload. Consequence: on backends that leave partially-written files behind (some S3/Ceph setups under network congestion), rclone's stricter behavior can fail an otherwise-retryable backup with 403 Forbidden instead of recovering. It won't corrupt data — the issue thread confirms it's "just annoyance that might lead to backup failing" — but it means your alerting will see failures the official server would have swallowed. Pick one backend and test delete-with-old-key explicitly (Step 5 does this). If you're on a flaky network, watch for Save(...) failed: unexpected HTTP response (403) in restic output and treat it as "append-only did its job; the upload was retried against an existing partial."
3. You granted write+delete "for convenience." The moment the backup key can delete, immutability is gone — that's the whole point of the second key. If you find yourself pruning from the same box that backs up, move prune to a maintenance window with the master key (or do it by hand over SFTP), and keep the append-only key as the only thing on the live host. Rollback for all three: delete the authorized_keys line / revoke the app key — your normal backups resume immediately; nothing about this setup is permanent.

What you've built//

A $5 VPS (or a storage bucket) holding an encrypted restic repo that your compromised hosts can append to but not erase, enforced server-side by a forced command they can't see or edit, with a verification step that proves both halves — the wall and the read path. The restore drill proved your backups are real; this one proves they're yours.