A Backup You Have Never Restored Is a Guess: The Self-Hosted Agent Restore Drill

Ask a self-hosted agent operator whether they have backups and the answer is almost always yes. Ask when they last restored one onto a clean machine and the room goes quiet.

A backup you have never restored is not a backup. It is a guess with a cron job attached. The failure modes are boring and brutal: the archive is missing the one file that holds your credentials, the database dump is from before the schema changed, the restore script assumes a path that only exists on the old box. You discover all of this at the worst possible moment, with the agent down and clients waiting.

The fix is a restore drill. It takes about thirty minutes, and you only need to do it once a quarter.

What an agent actually needs to come back

A self-hosted agent is more than its code. When it dies, these are the pieces you need to get back:

• Config: the main settings file, channel bindings, model routing, scheduled jobs.
• Secrets: API keys, tokens, webhook secrets. Usually in an env file that nobody remembered to include.
• Memory and state: notes, summaries, databases, task queues, anything the agent learned.
• Workspace files: prompts, skills, templates, project folders.
• Service definition: the systemd unit or equivalent that starts it all.
• Scheduler state: cron entries and timers that make it act without being asked.

Write that list down for your own setup. If you cannot enumerate it, you cannot back it up completely, and the drill will show you exactly where the holes are.

The drill, step by step

1. Pick a clean target

Use a spare machine, a fresh VM, or a new user account on a test box. The point is that nothing from the original installation can quietly rescue you. A restore that works only because the old files are still lying around has proven nothing.

2. Start the clock

Note the time. Recovery time is a number you should know before an outage, not after. If your honest answer to “how long to get back online” is “uh, a few hours?”, measure it.

3. Restore from the backup only

Pull the most recent backup from wherever it actually lives, whether that is an external drive, object storage, or another host. Do not touch the live system for hints. Follow only what is written in your recovery notes. If you find yourself improvising, write down what you improvised, because that is a missing step.

4. Reinstall and rewire

Install the agent software, drop in the config, secrets, and state, and register the service. Every manual fix you make along the way goes into the notes.

5. Prove it works with a real task

Starting without errors is not enough. Run a small, real task that touches the important pieces:

• Can it read its memory and reference something it learned last week?
• Can it call the model provider with the restored key?
• Can it send a test message through its channel?
• Does a scheduled job fire on time?

If any of these fail, the backup is incomplete. That is the whole value of the drill.

6. Stop the clock and record the result

Write down total time, every gap you found, and what you changed to close it. Then fix the backup job so the next archive includes whatever was missing.

The gaps you will probably find

After running drills on a handful of setups, the same findings come up again and again.

Secrets live outside the backup. The env file sits in a hidden directory the backup job skips. Without it, the restored agent boots and then fails every authenticated call. Decide on purpose where secrets are stored and how they are recovered, ideally encrypted, and never in the same place as an unencrypted archive.

State was captured mid-write. Copying a live database file can produce a corrupt copy. Use the database’s own dump or snapshot command, or stop the writer briefly.

Hard-coded paths and hostnames. Scripts that reference one specific home directory or machine name break on a new box. Better to find that during a drill than during an incident.

Scheduler state was forgotten. The agent restores fine but nothing runs on time, because cron entries lived in a user crontab nobody exported. A quiet agent looks healthy right up until you notice nothing happened for three days.

Backups are stale. The job failed silently six weeks ago and has been writing nothing since. Check the timestamp on the newest archive before you trust the process.

Only one copy, in one place. A backup on the same disk as the agent dies with the agent. Keep at least one copy somewhere that survives the loss of the machine.

Make the drill cheap enough to repeat

A drill nobody runs is worth as much as an untested backup. Keep it small:

1. Write a one-page recovery note. Ordered steps, exact commands, where each backup lives. Assume the reader is you at 2am, stressed, with no memory of how this works.
2. Script the boring parts. A restore script that rebuilds directories and registers the service removes most human error.
3. Put it on the calendar. Quarterly is plenty. Tie it to something you already do, like a major upgrade.
4. Track the time. If recovery takes ninety minutes today and forty next quarter, you are getting more resilient. If it creeps up, you added complexity without adding recovery.
5. Add a freshness alert. A simple check that warns you when the newest backup is older than a day catches silent failures between drills.

Why this matters more for agents than for most software

A normal web app is mostly code you can redeploy from a repository. An agent is different: a lot of its value is accumulated state, such as the memory it built, the preferences it learned, the jobs you tuned over months. Redeploying the code gets you a blank agent. The thing you actually depend on is the part most likely to be missing from a sloppy backup.

If you run an agent for clients, this is also a trust question. “We back up nightly” is a claim. “We restored to a clean machine last month in forty minutes and here is the log” is evidence. The second sentence wins contracts and keeps them.

The thirty-minute version

If you do nothing else this week:

1. List the six categories above and where each lives in your setup.
2. Confirm the newest backup is recent and contains all six.
3. Restore it to a clean spot and run one real task.
4. Write down what broke and fix the backup job.

You will almost certainly find at least one gap. Good. That gap was going to find you eventually, and it is much cheaper to meet it on a quiet afternoon than in the middle of an outage.

Backups are a promise. A restore drill is how you find out whether you can keep it.