Pull the Plug: The Reboot Test Your Self-Hosted Agent Has Never Passed
Most self-hosted agent boxes have been rebooted on purpose maybe twice: once during setup, and once when something was already broken. Every other restart was an accident. The power blinked during a storm. An unattended update pulled in a new kernel. Someone unplugged the Pi to plug in a phone charger.
The scary part is not that the box went down. It is that it came back up, the green light came on, and you assumed everything else came back with it.
Often it does not. The agent runtime starts before the network is ready and gives up. The cron jobs run, but they call a service that is still starting. The external drive with the agent’s memory mounts a minute late, so the agent starts with an empty workspace and happily writes new state on top of nothing. None of this throws an alert, because the thing that would send the alert is one of the things that did not come back.
You can find all of this in about 20 minutes, on a quiet afternoon, by rebooting on purpose. This post is that test.
Why “it survived a reboot once” proves nothing
A box that survived one restart during setup was in a very different state than it is today. Since then you have probably:
- added a second agent, a webhook listener, or a local model server
- moved logs or memory to a USB drive or NAS share
- started a process by hand in a
tmuxorscreensession “just for now” - added a cron job that depends on another service being up
- changed a password, token, or SSH key that something reads at startup
Each of those is a new way to fail at boot. The hand-started process is the classic one. It has been running for six weeks, it is doing real work, and it exists only in a terminal session that will vanish on the next restart. Nothing in your system knows it is supposed to exist.
What “came back” actually means
Before you pull anything, write down what a healthy box looks like. Not “the agent is running.” Be specific enough that you could check each line in under a minute:
- Every long-running process is up. The agent runtime, any gateway or webhook listener, a local model server like Ollama, a database, a reverse proxy or tunnel.
- Every mount is in place. External drives, network shares, anything the agent reads memory or config from.
- The network path works end to end. Not just “the box has an IP,” but “the agent can reach its model provider, and inbound webhooks reach the agent.”
- Scheduled jobs are registered. Your crontab and systemd timers exist, and the next run time is where you expect.
- The agent has its memory. It can answer a question that requires yesterday’s notes or state, not just a fresh “hello.”
- The alert path works. If something on this list were broken, you would hear about it.
That list is your pass/fail sheet for the test. If you cannot write it, that is the first finding.
The 20-minute reboot test
Step 1: Take a “before” snapshot (5 minutes)
Capture what is running right now so you have something to compare against:
systemctl list-units --type=service --state=runningfor system servicessystemctl --user list-units --type=serviceif you run services as your userps auxfiltered to your agent, model server, and anything you started by handfindmntordf -hfor mountscrontab -landsystemctl list-timers --alldocker psorpodman psif you use containers
Save the output to a file. You are going to diff against it.
Look at the process list closely. Anything running under tmux, screen, nohup, or a terminal you opened over SSH is a process with no plan for coming back. Mark each one.
Step 2: Reboot the boring way (2 minutes)
Run sudo reboot. Do not stop services first, do not prepare anything. You want the same restart an unattended update would give you.
If you are brave and the hardware tolerates it, a real power pull is the more honest test, because it also catches unclean shutdown problems like a corrupted SD card or a database that needs recovery. Do the clean reboot first. Save the power pull for after you have fixed what the clean one finds.
Step 3: Wait, then do not touch anything (5 minutes)
Give the box five full minutes. Some services legitimately take a while, and you want to see the state an unattended box settles into, not the state it is in at 30 seconds.
Then log in and check your list without starting anything by hand. The moment you run a command to “just get it going,” you have stopped testing and started covering up the finding.
Step 4: Diff and write it down (8 minutes)
Run the same commands from Step 1 and compare. For every difference, write one line: what is missing, and why you think it is missing. Then check the six-line health list, especially the last two. Ask the agent a question that needs yesterday’s state. Break something small on purpose, like stopping one service, and see whether you hear about it.
The usual suspects and their fixes
Most failures at boot fall into a short list.
Started by hand. A process that lives in a terminal session needs a real service definition. A small systemd unit with Restart=on-failure and WantedBy=multi-user.target (or default.target for a user service) is usually enough. For user services, run loginctl enable-linger <user> so they start at boot without anyone logging in. That one command is the fix for a surprising number of “it only works after I SSH in” problems.
Starts before the network. If the agent or a tunnel starts, fails to reach anything, and exits, add After=network-online.target and Wants=network-online.target to its unit, and make sure it restarts on failure with a short delay instead of giving up. A service that retries for two minutes is better than one that tries once at second four.
Starts before its mount. If memory or config lives on an external drive, the service should depend on that mount. RequiresMountsFor=/path/to/data in the unit stops the agent from starting against an empty directory. In /etc/fstab, use the drive’s UUID instead of /dev/sda1, since device names can change between boots, and consider nofail so a missing drive does not block the whole boot.
Containers that do not restart. A container started with a plain docker run stays stopped after a reboot. Set a restart policy such as --restart unless-stopped, or the equivalent in your compose file, and make sure the container engine itself is enabled at boot.
Cron jobs that race the services. A job scheduled with @reboot or one that happens to fire right after startup can run before the thing it calls is ready. Either have the job check that its dependency answers before doing work, or move it to a systemd timer that is ordered after the service.
The agent forgot who it is. If the agent comes back with no memory, look for state written to /tmp or another path that is cleared on boot, and for environment variables that were exported in a shell instead of written to a file the service actually loads.
The alarm was on the same box. If the only thing watching your agent runs on the agent’s box, a dead box is a silent box. At least one check should live somewhere else, such as a hosted uptime ping or a scheduled job on another machine, that expects to hear from your agent and complains when it does not. That is the whole idea behind a stalled automation alarm, and a reboot is the cheapest way to prove yours works.
Run it again, then put it on the calendar
After you fix what you found, reboot again and rerun the checklist. The test passes when you can restart the box, walk away, come back five minutes later, and find every line on the health list true without touching anything.
Then make it routine. A reboot test once a month, or after any change that adds a service, mount, or scheduled job, takes 20 minutes and keeps the list honest. It pairs naturally with two other habits: a config drift snapshot, which tells you what changed since the last time things worked, and a restore drill, which tells you whether you could rebuild the box if the reboot never finished at all.
Together they answer the three questions every self-hosted operator eventually gets asked by an outage: what changed, does it come back on its own, and could we rebuild it if it did not.
The short version
- Write down what a healthy box looks like, in six checkable lines.
- Snapshot everything that is running.
- Reboot, wait five minutes, touch nothing.
- Diff against the snapshot and fix each gap with a real service, a dependency, or a restart policy.
- Make sure at least one alarm lives off the box.
- Repeat until a reboot is boring.
A self-hosted agent that only works while you keep it alive by hand is a hobby. One that comes back on its own after a power blink is infrastructure.
Turn the checklist into a prompt
If you want your agent to run the “before” snapshot and the diff for you, start with MarketMai’s free AI prompt pack and adapt one into a “list every service, mount, and scheduled job, then compare against last month” check. When you are ready to run more of your business on agents, the MarketMai Ultimate Bundle puts 13 prompt, automation, and agent-operations products in one $49 download.
More from the build log
Suggested
Want the full MarketMai stack?
Get the core MarketMai guides and operator playbooks in one premium bundle for $49.
View Bundle