Your Workspace Path Exists. The Disk Behind It Isn't Mounted.
You check in after a reboot, or after the power blinked overnight. The agent service is green. ls /mnt/agent-data works, so the workspace path exists. But last night’s run artifacts are gone. Or a job logged that it wrote its output successfully, and an hour later, once the USB disk finished coming online, those files vanished.
Nothing was deleted. The agent wrote into an empty directory on the SD card, and then the real disk mounted on top of it and hid everything underneath.
This guide covers how to confirm the path is not on the disk you intended, stop writing to the empty mountpoint, recover the hidden files, fix fstab and the systemd unit, and add a fail-closed preflight so the next job does not run against the wrong filesystem.
The errors you’ll actually see
- The workspace is empty after a reboot, even though the disk has data on it when you check later.
df -hP /mnt/agent-datareports/dev/mmcblk0p2(the SD card root) instead of/dev/sda1or your SSD.findmnt /mnt/agent-dataprints nothing and exits non-zero.- A job created files, a status check confirmed them, and after the disk mounted they were gone.
No such file or directoryappears only after a remount, for paths a previous step just wrote.- A disk-space alarm fires on the SD card while the data disk shows plenty of room.
Rule out the cousins first
- A read-only remount. If writes fail with
Read-only file system, the SD card is likely failing and the kernel remounted it read-only. That is a different problem, covered in the read-only remount post. - A restore or backup that looks empty. If you pointed a backup or restore at the mountpoint while the disk was down, you captured or checked an empty directory. The restore drill post covers proving a backup has real contents.
- A path typo or a sandboxed path. If the agent runs in a container, chroot, or sandbox, its
/mnt/agent-datamay be a different directory from the host’s. Compare the configured path with the one you inspect, and check the host side of any bind mount.
Why it hides
A mountpoint is just a directory. If nothing is mounted there, writes land on whatever filesystem holds that directory, usually the SD card root. When the real disk mounts later, it covers that directory. The earlier files are hidden, not deleted.
Several common habits keep this invisible:
- Scripts test
[ -d "$DATA_DIR" ]or runls. The mountpoint directory always exists, so both pass. - A systemd unit without
RequiresMountsFor=can start as soon aslocal-fs.targetis reached, even if an optional mount has not finished. - fstab
nofaillets the box boot when the USB disk is slow or missing. That keeps a headless Pi reachable, but it also means nothing stops services from starting against the empty directory. - USB disks can enumerate late, especially behind hubs or after a brownout, so the agent can already be writing by the time the disk appears.
Step 1: Confirm which filesystem the path is on
DATA_DIR=/mnt/agent-data
findmnt -n -o SOURCE,TARGET,FSTYPE,UUID "$DATA_DIR" || echo "NOT a mount: $DATA_DIR"
mountpoint -q "$DATA_DIR" && echo "mountpoint: yes" || echo "mountpoint: no"
df -hP "$DATA_DIR"
echo "root source: $(findmnt -n -o SOURCE /)"
lsblk -o NAME,UUID,SIZE,FSTYPE,MOUNTPOINT
If findmnt prints nothing, mountpoint says no, or df shows the same source as /, the agent has been writing to the root filesystem.
Write down the data disk’s UUID from lsblk or sudo blkid and treat it as your expected UUID. Then, once you have confirmed the right disk is mounted, create a canary file on it:
EXPECTED_UUID=your-disk-uuid-here
[ "$(findmnt -n -o UUID --mountpoint "$DATA_DIR")" = "$EXPECTED_UUID" ] \
&& sudo sh -c "echo 'agent data disk canary' > '$DATA_DIR/.agent-disk-canary'"
The canary lives on the data disk, not on the SD card, so the empty mountpoint will always fail a check for it.
Step 2: Stop writes, then recover the hidden files
First, stop anything that writes to the workspace, so nothing new lands in the wrong place:
sudo systemctl stop openclaw-agent.service
# Pause any cron job or systemd timer that writes to $DATA_DIR. Do not wipe the crontab.
If the real disk is now mounted and files from the gap are missing, they are sitting underneath it on the root filesystem. A plain bind mount of / is non-recursive, so it shows the root filesystem without the data disk on top:
sudo mkdir -p /tmp/mnt-root
sudo mount --bind / /tmp/mnt-root
sudo ls -la "/tmp/mnt-root$DATA_DIR"
Copy those files somewhere separate before you clean anything:
RESCUE="$DATA_DIR/recovered-from-root-$(date +%F)"
sudo mkdir -p "$RESCUE"
sudo rsync -a "/tmp/mnt-root$DATA_DIR/" "$RESCUE/"
sudo ls -la "$RESCUE"
sudo umount /tmp/mnt-root
Merge the recovered files into the workspace by hand once you know which copy is newer. Do not rm -rf the mountpoint, and do not delete anything under /tmp/mnt-root while the bind is active. A recursive delete against the wrong tree is the fastest way to lose the real data disk. Leave stray files on the SD card and clean them later, one directory at a time.
Step 3: Fix fstab and the systemd unit
Mount by UUID, not by /dev/sda1. Device names can change when a second USB device is plugged in.
sudo cp /etc/fstab /etc/fstab.bak
A typical line:
UUID=your-disk-uuid-here /mnt/agent-data ext4 defaults,noatime,nofail,x-systemd.device-timeout=30s 0 2
The nofail tradeoff is real. With it, a missing or slow disk does not drop a headless Pi into emergency mode, so you can still reach it over SSH. Without a hard dependency, though, services will happily start against the empty directory. x-systemd.device-timeout=30s caps how long boot waits for the disk so a missing drive does not hang startup. Test the file with sudo findmnt --verify and sudo mount -a before you reboot.
Then give the agent service a hard dependency on the mount. Find the mount unit name:
systemd-escape -p --suffix=mount /mnt/agent-data
# mnt-agent\x2ddata.mount
Create a drop-in with sudo systemctl edit openclaw-agent.service:
[Unit]
RequiresMountsFor=/mnt/agent-data
After=mnt-agent\x2ddata.mount
[Service]
Environment=DATA_DIR=/mnt/agent-data
Environment=EXPECTED_UUID=your-disk-uuid-here
ExecStartPre=/usr/local/bin/agent-disk-preflight
RequiresMountsFor= tells systemd to start the service only after that path is mounted. Keep the preflight anyway: cron jobs, manual runs, and anything started outside this unit never see that dependency, and a failed optional mount can still leave the directory bare.
Step 4: Add a fail-closed preflight
Save this as /usr/local/bin/agent-disk-preflight and run sudo chmod +x on it. It passes only if every check can be verified; anything it cannot confirm counts as a failure.
#!/usr/bin/env bash
set -u
DATA_DIR="${DATA_DIR:-/mnt/agent-data}"
EXPECTED_UUID="${EXPECTED_UUID:-}"
CANARY="$DATA_DIR/.agent-disk-canary"
MARKER="${XDG_RUNTIME_DIR:-/run}/agent-disk-preflight.alerted"
fail() {
local msg="agent-disk-preflight FAIL on $(hostname): $*"
logger -t agent-disk-preflight "$msg" 2>/dev/null
echo "$msg" >&2
if [ ! -e "$MARKER" ] && [ -n "${DISCORD_WEBHOOK_URL:-}" ]; then
curl -fsS -m 10 -H 'Content-Type: application/json' \
-d "{\"content\":\"$msg\"}" "$DISCORD_WEBHOOK_URL" >/dev/null 2>&1 \
&& touch "$MARKER"
fi
exit 1
}
command -v findmnt >/dev/null 2>&1 || fail "findmnt not available"
[ -n "$EXPECTED_UUID" ] || fail "EXPECTED_UUID is not set"
mountpoint -q "$DATA_DIR" || fail "$DATA_DIR is not a mountpoint"
src="$(findmnt -n -o SOURCE --mountpoint "$DATA_DIR")" || fail "findmnt cannot read $DATA_DIR"
root_src="$(findmnt -n -o SOURCE /)" || fail "findmnt cannot read /"
[ -n "$src" ] && [ "$src" != "$root_src" ] || fail "$DATA_DIR source matches root ($root_src)"
uuid="$(findmnt -n -o UUID --mountpoint "$DATA_DIR")"
[ -n "$uuid" ] || fail "cannot read UUID for $DATA_DIR"
[ "$uuid" = "$EXPECTED_UUID" ] || fail "UUID mismatch on $DATA_DIR (got $uuid)"
[ -f "$CANARY" ] || fail "canary missing at $CANARY"
rm -f "$MARKER"
echo "agent-disk-preflight OK: $DATA_DIR on $src"
It logs one clear line, alerts once per outage if DISCORD_WEBHOOK_URL is set in the environment, and never prints the webhook. The marker lives in /run (or your user runtime directory), which is cleared on reboot, and a passing run removes it so the next outage alerts again. The exit code of 1 is what makes it chainable.
For cron, keep the webhook in a root-readable env file rather than in the crontab line:
*/30 * * * * . /etc/agent/preflight.env && /usr/local/bin/agent-disk-preflight && /usr/local/bin/run-agent-job
For systemd, the ExecStartPre= in the drop-in above already blocks the service; add EnvironmentFile=/etc/agent/preflight.env under [Service] if you want the Discord alert from that path too. If jobs stop running because the preflight keeps failing, a stalled automation alarm tells you that the pipeline went quiet, not just that one check failed.
Standing routine
- Every job: run the preflight first and chain the job with
&&. - After every reboot or power blip: run
findmnt "$DATA_DIR"and the preflight before trusting the workspace. Add this mount check to your reboot and power-loss test so a late disk shows up during a drill instead of after a lost run. - After adding or swapping a disk: recheck
lsblk -o NAME,UUID,SIZE,MOUNTPOINT, updateEXPECTED_UUIDand fstab, and recreate the canary on the new disk. - Monthly: confirm the UUID in fstab, the unit, and the preflight env all match, and look under a bind mount of
/for stray files hiding beneath the mountpoint.
Keep the agent on the right disk
An existing directory proves nothing about which disk you are writing to. A mountpoint check, a UUID match, and a canary file do. If you want the prompts and checklists we use to keep unattended agents honest, start with the free prompts, go deeper with the OpenClaw Agent Ops Toolkit, or get everything in the Ultimate Bundle.
More from the build log
Suggested
Want the full MarketMai stack?
Get the core MarketMai guides and operator playbooks in one premium bundle for $49.
View Bundle