Your Workspace Path Exists. The Disk Behind It Isn't Mounted.

You check in after a reboot, or after the power blinked overnight. The agent service is green. ls /mnt/agent-data works, so the workspace path exists. But last night’s run artifacts are gone. Or a job logged that it wrote its output successfully, and an hour later, once the USB disk finished coming online, those files vanished.

Nothing was deleted. The agent wrote into an empty directory on the SD card, and then the real disk mounted on top of it and hid everything underneath.

This guide covers how to confirm the path is not on the disk you intended, stop writing to the empty mountpoint, recover the hidden files, fix fstab and the systemd unit, and add a fail-closed preflight so the next job does not run against the wrong filesystem.

The errors you’ll actually see

  • The workspace is empty after a reboot, even though the disk has data on it when you check later.
  • df -hP /mnt/agent-data reports /dev/mmcblk0p2 (the SD card root) instead of /dev/sda1 or your SSD.
  • findmnt /mnt/agent-data prints nothing and exits non-zero.
  • A job created files, a status check confirmed them, and after the disk mounted they were gone.
  • No such file or directory appears only after a remount, for paths a previous step just wrote.
  • A disk-space alarm fires on the SD card while the data disk shows plenty of room.

Rule out the cousins first

  • A read-only remount. If writes fail with Read-only file system, the SD card is likely failing and the kernel remounted it read-only. That is a different problem, covered in the read-only remount post.
  • A restore or backup that looks empty. If you pointed a backup or restore at the mountpoint while the disk was down, you captured or checked an empty directory. The restore drill post covers proving a backup has real contents.
  • A path typo or a sandboxed path. If the agent runs in a container, chroot, or sandbox, its /mnt/agent-data may be a different directory from the host’s. Compare the configured path with the one you inspect, and check the host side of any bind mount.

Why it hides

A mountpoint is just a directory. If nothing is mounted there, writes land on whatever filesystem holds that directory, usually the SD card root. When the real disk mounts later, it covers that directory. The earlier files are hidden, not deleted.

Several common habits keep this invisible:

  • Scripts test [ -d "$DATA_DIR" ] or run ls. The mountpoint directory always exists, so both pass.
  • A systemd unit without RequiresMountsFor= can start as soon as local-fs.target is reached, even if an optional mount has not finished.
  • fstab nofail lets the box boot when the USB disk is slow or missing. That keeps a headless Pi reachable, but it also means nothing stops services from starting against the empty directory.
  • USB disks can enumerate late, especially behind hubs or after a brownout, so the agent can already be writing by the time the disk appears.

Step 1: Confirm which filesystem the path is on

DATA_DIR=/mnt/agent-data

findmnt -n -o SOURCE,TARGET,FSTYPE,UUID "$DATA_DIR" || echo "NOT a mount: $DATA_DIR"
mountpoint -q "$DATA_DIR" && echo "mountpoint: yes" || echo "mountpoint: no"
df -hP "$DATA_DIR"

echo "root source: $(findmnt -n -o SOURCE /)"
lsblk -o NAME,UUID,SIZE,FSTYPE,MOUNTPOINT

If findmnt prints nothing, mountpoint says no, or df shows the same source as /, the agent has been writing to the root filesystem.

Write down the data disk’s UUID from lsblk or sudo blkid and treat it as your expected UUID. Then, once you have confirmed the right disk is mounted, create a canary file on it:

EXPECTED_UUID=your-disk-uuid-here
[ "$(findmnt -n -o UUID --mountpoint "$DATA_DIR")" = "$EXPECTED_UUID" ] \
  && sudo sh -c "echo 'agent data disk canary' > '$DATA_DIR/.agent-disk-canary'"

The canary lives on the data disk, not on the SD card, so the empty mountpoint will always fail a check for it.

Step 2: Stop writes, then recover the hidden files

First, stop anything that writes to the workspace, so nothing new lands in the wrong place:

sudo systemctl stop openclaw-agent.service
# Pause any cron job or systemd timer that writes to $DATA_DIR. Do not wipe the crontab.

If the real disk is now mounted and files from the gap are missing, they are sitting underneath it on the root filesystem. A plain bind mount of / is non-recursive, so it shows the root filesystem without the data disk on top:

sudo mkdir -p /tmp/mnt-root
sudo mount --bind / /tmp/mnt-root
sudo ls -la "/tmp/mnt-root$DATA_DIR"

Copy those files somewhere separate before you clean anything:

RESCUE="$DATA_DIR/recovered-from-root-$(date +%F)"
sudo mkdir -p "$RESCUE"
sudo rsync -a "/tmp/mnt-root$DATA_DIR/" "$RESCUE/"
sudo ls -la "$RESCUE"
sudo umount /tmp/mnt-root

Merge the recovered files into the workspace by hand once you know which copy is newer. Do not rm -rf the mountpoint, and do not delete anything under /tmp/mnt-root while the bind is active. A recursive delete against the wrong tree is the fastest way to lose the real data disk. Leave stray files on the SD card and clean them later, one directory at a time.

Step 3: Fix fstab and the systemd unit

Mount by UUID, not by /dev/sda1. Device names can change when a second USB device is plugged in.

sudo cp /etc/fstab /etc/fstab.bak

A typical line:

UUID=your-disk-uuid-here  /mnt/agent-data  ext4  defaults,noatime,nofail,x-systemd.device-timeout=30s  0  2

The nofail tradeoff is real. With it, a missing or slow disk does not drop a headless Pi into emergency mode, so you can still reach it over SSH. Without a hard dependency, though, services will happily start against the empty directory. x-systemd.device-timeout=30s caps how long boot waits for the disk so a missing drive does not hang startup. Test the file with sudo findmnt --verify and sudo mount -a before you reboot.

Then give the agent service a hard dependency on the mount. Find the mount unit name:

systemd-escape -p --suffix=mount /mnt/agent-data
# mnt-agent\x2ddata.mount

Create a drop-in with sudo systemctl edit openclaw-agent.service:

[Unit]
RequiresMountsFor=/mnt/agent-data
After=mnt-agent\x2ddata.mount

[Service]
Environment=DATA_DIR=/mnt/agent-data
Environment=EXPECTED_UUID=your-disk-uuid-here
ExecStartPre=/usr/local/bin/agent-disk-preflight

RequiresMountsFor= tells systemd to start the service only after that path is mounted. Keep the preflight anyway: cron jobs, manual runs, and anything started outside this unit never see that dependency, and a failed optional mount can still leave the directory bare.

Step 4: Add a fail-closed preflight

Save this as /usr/local/bin/agent-disk-preflight and run sudo chmod +x on it. It passes only if every check can be verified; anything it cannot confirm counts as a failure.

#!/usr/bin/env bash
set -u

DATA_DIR="${DATA_DIR:-/mnt/agent-data}"
EXPECTED_UUID="${EXPECTED_UUID:-}"
CANARY="$DATA_DIR/.agent-disk-canary"
MARKER="${XDG_RUNTIME_DIR:-/run}/agent-disk-preflight.alerted"

fail() {
  local msg="agent-disk-preflight FAIL on $(hostname): $*"
  logger -t agent-disk-preflight "$msg" 2>/dev/null
  echo "$msg" >&2
  if [ ! -e "$MARKER" ] && [ -n "${DISCORD_WEBHOOK_URL:-}" ]; then
    curl -fsS -m 10 -H 'Content-Type: application/json' \
      -d "{\"content\":\"$msg\"}" "$DISCORD_WEBHOOK_URL" >/dev/null 2>&1 \
      && touch "$MARKER"
  fi
  exit 1
}

command -v findmnt >/dev/null 2>&1 || fail "findmnt not available"
[ -n "$EXPECTED_UUID" ] || fail "EXPECTED_UUID is not set"
mountpoint -q "$DATA_DIR" || fail "$DATA_DIR is not a mountpoint"

src="$(findmnt -n -o SOURCE --mountpoint "$DATA_DIR")" || fail "findmnt cannot read $DATA_DIR"
root_src="$(findmnt -n -o SOURCE /)" || fail "findmnt cannot read /"
[ -n "$src" ] && [ "$src" != "$root_src" ] || fail "$DATA_DIR source matches root ($root_src)"

uuid="$(findmnt -n -o UUID --mountpoint "$DATA_DIR")"
[ -n "$uuid" ] || fail "cannot read UUID for $DATA_DIR"
[ "$uuid" = "$EXPECTED_UUID" ] || fail "UUID mismatch on $DATA_DIR (got $uuid)"

[ -f "$CANARY" ] || fail "canary missing at $CANARY"

rm -f "$MARKER"
echo "agent-disk-preflight OK: $DATA_DIR on $src"

It logs one clear line, alerts once per outage if DISCORD_WEBHOOK_URL is set in the environment, and never prints the webhook. The marker lives in /run (or your user runtime directory), which is cleared on reboot, and a passing run removes it so the next outage alerts again. The exit code of 1 is what makes it chainable.

For cron, keep the webhook in a root-readable env file rather than in the crontab line:

*/30 * * * * . /etc/agent/preflight.env && /usr/local/bin/agent-disk-preflight && /usr/local/bin/run-agent-job

For systemd, the ExecStartPre= in the drop-in above already blocks the service; add EnvironmentFile=/etc/agent/preflight.env under [Service] if you want the Discord alert from that path too. If jobs stop running because the preflight keeps failing, a stalled automation alarm tells you that the pipeline went quiet, not just that one check failed.

Standing routine

  • Every job: run the preflight first and chain the job with &&.
  • After every reboot or power blip: run findmnt "$DATA_DIR" and the preflight before trusting the workspace. Add this mount check to your reboot and power-loss test so a late disk shows up during a drill instead of after a lost run.
  • After adding or swapping a disk: recheck lsblk -o NAME,UUID,SIZE,MOUNTPOINT, update EXPECTED_UUID and fstab, and recreate the canary on the new disk.
  • Monthly: confirm the UUID in fstab, the unit, and the preflight env all match, and look under a bind mount of / for stray files hiding beneath the mountpoint.

Keep the agent on the right disk

An existing directory proves nothing about which disk you are writing to. A mountpoint check, a UUID match, and a canary file do. If you want the prompts and checklists we use to keep unattended agents honest, start with the free prompts, go deeper with the OpenClaw Agent Ops Toolkit, or get everything in the Ultimate Bundle.

More from the build log

Suggested

Want the full MarketMai stack?

Get the core MarketMai guides and operator playbooks in one premium bundle for $49.

View Bundle