Your Agent Job Exited 0. The SD Card Had Gone Read-Only Hours Ago.

· Updated

You check the Discord channel in the morning and nothing looks wrong. The uptime check is green. The Pi answers ping. The OpenClaw dashboard loads. Then you notice that yesterday’s post never showed up in the repo, the log file stops partway through last night, and the state file still holds a timestamp from two days ago. The cron job “ran.” Some steps even exited 0.

Very often the cause is that the root filesystem on the agent box remounted itself read-only. The kernel saw I/O errors from the SD card, or ext4 found an inconsistency, and it stopped allowing writes to protect what was left. Reads still work, so anything that only fetches a page or pings the host looks healthy. Anything that writes a log, commits to the working tree, rotates a file, or updates state fails. Depending on how your scripts handle errors, it fails loudly, quietly, or not at all.

If you searched for “Raspberry Pi SD card read-only,” this is the usual chain behind it. Below is how to confirm the remount, find out why it happened, recover without making the damage worse, and add a preflight that stops write jobs before they start.

The errors you’ll actually see

When root goes read-only, every write attempt returns the same kernel error: EROFS, errno 30. Tools print it in slightly different ways:

  • Shell and coreutils: touch: cannot touch 'state.json': Read-only file system
  • Git: error: unable to create file ...: Read-only file system or fatal: Unable to create '.../.git/index.lock': Read-only file system
  • Python: OSError: [Errno 30] Read-only file system: '/home/pi/agent/state.json'
  • Node: Error: EROFS: read-only file system, open '...'

If an agent job’s output contains any of these, skip straight to Step 1. The problem is the disk, not your code, your API key, or the model.

Why this failure hides so well

Most Raspberry Pi OS images mount root with errors=remount-ro. That is a sensible default: when ext4 hits a serious error, it stops writing so it doesn’t corrupt more data. On a Pi, the two most common triggers are a worn-out SD card returning I/O errors and a power problem that interrupts a write mid-flight. Either way, the kernel flips root to read-only and the box enters a confusing half-alive state:

  • Processes already in memory keep running.
  • curl against a local service still returns 200, because serving only needs reads.
  • /tmp and /run are often tmpfs, so writes there still succeed.
  • Scripts without set -e skip past a failed echo >> log.
  • Some tools catch the write error, warn to a log that can’t be written, and exit 0.

The scheduler and healthcheck both record success while the actual work stops.

Step 1: Confirm the filesystem is actually read-only

Start with the mount table:

mount | grep ' / '

Look for ro in the options. A healthy root shows rw,noatime,.... A remounted root shows ro,....

findmnt gives a cleaner answer and is easier to script:

findmnt -O ro
findmnt -n -o TARGET,SOURCE,OPTIONS /

The first command lists every mount that is currently read-only. On an agent box, the only entries should be things you deliberately made read-only, if any. If / or the partition holding your working tree shows up, you’ve found the problem.

Next, confirm with an actual write. Test the real paths your jobs use, and test /tmp alongside them so the contrast is visible:

touch /tmp/rw-test && echo "/tmp writable"
touch ~/agent/.rw-test && echo "working tree writable"

If /tmp works and the working tree returns Read-only file system, then /tmp is tmpfs and root has gone read-only. This is why a touch /tmp/... check alone is not a healthcheck. It can pass on a box that can’t save anything to disk.

Step 2: Find the cause in dmesg and the journal

Do not remount yet. First find out why the kernel stopped writing. The kernel ring buffer is the quickest place to look:

dmesg -T | grep -iE 'ext4|mmc|i/o error|remount|read-only' | tail -n 50

The lines to look for are:

  • EXT4-fs error (device mmcblk0p2): ... means ext4 found a problem with its own metadata.
  • Remounting filesystem read-only is the moment it flipped.
  • mmc0: timeout waiting for hardware interrupt or blk_update_request: I/O error, dev mmcblk0 means the SD card or its controller failed a read or write.
  • Buffer I/O error on dev mmcblk0p2 points to the same thing.

The journal has the same kernel messages with better context. When root is read-only, journald usually falls back to volatile storage in /run, so the current boot is still visible:

journalctl -k -b --no-pager | grep -iE 'ext4|mmc|i/o error|read-only'

If you already rebooted, check the previous boot (only useful if persistent journaling was on and written before the remount):

journalctl -k -b -1 --no-pager | tail -n 100

Two other checks often explain the chain. First, power:

vcgencmd get_throttled

Any nonzero value deserves attention. 0x50000 or 0x50005 means under-voltage happened, a common reason SD writes fail on a Pi. A weak supply plus nightly agent writes wears cards out quickly. The undervoltage and throttling guide covers how to read every bit of that value and what to fix.

Second, the ext4 error record:

sudo tune2fs -l /dev/mmcblk0p2 | grep -iE 'state|error'

Filesystem state: clean with errors along with an error count and a first-error timestamp tells you ext4 recorded the problem and when it began.

If you see repeated mmc I/O errors and no under-voltage, the card itself is the likely culprit. SD cards have a finite write life, and an agent that logs, commits, and rewrites state every few minutes can use it up faster than you’d expect. The write budget and SD wear post shows how to measure that.

Step 3: Get your data off before you fix anything

The box can still read, so use that while it lasts. Copy what matters to another machine before a remount or fsck: uncommitted drafts, recent logs, state files, and .env files.

rsync -a ~/agent/ user@other-host:/backup/agent-$(date +%F)/

If the card is failing, every extra write and reboot is a gamble. A backup from the read-only box is the cheapest insurance you’ll get.

Step 4: Recover safely

You have two paths, depending on what dmesg showed.

The cause was a one-off and the card looks fine. For example, there was a single error during a power blip and no repeated I/O errors since. You can try a temporary remount to finish urgent work:

sudo mount -o remount,rw /

ext4 may refuse if it recorded errors, and that refusal is the filesystem telling you to run fsck. Even if the remount succeeds, treat it as a short window to wrap up and back up, not as a fix. The filesystem still has an error recorded, and it should be checked before you trust it again.

The filesystem needs a real check. Never run fsck on a mounted root partition. You have two options.

The first is a forced check at the next boot. On current Raspberry Pi OS, add fsck.mode=force fsck.repair=yes to the single line in /boot/firmware/cmdline.txt, or /boot/cmdline.txt on older images. The boot partition is usually a separate FAT partition and may still be writable. Reboot, then remove those flags afterward.

The second, which is better if the card is suspect, is to work from another machine. A repair writes to the card, and a dying card may not survive many more writes, so image the whole card first.

Shut the Pi down, move the SD card to a Linux box with a reader, and find the device:

lsblk

Identify the whole card (for example sdX, with partitions sdX1 and sdX2 under it) by its size. Then copy the entire card to an image file:

sudo dd if=/dev/sdX of=/backups/pi-sd-$(date +%F).img bs=4M status=progress conv=fsync

Use the whole-card device here, not a partition. Double-check the name before you press Enter. Getting if= and of= wrong, or pointing at your own system disk, will destroy data.

With the image saved, run the filesystem check on the root partition:

sudo e2fsck -f /dev/sdX2

Replace sdX2 with the root partition lsblk actually shows. Double-check it here too.

If e2fsck reports many repaired inodes, or dmesg showed repeated mmc I/O errors, plan to replace the card. Fixing the filesystem on a dying card only buys you time. You can write the image (or a repaired copy) to a new high-endurance card:

sudo dd if=/backups/pi-sd-YYYY-MM-DD.img of=/dev/sdY bs=4M status=progress conv=fsync

Here sdY is the new card, confirmed with lsblk, and it must be at least as large as the old one. The other option is to move root to a USB or NVMe SSD and stop running the agent’s writes on SD at all.

Step 5: Add a preflight that fails closed

Detecting this after the fact is the expensive part. The fix is a preflight that every write job runs first. It should exit non-zero, and send an alert, when the target paths aren’t writable.

#!/usr/bin/env bash
# /usr/local/bin/agent-preflight.sh
set -euo pipefail

PATHS=("/" "$HOME/agent" "$HOME/agent/logs" "$HOME/agent/state")
WEBHOOK="${DISCORD_WEBHOOK_URL:-}"
MARKER="/run/agent-ro-alerted"   # tmpfs, writable even if root is ro

fail() {
  echo "PREFLIGHT FAIL: $1" >&2
  if [[ -n "$WEBHOOK" && ! -e "$MARKER" ]]; then
    curl -fsS -H 'Content-Type: application/json' \
      -d "{\"content\":\"agent box preflight failed on $(hostname): $1\"}" \
      "$WEBHOOK" >/dev/null || true
    touch "$MARKER" 2>/dev/null || true
  fi
  exit 1
}

for p in "${PATHS[@]}"; do
  opts=$(findmnt -n -o OPTIONS --target "$p") || fail "cannot resolve mount for $p"
  [[ ",$opts," == *",ro,"* ]] && fail "$p is mounted read-only ($opts)"
done

for p in "${PATHS[@]:1}"; do
  t="$p/.preflight-$$"
  { : > "$t" && rm -f "$t"; } 2>/dev/null || fail "write test failed in $p"
done

avail=$(df --output=avail -k "$HOME/agent" | tail -n1)
(( avail > 204800 )) || fail "less than 200MB free on agent volume"

rm -f "$MARKER" 2>/dev/null || true
echo "preflight ok"

A few details are doing real work here:

  • findmnt --target resolves the mount that actually contains each path, so it still works if you later move the working tree to a separate USB drive.
  • The write test runs in the real directories, not in /tmp.
  • The alert dedupe marker lives in /run, so it can still be created when root is read-only. You get one Discord alert per incident instead of one every five minutes.
  • A free-space check is included because a full disk produces the same “job ran, nothing saved” symptoms.

Then chain every write job behind it in cron:

*/30 * * * * /usr/local/bin/agent-preflight.sh && /home/pi/agent/run-publish.sh

With &&, the publish job never starts on a read-only box. The scheduler records a failure instead of a fake success, and you get a Discord alert that names the actual problem.

Add the same findmnt -O ro check to your external healthcheck so “box responds” becomes “box responds and can write.”

Make the next remount less likely

The preflight catches the problem. These reduce how often it happens:

  • Move hot writes off the boot SD. Booting from a USB or NVMe SSD is the biggest single improvement. If you stay on SD, keep /tmp and scratch files on tmpfs, and use log2ram or a tmpfs /var/log so constant log writes don’t grind the card.
  • Fix the power. Use an official-rated Pi power supply with a short, thick cable. Under-voltage during writes is one of the most common ways to end up with ext4 errors. See the undervoltage and throttling guide.
  • Measure how much your agent writes. The write budget and SD wear post covers how to see which jobs write the most and estimate how long a card will last.

Make this a standing routine

None of this is complicated, but it only helps if it runs before every write job, not after you notice a missing post. Grab one of the free prompts and adapt it into a “check whether the root filesystem is read-only before any write job, and alert if it is” routine. For ready-made preflights, recovery checklists, and the rest of the guardrails for scheduled agent jobs, the OpenClaw Agent Ops Toolkit collects them in one place. The Ultimate Bundle includes everything for $49.

More from the build log

Suggested

Want the full MarketMai stack?

Get the core MarketMai guides and operator playbooks in one premium bundle for $49.

View Bundle