Your Disk Has Room. Your Agent Still Can't Create One More File.
You check in on the box in the morning and the overnight job has failed. The log says No space left on device. Your first instinct is that the disk is full, so you run df -h. It shows 11G free. You delete a few old logs anyway, rerun the job, and it fails the same way. Now you start suspecting the SD card, the agent, or the last change you made.
The disk is not out of bytes. It is out of inodes. Every file and directory on an ext4 filesystem needs one inode, and when the inode table is used up, the kernel refuses new files with the same error it uses for a full disk.
This guide covers how to confirm it, find the directories holding huge numbers of tiny files, clean up without deleting the wrong things, and add a fail-closed preflight so the next job stops before it walks into the same wall.
The errors you’ll actually see
The kernel returns ENOSPC (errno 28) for both out-of-blocks and out-of-inodes. Every layer above it reports the same message, which is why df -h alone misleads you.
Python:
OSError: [Errno 28] No space left on device: '/home/pi/agent/runs/2026-11-18T03-00-02/transcript.json'
Node:
Error: ENOSPC: no space left on device, open '/home/pi/agent/workspace/.cache/session-8841.json'
Shell tools inside a job:
mkdir: cannot create directory 'runs/2026-11-18': No space left on device
touch: cannot touch 'heartbeat': No space left on device
A useful clue: appending to a file that already exists often still works, because it needs blocks, not a new inode. If logs keep growing but new run folders cannot be created, think inodes.
Rule out the cousins first
- The disk really is full of bytes.
df -hshowsUse%at or near 100%. That is a byte problem; see the cleanup budget post. Inode cleanup will not help there. - The filesystem went read-only. After SD card errors, ext4 can remount itself read-only. The error is
Read-only file system(EROFS), not “no space.” Checkfindmnt -no OPTIONS /forroand follow the read-only remount guide. - Too many open files.
EMFILE/Too many open filesis about file descriptors per process, governed byulimit -n. It has nothing to do with inodes. See the EMFILE post. - Disk quota. Quotas produce
Disk quota exceeded(EDQUOT). Most home Pi setups do not enable them;quota -swill tell you.
Why it hides
Operators run df -h because that is the command everyone remembers. It reports bytes only. Service healthchecks usually stay green too, because a running process can keep writing to files it already has open.
The cause is also counterintuitive. A few multi-gigabyte caches eat bytes. Inodes are eaten by sheer count: per-run directories that never get pruned, one transcript file per step, node_modules trees in every workspace, package caches, headless browser profiles, __pycache__ folders, and leftover job directories from runs that crashed before cleanup. A 4 KiB file costs the same one inode as a 4 GiB file.
ext4 commonly allocates one inode per 16 KiB of space, so a 32 GB card has roughly two million inodes. That is not plenty once an agent writes a few hundred files per run, several times an hour, for a year, plus several node_modules trees.
Step 1: Check the filesystem the agent actually writes to
Do not check only /. Find the mount behind the agent’s data directory, then look at bytes and inodes together.
AGENT_DIR=/home/pi/agent # change to your agent's data path
findmnt -T "$AGENT_DIR"
df -h "$AGENT_DIR"
df -i "$AGENT_DIR"
If IUse% is at or near 100% and IFree is near zero while df -h still looks comfortable, that is inode exhaustion. stat gives the same counts:
stat -f -c 'total_inodes=%c free_inodes=%d' "$AGENT_DIR"
Check /tmp and any bind mounts separately. /tmp may be a tmpfs with its own inode limit, and a bind-mounted workspace may live on a different disk:
df -i /tmp
findmnt -t ext4,tmpfs,vfat,btrfs
Step 2: Find the directories with huge file counts
du --inodes counts entries instead of bytes. Always use -x so it stays on one filesystem and does not wander into /proc, network mounts, or USB drives.
sudo du --inodes -x -d 3 "$AGENT_DIR" 2>/dev/null | sort -n | tail -n 25
To see which directories hold the most direct entries, group find output by parent, and keep -xdev:
sudo find "$AGENT_DIR" -xdev -printf '%h\n' 2>/dev/null \
| sort | uniq -c | sort -n | tail -n 25
A run folder with 300 files is normal. A runs/ directory with 40,000 subfolders, or 15 copies of node_modules, is your answer.
Deleted files still held open by a process keep their inodes until that process exits:
sudo lsof -nP +L1 2>/dev/null | head -n 20
If a long-running process is holding thousands of deleted files, restarting it frees them.
Step 3: Clean up without deleting the wrong things
Do not rm -rf a tree you do not recognize. List first, and clean only regenerable or expired categories.
List and count run folders older than 14 days:
find "$AGENT_DIR/runs" -xdev -mindepth 1 -maxdepth 1 -type d -mtime +14 -print | head
find "$AGENT_DIR/runs" -xdev -mindepth 1 -maxdepth 1 -type d -mtime +14 | wc -l
To keep those folders, move them to another filesystem, which returns the inodes without destroying the files:
mkdir -p /mnt/usb/agent-runs-archive
find "$AGENT_DIR/runs" -xdev -mindepth 1 -maxdepth 1 -type d -mtime +14 \
-exec mv -t /mnt/usb/agent-runs-archive -- {} +
If they can go, delete them:
find "$AGENT_DIR/runs" -xdev -mindepth 1 -maxdepth 1 -type d -mtime +14 \
-exec rm -rf -- {} +
Regenerable caches are safer targets. If you use pip or npm:
find "$AGENT_DIR" -xdev -type d -name __pycache__ -prune -exec rm -rf -- {} +
pip cache purge
npm cache clean --force
For stale node_modules, list them first and delete only those in workspaces you are retiring. They can be reinstalled from the lockfile.
find "$AGENT_DIR" -xdev -type d -name node_modules -prune -print
After cleanup, confirm IUse% actually dropped, not just bytes:
df -i "$AGENT_DIR"
On an existing ext4 filesystem, the inode count is fixed at creation time. tune2fs cannot raise it in place, so cleanup and prevention are the real fix. Recreating the filesystem with a different -i or -N is a last resort and out of scope here.
A one-time cleanup is not the whole fix. Give each recurring job a retention window, and have it delete or move its own scratch directory on exit, including on failure. That is cheaper than discovering 40,000 leftover folders the next time ENOSPC shows up. If you already cap how much an agent may write, to limit SD card wear, treat inode count as a second cap. Bytes and inodes fail independently.
Step 4: Add a fail-closed preflight
This script checks inode use and free count on the agent data mount, logs one clear line, alerts once per outage through a Discord webhook if one is configured, and exits 1 so cron can chain it with &&. If df cannot be read, it fails closed.
Save it as /usr/local/bin/agent-inode-preflight and chmod 755 it:
#!/usr/bin/env bash
# Fail closed if the agent data mount is low on inodes.
set -u
DATA_DIR="${AGENT_DATA_DIR:-/home/pi/agent}"
MAX_IUSE_PCT="${MAX_IUSE_PCT:-90}"
MIN_IFREE="${MIN_IFREE:-50000}"
MARKER_DIR="${MARKER_DIR:-/run/agent-preflight}"
MARKER="$MARKER_DIR/inode.alerted"
WEBHOOK="${DISCORD_WEBHOOK_URL:-}"
TAG="inode-preflight"
log() { logger -t "$TAG" -- "$1" 2>/dev/null; echo "$TAG: $1" >&2; }
notify() {
[ -n "$WEBHOOK" ] || return 0
safe="$(printf '%s' "$1" | tr -d '"\\')"
curl -fsS -m 10 -H 'Content-Type: application/json' \
-d "{\"content\":\"$safe\"}" "$WEBHOOK" >/dev/null 2>&1 \
|| log "webhook post failed"
}
fail() {
log "FAIL $1"
mkdir -p "$MARKER_DIR" 2>/dev/null || true
if [ ! -e "$MARKER" ]; then
notify "$(hostname): $TAG FAIL $1"
touch "$MARKER" 2>/dev/null || log "cannot write marker in $MARKER_DIR"
fi
exit 1
}
mkdir -p "$MARKER_DIR" 2>/dev/null || true
row="$(df -P -i -- "$DATA_DIR" 2>/dev/null | awk 'NR==2 {print $2, $4, $5, $6}')"
[ -n "$row" ] || fail "cannot read df -i for $DATA_DIR"
read -r total free pct mnt <<< "$row"
pct="${pct%\%}"
case "$total$free$pct" in
*[!0-9]*|'') fail "unparseable df output for $DATA_DIR: $row" ;;
esac
[ "$total" -gt 0 ] || fail "$mnt reports no inode table; point DATA_DIR at the real data mount"
if [ "$pct" -ge "$MAX_IUSE_PCT" ] || [ "$free" -lt "$MIN_IFREE" ]; then
fail "$mnt inodes ${pct}% used, ${free} free (limits ${MAX_IUSE_PCT}% / ${MIN_IFREE})"
fi
if [ -e "$MARKER" ]; then
rm -f "$MARKER"
notify "$(hostname): $TAG recovered, $mnt inodes ${pct}% used"
fi
log "OK $mnt inodes ${pct}% used, ${free} free"
exit 0
The webhook URL comes from the environment and is never printed. The marker lives under /run, so it clears on reboot and you get one alert per outage, not one per cron tick. If cron runs as a non-root user, create the marker directory at boot with a tmpfiles rule:
echo 'd /run/agent-preflight 0755 pi pi -' | sudo tee /etc/tmpfiles.d/agent-preflight.conf
sudo systemd-tmpfiles --create /etc/tmpfiles.d/agent-preflight.conf
Chain it in cron:
0 3 * * * AGENT_DATA_DIR=/home/pi/agent /usr/local/bin/agent-inode-preflight && /home/pi/agent/run-nightly.sh
If the preflight fails, the job never starts, and the failure shows up in the journal with one readable line. Pair it with a stalled automation alarm so a skipped run is noticed even when nobody reads the journal.
Add the same df -i check to your reboot and power-loss test. After boot, confirm the agent mount has inode headroom before you declare the box healthy.
Standing routine
- Every job: run the preflight first, chained with
&&. Have jobs delete their own scratch directories on exit, including on failure. - Daily: log
df -handdf -ifor the agent mount,/, and/tmpside by side. Watch the inode trend, not just the byte trend. - After cleanup: confirm
IUse%dropped. Bytes dropping is not proof that inodes came back. - After a deploy or new workspace: run the
du --inodes -xcheck once. Newnode_modulestrees and browser profiles are the usual surprise. - Monthly: prune or archive run folders past your retention window, and test the preflight by lowering
MIN_IFREEabove the current free count.
If you want the prompts and checklists we use to keep unattended agents honest, start with the free prompts, go deeper with the OpenClaw Agent Ops Toolkit, or get everything in the Ultimate Bundle.
More from the build log
Suggested
Want the full MarketMai stack?
Get the core MarketMai guides and operator playbooks in one premium bundle for $49.
View Bundle