Your Agent Service Is Green. It Just Can't Open One More File.

You check the channel in the morning and it’s quiet. Not “everything went fine” quiet. Just quiet.

systemctl status says active (running). The healthcheck is green. Cron fired on schedule. Then you scroll the journal and find a short line, repeated every few minutes since about 3 a.m.:

OSError: [Errno 24] Too many open files

Nothing crashed. Nothing restarted. Your agent is alive, holding a connection open, and completely unable to open anything new. It ran out of file descriptors.

This guide covers how to confirm that’s what happened, find what’s eating the descriptors, fix it without just moving the cliff further away, and add a preflight that fails closed before your next job walks into it.

The errors you’ll actually see

Errno 24 is EMFILE. Depending on your stack, it shows up as:

OSError: [Errno 24] Too many open files
EMFILE: too many open files
Error: EMFILE: too many open files, open '/var/lib/agent/runs/2026-11-15/output.json'

It also appears on sockets. A Python server’s socket.accept() raises the same OSError: [Errno 24], and outbound connect() calls to GitHub or a model API fail the same way, because every socket is a file descriptor. Spawning a subprocess fails too, since the pipes for stdout and stderr need descriptors.

The worst variant is the one you don’t see. A retry wrapper or a broad try/except swallows the error, logs “no items to process,” and exits 0. The job “succeeded.” It just did nothing. If that sounds familiar, the stalled automation alarm guide covers catching “process up, work not happening” in general. This post is about one specific reason it happens.

Rule out the cousins first

A few failures look similar and send you down the wrong path.

  • fork: retry: Resource temporarily unavailable or EAGAIN when starting processes points at a process or thread limit (TasksMax, nproc), not descriptors.
  • The OOM killer leaves a trail. Check journalctl -k | grep -i "killed process". If the kernel killed something, read the OOM killer and memory limits guide. EMFILE leaves no kernel log entry, and the process keeps running.
  • No space left on device is usually the disk. Run df -h first. One trap: a file watcher that hits the inotify watch limit also reports ENOSPC, even with plenty of disk free.
  • A crash loop means the process keeps dying. If systemd gave up restarting, see the restart loop and start limit guide. With EMFILE, the main PID never exits. That’s the whole problem.

Why it hides

The default soft limit is small. On Raspberry Pi OS and Debian, services commonly start with a soft limit of 1024 open files. That sounds like a lot until a leak adds a few descriptors per job.

Long-running processes never reset. A short cron job that leaks five descriptors exits and gets them all back. An agent daemon that leaks five per run accumulates them for days, until it hits the ceiling around the same hour every week.

Healthchecks stay green. A healthcheck over a keep-alive connection reuses a socket that’s already open. A plain port check succeeds because the kernel completes the TCP handshake before the application calls accept(). The connection sits in the backlog while the app fails to accept it.

“active (running)” means one thing. It means the main PID hasn’t exited. It says nothing about whether that PID can open a file.

Step 1: Confirm it in the service’s context

Don’t trust ulimit -n in your SSH session. systemd sets limits for services itself and does not read the PAM limits.conf that a login shell loads. Your shell can say 65535 while the service is stuck at 1024.

Find the PID. Substitute your unit name:

PID=$(systemctl show -p MainPID --value your-agent.service)
echo "$PID"

If that prints 0, the service isn’t running, and you have a different problem.

Read the limits the process actually has:

grep "Max open files" /proc/$PID/limits
Max open files            1024                 524288               files

The first number is the soft limit, which is what bites. The second is the hard limit.

Count what it has open right now:

sudo ls -1 /proc/$PID/fd | wc -l

If that number is at or near the soft limit, you’ve confirmed EMFILE. If it’s at 1021 of 1024, there’s your 3 a.m.

Check what systemd thinks it configured:

systemctl show your-agent.service -p LimitNOFILE

You don’t need lsof for any of this. It often isn’t installed on a minimal Pi image, and /proc is always there.

Step 2: See what’s holding the descriptors

sudo readlink /proc/$PID/fd/* 2>/dev/null | sort | uniq -c | sort -nr | head

Files show up by path, so a log file opened 600 times will top the list. Sockets, pipes, and anonymous inodes each carry a unique number, so group them:

sudo readlink /proc/$PID/fd/* 2>/dev/null | sed 's/\[[0-9]*\]//' | sort | uniq -c | sort -nr | head
    912 socket:
     41 pipe:
     12 /var/log/agent/run.log
      6 anon_inode:inotify
      3 /dev/null

Here’s how to read the top entry:

  • Hundreds of socket: means leaked HTTP, WebSocket, or database connections, or a browser/CDP session that never closed.
  • Many pipe: means subprocesses whose stdout/stderr pipes were never closed or waited on.
  • The same log path repeated means something opens a log handler per run or per request.
  • Lots of anon_inode:inotify means file watchers being created and not torn down.

Run it twice, ten minutes apart. The category that grows is your leak.

Step 3: Common causes on a Pi agent box

Log handlers per run. Code that calls logging.FileHandler(...) or fs.createWriteStream(...) inside the job loop opens a new handle every time and never closes the old one. This is a descriptor problem, not a disk problem, but if your log files are also multiplying, the log hygiene guide covers that side.

File watchers on a big tree. Watching an entire repo or node_modules recursively burns descriptors and watches. Watch the one directory you need.

Browser sessions left open. Playwright, Puppeteer, or a raw CDP connection that isn’t closed after a job leaves sockets and child process pipes behind. This is one of the most common leaks we see on agent boxes. The browser session preflight guide shows how to check and recycle them.

Subprocess pipes. Starting a tool with captured output and never reading to the end or waiting on it leaks two or three descriptors per call.

Connection leaks. A new HTTP client per request instead of one shared client, WebSocket reconnects that don’t close the old socket, or database connections opened per task without a pool.

Step 4: Fix it, in the right order

Restart to recover, not to fix. sudo systemctl restart your-agent.service gets you running again today. The counter starts climbing from zero again.

Fix the leak. Create log handlers once at startup, or log to stdout and let journald handle it. Reuse one HTTP client. Close browser contexts and pages in a finally block, and recycle the whole browser between jobs. Use with blocks or explicit close() for files and sockets.

Raise the limit only with a finite number, and only after you understand the leak. Raising it without fixing the leak just moves the failure from Tuesday to next month. If your workload legitimately needs more than 1024, add a drop-in:

sudo systemctl edit your-agent.service
[Service]
LimitNOFILE=8192

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart your-agent.service

Pick 4096 or 8192, not infinity, and do not set it to unlimited. A finite ceiling is what makes a leak visible before it takes the whole box down with it.

Verify on the process, not the config:

PID=$(systemctl show -p MainPID --value your-agent.service)
grep "Max open files" /proc/$PID/limits

If it still says 1024, the drop-in didn’t apply. Check systemctl cat your-agent.service for a typo or a later override.

Step 5: A fail-closed preflight

This script checks descriptor headroom on the agent’s main PID and refuses to continue if usage is at or above 80 percent, or if it can’t find or read the process. It sends one Discord alert per incident and never prints the webhook URL.

Save it as /usr/local/bin/agent-fd-preflight.sh:

#!/usr/bin/env bash
set -euo pipefail

UNIT="${AGENT_UNIT:-your-agent.service}"
THRESHOLD_PCT="${FD_THRESHOLD_PCT:-80}"
MARKER="/run/agent-fd-preflight.alerted"
ENV_FILE="/etc/agent-ops/discord.env"

DISCORD_WEBHOOK_URL=""
if [[ -r "$ENV_FILE" ]]; then
  # shellcheck disable=SC1090
  source "$ENV_FILE"
fi

log() {
  echo "agent-fd-preflight: $*" >&2
  logger -t agent-fd-preflight -- "$*" || true
}

alert_once() {
  local msg="$1"
  if [[ -e "$MARKER" ]]; then
    return 0
  fi
  if [[ -n "${DISCORD_WEBHOOK_URL:-}" ]]; then
    curl -fsS -m 10 -H "Content-Type: application/json" \
      -d "{\"content\": \"$(hostname): $msg\"}" \
      "$DISCORD_WEBHOOK_URL" >/dev/null 2>&1 || log "discord alert failed"
  fi
  touch "$MARKER"
}

fail() {
  log "FAIL $*"
  alert_once "FD preflight failed for $UNIT - $*"
  exit 1
}

pid="$(systemctl show -p MainPID --value "$UNIT" 2>/dev/null || echo 0)"
if [[ -z "$pid" || "$pid" == "0" || ! -d "/proc/$pid" ]]; then
  fail "no running MainPID"
fi

soft="$(awk '/^Max open files/ {print $4}' "/proc/$pid/limits" 2>/dev/null || true)"
if [[ -z "$soft" || ! "$soft" =~ ^[0-9]+$ ]]; then
  fail "could not read numeric soft limit for pid $pid (got '${soft:-none}')"
fi

if ! fd_list="$(ls -1 "/proc/$pid/fd" 2>/dev/null)"; then
  fail "cannot read /proc/$pid/fd (run as root or the service user)"
fi
count="$(awk 'NF {n++} END {print n+0}' <<<"$fd_list")"

pct=$(( count * 100 / soft ))

if (( pct >= THRESHOLD_PCT )); then
  fail "open fds $count of soft limit $soft (${pct}%) on pid $pid"
fi

log "OK open fds $count of soft limit $soft (${pct}%) on pid $pid"
rm -f "$MARKER"
sudo chmod 755 /usr/local/bin/agent-fd-preflight.sh
sudo /usr/local/bin/agent-fd-preflight.sh; echo "exit=$?"

Success looks like this:

agent-fd-preflight: OK open fds 214 of soft limit 1024 (20%) on pid 1873
exit=0

Failure looks like this:

agent-fd-preflight: FAIL open fds 917 of soft limit 1024 (89%) on pid 1873
exit=1

Put the webhook in /etc/agent-ops/discord.env as DISCORD_WEBHOOK_URL=..., owned by root with mode 600. The marker lives in /run, so it clears on reboot and after the first healthy check.

For recurring jobs, chain the preflight with && so the job never starts on a starved process. In /etc/cron.d/agent-jobs:

15 3 * * * root /usr/local/bin/agent-fd-preflight.sh && /usr/local/bin/run-nightly-publish.sh

For the daemon itself, run the check on its own every 15 minutes:

*/15 * * * * root /usr/local/bin/agent-fd-preflight.sh

Don’t rely on ExecStartPre= for a long-lived daemon. It runs before the process exists, when the count is zero. The leak happens days later. ExecStartPre= only makes sense for a oneshot unit that checks a different, already-running service.

Make headroom part of normal checks

Add the descriptor count and the soft limit to your external healthcheck or your shift report. “214 of 1024” on Monday and “640 of 1024” on Thursday tells you about the leak before anything fails.

During your next reboot test, don’t stop at “it came back up.” Check /proc/PID/fd again after the service has been running for a day. At boot, everything looks clean. The leak shows up later.

Standing routine

  • Every 15 minutes: run agent-fd-preflight.sh against the daemon’s MainPID.
  • Before every recurring job: chain the preflight with &&.
  • Daily: log the open count and the soft limit in your shift report, and watch the trend.
  • After any deploy: run the grouped readlink command twice, ten minutes apart, and confirm no category is growing.
  • When it fires: restart to recover, find the growing category, fix the leak, and only then consider a finite LimitNOFILE=.
  • After any limit change: verify on /proc/PID/limits, not on the unit file.

If you want the prompts and checklists we use to keep unattended agents honest, start with the free prompts, go deeper with the OpenClaw Agent Ops Toolkit, or get everything in the Ultimate Bundle.

More from the build log

Suggested

Want the full MarketMai stack?

Get the core MarketMai guides and operator playbooks in one premium bundle for $49.

View Bundle