Your Agent Service Isn't Restarting. systemd Gave Up and Didn't Tell You.
You put Restart=on-failure in your agent’s unit file and stopped worrying. Then one of two things happened.
Either your agent stopped replying on Friday night and was dead all weekend, even though “systemd restarts it automatically.” Or it never looked dead at all, but your API bill spiked overnight and your logs grew by a few hundred megabytes because the agent was crashing and restarting every 10 seconds.
Both failures look fine from a distance. Neither one sends you a message. This guide shows you how to tell which one you have, why the math produces it, and how to set restarts that stop, alert you, and get tested on purpose.
Why it happens
Restart=on-failure restarts your service when it exits with a non-zero code, gets killed by a signal, times out, or misses a watchdog ping. The delay before each restart is RestartSec=, which defaults to 100ms.
systemd also has a start rate limit, set by StartLimitIntervalSec= and StartLimitBurst= in the [Unit] section. If you don’t set them, they come from DefaultStartLimitIntervalSec=10s and DefaultStartLimitBurst=5 in /etc/systemd/system.conf. More than 5 starts inside 10 seconds and systemd puts the unit in a failed state and stops trying.
Now the math:
- Default RestartSec (100ms): a script that crashes instantly burns through 5 starts in about a second. The limit trips, the unit fails, and it stays dead. No retry later, no alert.
- RestartSec=10 with default limits: 5 starts take 40 seconds or more. That can never fit inside a 10 second window, so the limit is never reached. The service restarts forever.
The rule to remember: StartLimitBurst times RestartSec has to be smaller than StartLimitIntervalSec, or the limit can never trip.
What actually causes the crashes on agent boxes is usually boring:
- An expired or revoked API key or OAuth token
- The network isn’t up yet at boot
- A port still held by a leftover process
- A config typo after a quick edit
- The kernel’s OOM killer (see memory limits on a Pi)
How to confirm it
Start with status:
systemctl status agent.service
If you see activating (auto-restart), you’re probably in the infinite loop. If you see failed (Result: start-limit-hit), systemd gave up.
Get the exact state in one line:
systemctl show agent.service -p ActiveState -p SubState -p Result -p NRestarts -p ExecMainStatus
NRestarts is the number of automatic restarts. ExecMainStatus is the last exit code. A gave-up unit shows Result=start-limit-hit.
Count how many times it started since boot:
journalctl -u agent.service -b --no-pager | grep -cE "Started|Scheduled restart job"
A healthy long-running service shows a small number. Hundreds means a loop. The journal for a gave-up unit will contain Start request repeated too quickly. and Failed with result 'start-limit-hit'.
Then read the actual error the agent printed before each exit:
journalctl -u agent.service -b -n 50
To bring a start-limit-hit unit back, reset it first. A plain start may be refused until you do:
sudo systemctl reset-failed agent.service && sudo systemctl start agent.service
Fix it
Make all changes with a drop-in so package updates don’t overwrite them:
sudo systemctl edit agent.service
That writes /etc/systemd/system/agent.service.d/override.conf and reloads systemd for you. Verify afterwards:
systemctl cat agent.service
systemd-analyze verify /etc/systemd/system/agent.service
1. Set a bounded restart policy
[Unit]
StartLimitIntervalSec=600
StartLimitBurst=5
[Service]
Restart=on-failure
RestartSec=15
This means: up to 5 starts in 10 minutes, then stop and alert. Five starts at 15 second spacing take about a minute, well inside 600 seconds, so the limit can actually trip. A brief network blip still gets retried. A real breakage stops instead of burning API calls all night.
If your crashes come from provider rate limits, the agent should be backing off internally rather than relying on systemd. See rate limit backoff with jitter.
2. Use exponential backoff if your systemd has it
RestartSteps= and RestartMaxDelaySec= exist only in systemd 254 and newer. Check:
systemctl --version
Raspberry Pi OS and Debian 12 (bookworm) ship systemd 252, so you won’t have these. Debian 13 does. If you’re on 254 or newer:
[Service]
RestartSec=5
RestartSteps=6
RestartMaxDelaySec=5min
Delays grow from 5 seconds toward 5 minutes. Keep the start limit math in mind: longer delays make the burst harder to reach, so widen StartLimitIntervalSec to match.
3. Don’t loop on errors that won’t fix themselves
A revoked key will be revoked on every retry. Have your agent (or its wrapper script) exit with a specific code for bad config or a missing or rejected API key. Code 78 is EX_CONFIG from sysexits:
if [ -z "${AGENT_API_KEY:-}" ]; then
echo "AGENT_API_KEY missing" >&2
exit 78
fi
Do the same when the provider returns an auth error at startup. Then tell systemd not to retry that code:
[Service]
RestartPreventExitStatus=78
Now a revoked key fails once, loudly, instead of hammering the provider. Pair this with a credential expiry calendar so you see it coming.
While you’re in the unit file, fix the boot race too:
[Unit]
Wants=network-online.target
After=network-online.target
4. Alert when systemd gives up
Add to the agent’s [Unit]:
[Unit]
OnFailure=agent-alert@%n.service
On current systemd versions this fires when the unit ends up failed (restarts exhausted or prevented), not on every automatic restart.
Create /etc/systemd/system/agent-alert@.service:
[Unit]
Description=Alert for %i
[Service]
Type=oneshot
EnvironmentFile=/etc/agent-alert.env
ExecStart=/usr/local/bin/agent-alert %i
Put DISCORD_WEBHOOK_URL=https://... in /etc/agent-alert.env and chmod 600 it. Then /usr/local/bin/agent-alert:
#!/usr/bin/env bash
set -u
UNIT="${1:-unknown}"
REASON="${2:-systemd gave up on}"
HOST="$(hostname)"
LOGS="$(journalctl -u "$UNIT" -n 15 --no-pager -o cat 2>&1 | tail -c 1500)"
MSG="${REASON} ${UNIT} on ${HOST}. Last log lines:
${LOGS}"
if command -v jq >/dev/null 2>&1; then
BODY="$(printf '%s' "$MSG" | jq -Rs '{content: .}')"
else
SAFE="$(printf '%s' "$MSG" | tr -d '"\\\t\r' | awk '{printf "%s\\n", $0}')"
BODY="{\"content\": \"${SAFE}\"}"
fi
curl -fsS -m 10 -H "Content-Type: application/json" \
-d "$BODY" "$DISCORD_WEBHOOK_URL" >/dev/null
chmod 755 it. The script never prints the webhook URL, so it won’t end up in your journal. For more on keeping logs useful, see log hygiene and observability.
5. Catch the loop that never trips the limit
OnFailure can’t help if the unit never fails. Add a small check that watches NRestarts. It counts automatic restarts since the unit was last started by hand, so compare against the last value you saw and handle it going down.
/usr/local/bin/agent-restart-check:
#!/usr/bin/env bash
set -u
UNIT=agent.service
MAX=6
STATE=/var/lib/agent-restart-check/last
mkdir -p "$(dirname "$STATE")"
NOW="$(systemctl show -p NRestarts --value "$UNIT")"
LAST="$(cat "$STATE" 2>/dev/null || echo 0)"
[ "$NOW" -lt "$LAST" ] && LAST=0
echo "$NOW" > "$STATE"
if [ $((NOW - LAST)) -gt "$MAX" ]; then
/usr/local/bin/agent-alert "$UNIT" "Restart loop detected for"
fi
Run it every 15 minutes from /etc/cron.d/agent-restart-check:
*/15 * * * * root set -a; . /etc/agent-alert.env; /usr/local/bin/agent-restart-check
This complements a stalled automation alarm, which catches the other quiet failure: a process that’s running but not doing anything.
Test it on purpose
An alert you’ve never seen fire is a guess. Break the service deliberately with a separate drop-in:
sudo mkdir -p /etc/systemd/system/agent.service.d
printf '[Service]\nExecStart=\nExecStart=/bin/false\n' | \
sudo tee /etc/systemd/system/agent.service.d/break.conf
sudo systemctl daemon-reload
sudo systemctl restart agent.service
journalctl -u agent.service -f
The empty ExecStart= line clears the original command. With the policy above you should see about five restarts roughly 15 seconds apart, then Start request repeated too quickly., then a Discord message with the last log lines.
Test the config-error path too. Change the drop-in to ExecStart=/bin/sh -c 'exit 78', run daemon-reload, then reset-failed (the unit is still in start-limit-hit from the first test), start it, and confirm it fails once with no retries and one alert.
Then clean up:
sudo rm /etc/systemd/system/agent.service.d/break.conf
sudo systemctl daemon-reload
sudo systemctl reset-failed agent.service
sudo systemctl start agent.service
Add this to your reboot and power loss test so it runs whenever you change the box.
Make it a standing check
Unit files drift. Someone adds a new service, copies an old one, and the restart policy goes back to defaults. Turn one of the free prompts into a monthly routine that reviews every agent unit file (systemctl cat output) and flags three things: missing StartLimitIntervalSec/StartLimitBurst, RestartSec times StartLimitBurst greater than or equal to the interval, and no OnFailure=. Ten minutes a month beats a dead weekend.
- free prompts
- OpenClaw Agent Ops Toolkit (wrappers, preflight checks, alerting, runbooks)
- Ultimate Bundle, which includes everything for $49
More from the build log
Suggested
Want the full MarketMai stack?
Get the core MarketMai guides and operator playbooks in one premium bundle for $49.
View Bundle