Your AI Agent's Lock File Is Lying: Crash Recovery for Unattended Automation
Every unattended automation eventually grows a lock file. Two runs of the same job must not overlap, so the first thing the job does is create /tmp/job.lock, and the last thing it does is delete it. Simple, sensible, and a trap.
The trap springs the first time the job dies in the middle. The power blips, the process gets OOM-killed, the SD card hiccups, someone reboots the box. The lock file stays. From that moment on, every scheduled run starts, sees the lock, concludes “another run is in progress,” and exits politely. Nothing errors. Nothing alerts. The job just stops doing its work, and the logs say everything is fine.
This is one of the most common ways self-hosted agent setups go quiet for days. Here is how to build locks that don’t lie.
Why the naive lock fails
A lock file records one fact: “someone created me.” What the next run actually needs to know is “is the owner still alive and still working?” Those are different questions, and the gap between them is where silent outages live.
Three failure modes show up repeatedly:
- Crash without cleanup. The process dies before the
rmruns. The lock outlives its owner forever. - Hang without dying. The process is alive but stuck on a dead network call. The lock is technically valid, but nothing is progressing.
- Reboot persistence. Locks stored somewhere that survives reboot (a home directory instead of
/run) outlive the entire boot session they were created in.
Each needs a different answer, but they share one fix: a lock has to carry enough information to be judged.
Put identity inside the lock
Instead of an empty file, write a small record: the PID, the hostname, the start time, and the job name. Now the next run can ask real questions.
- Does that PID exist right now?
- Does that PID belong to the same command, or has the OS reused the number for something unrelated?
- Is the start time older than this job should ever take?
PID reuse is the subtle one. After a reboot, PID 2841 might be your browser. If you only check “does the process exist,” you will decide the lock is valid and skip forever. Compare the process start time or command line, not just the number.
On Linux, flock solves the crash case cleanly because the kernel releases the lock when the owning process dies, no matter how it died. If your jobs are shell scripts, wrapping them in flock -n /run/lock/job.lock your-command removes an entire category of stale-lock bugs for free. Prefer it over hand-rolled lock files whenever it’s available.
Give every lock a maximum age
Even with identity checks, set a ceiling. If a job normally takes four minutes and the lock is ninety minutes old, something is wrong regardless of whether the PID looks alive. A hung process holding a lock is worse than a dead one, because it looks healthy.
Pick the ceiling from observed reality: take your longest successful run over the past month and multiply by three. Anything older than that is treated as abandoned.
The key design choice is what happens at expiry. There are two options:
- Break the lock and continue. Good for idempotent jobs where a double run is harmless.
- Refuse, alert, and wait for a human. Right for jobs that spend money, send messages, or publish. Taking over a lock from a possibly-still-running job can produce exactly the duplicate you were trying to prevent.
Match the behavior to the blast radius. Reading and summarizing can self-heal. Sending and charging should escalate.
Make the skip loud
The root problem in the silent outage isn’t the stale lock. It’s that skipping looked identical to success.
When a run exits because of a lock, it should write a distinct, countable outcome: skipped_locked, with the lock’s age and owner. Then add one rule: if a job reports skipped_locked more than twice in a row, or if the lock age exceeds the ceiling, raise an alert.
That single counter converts an invisible failure into a message in your inbox. It also gives you the right instinct for every other guard in your system: a guard that stops work must report that it stopped work.
Heartbeats for long jobs
For jobs that legitimately run a long time, such as research sweeps, batch publishing, or video processing, the age ceiling is too blunt. Use a heartbeat instead: the running job touches the lock file (or updates a timestamp inside it) every minute. A fresh heartbeat means alive and working. A heartbeat older than a few intervals means stuck, even if the PID exists.
This also gives you a cheap progress signal. A job that holds a lock with a fresh heartbeat but hasn’t advanced its step counter is a different kind of problem than one that stopped heartbeating entirely, and your alert text can say which.
Store locks where reboots clear them
Where the lock lives matters. A lock in /run or /tmp on a tmpfs disappears at reboot, which is exactly what you want: no process survives a reboot, so no lock should either. A lock in your home directory or a project folder survives, and you inherit stale state from a boot session that no longer exists.
If you must persist the lock for some reason, include the boot ID (/proc/sys/kernel/random/boot_id on Linux) in the record. A lock whose boot ID differs from the current one is stale by definition, with no guessing about PIDs.
A practical checklist
Before you trust any lock in an unattended job, check that it:
- Records PID, start time, and command, not just existence.
- Verifies the owner by more than PID number.
- Has a maximum age based on real run times.
- Has a deliberate policy at expiry: break for safe jobs, escalate for risky ones.
- Reports skips as a distinct, counted outcome.
- Lives somewhere a reboot clears, or carries a boot ID.
- Uses
flockwhere the platform allows it.
The takeaway
A lock file is a promise that someone is working. Promises need expiry dates and proof of life. If your automation can only tell you “a lock exists,” then the first crash will turn your scheduler into a very polite machine that does nothing, forever, and reports no problems.
Audit your jobs this week: list every lock, find its age, and check whether the owner is still real. If you find one that’s been sitting for a month, you’ve just found a job that has been quietly dead the whole time. Fix the lock, then fix the reporting, so the next one tells you on day one instead of day thirty.
More from the build log
Suggested
Want the full MarketMai stack?
Get the core MarketMai guides and operator playbooks in one premium bundle for $49.
View Bundle