The Agent Run That Never Ended: Hard Timeouts and Overlap Protection for Scheduled Jobs

Scheduled agent jobs have a failure mode that ordinary scripts rarely show: they do not crash, they just never finish. A model call stalls on a dropped connection. A tool waits on a prompt nobody will answer. A browser step sits on a page that never loads. The process is alive, the log is quiet, and nothing in your monitoring says anything is wrong, because nothing is wrong yet. Something is merely not done.

Then the next scheduled run starts. Now two runs are fighting over the same files, the same API quota, and the same git checkout. By the time you notice, you have three, and the Pi is swapping.

This post is a small, boring fix for that: every scheduled run gets a hard time limit, a single-instance lock, and a named way to end.

Why agent jobs hang more than scripts do

A normal cron script does a fixed amount of work. An agent run is a loop: think, call a tool, read the result, think again. Each turn depends on something outside your control, and the number of turns is not fixed.

That gives you several ways to hang without erroring:

  • Network stalls. A request with no timeout waits as long as the operating system lets it, which can be many minutes.
  • Interactive prompts. A command asks for confirmation, an approval, or a password. Unattended, nobody answers, and the run waits forever.
  • Unbounded loops. The agent keeps retrying a step that cannot succeed, each retry looking like progress.
  • Slow children. A build, a test suite, or a deploy takes ten times longer than usual because the machine is under load.

None of these produce an exit code. They produce silence, and silence is the hardest thing to alert on.

Layer one: a hard wall-clock limit

Every scheduled run needs a maximum lifetime enforced from outside the agent. Do not rely on the agent to notice it has been running too long. It is the thing that is stuck.

On a Linux box the simplest tool is timeout:

timeout --signal=TERM --kill-after=30 25m ./run-daily-job.sh

That sends a polite TERM at 25 minutes and a hard KILL 30 seconds later if the process ignores it. Exit code 124 tells you the limit fired, which is a distinct, loggable outcome rather than a mystery.

Pick the limit from observed behavior, not hope. Look at the last ten successful runs, take the slowest, and give it roughly double. A job that normally takes six minutes gets a fifteen-minute ceiling. Too tight and you kill healthy runs on a slow day. Too loose and the limit never helps.

If your scheduler has its own run timeout setting, use that too. Two independent limits are fine. The outer one catches the case where the inner one is misconfigured.

Layer two: one run at a time

A time limit stops one run from living forever. It does not stop the next run from starting while the current one is still legitimately busy. For that you want a lock that makes a second instance refuse to start.

flock does this without any lock file cleanup problems, because the lock is held by the open file descriptor and released by the kernel when the process dies, however it dies:

flock --nonblock /tmp/daily-job.lock ./run-daily-job.sh || echo "previous run still active, skipping"

The --nonblock flag matters. Without it, the second run queues behind the first and you rebuild the pile-up one step later. With it, the late run exits immediately and says so.

Decide deliberately what a skipped run means. For a daily publish job, skipping is usually correct: you would rather miss a slot than run two deploys at once. For a job that processes a queue, skipping is also fine because the running instance will pick up the work. For a job that must run at a specific time, a skip should raise a flag, because the schedule just did not happen.

If you have been bitten by lock files that outlive a crash, that is a separate problem with a separate fix, and it is one reason to prefer kernel-held locks over a file that merely exists.

Layer three: shrink the unbounded pieces

The outer limit is a backstop. The better move is to bound the parts inside the job so they fail fast and tell you why.

  • Set timeouts on every network call. A connect timeout of ten seconds and a total timeout of a minute or two covers most APIs. No default should be “forever.”
  • Run tools non-interactively. Pass the flags that suppress prompts, and close stdin so a stray prompt fails immediately instead of waiting.
  • Cap retries. Three attempts with backoff, then stop and record the failure. An uncapped retry loop is a hang with extra steps.
  • Cap agent turns. If your runtime lets you limit the number of tool calls per run, set it. A run that needs 200 steps to do a 20-step job is lost, not thorough.

Each of these converts “stuck” into “failed at a named step,” which is what you can actually act on.

Layer four: every run ends in a named state

The last piece is what the run says when it ends. Three outcomes cover almost everything:

  1. Done. The work finished and was verified.
  2. Failed, with a reason. A specific step errored, and the message says which.
  3. Timed out or skipped. The limit fired or the lock was held.

The important part is that the second and third outcomes are not quiet. A timeout should produce a message as visible as a success report, with the step that was running when it fired if you can capture it. Write a one-line status file at the end of every run, success or not, with a timestamp and a state. Then a separate check can alert on a status file that is missing or too old, which catches the case where even your own wrapper failed to run.

That pairs well with an independent heartbeat that lives off the box, so that a dead machine and a dead job both end up as visible alarms rather than good-looking silence.

A 30-minute rollout

You do not need to rebuild your schedule. Do this once per job:

  1. List your scheduled jobs. Write down how long each one normally takes.
  2. Wrap each in timeout. Use roughly twice the slowest healthy run.
  3. Wrap each in flock --nonblock. Give every job its own lock path.
  4. Add explicit timeouts to the network calls inside each job. Search the job for any request that has none.
  5. Write a status line at the end. State, timestamp, and the last step reached.
  6. Break one on purpose. Add a sleep longer than the limit to a test copy and confirm you get a visible timeout message, not silence.

That last step is the one people skip, and it is the only one that proves the rest works. A timeout you have never seen fire is a guess.

The payoff

Nothing here makes your agent smarter. It makes your schedule honest. A job either finishes, fails with a reason, or gets cut off and says so. You stop discovering on Thursday that Tuesday’s run is still open, and you stop explaining a slow Pi by finding five copies of the same job in the process list.

Unattended automation is only unattended if the bad days end on their own.

More from the build log

Suggested

Want the full MarketMai stack?

Get the core MarketMai guides and operator playbooks in one premium bundle for $49.

View Bundle