Your Agent Job Didn't Fail. It's Waiting on a Prompt Nobody Will Answer.
Most “failed” overnight agent jobs never actually failed. They started, ran one command that asked a question, and then waited. Nobody was there to answer. The log has no error and the exit code never arrives. When you check in the morning, the lock file is still there and the last two scheduled runs skipped themselves.
This post covers how to spot that kind of hang and how to reproduce it on purpose. It also covers how to make every common tool on a Raspberry Pi or small Linux box either answer itself or fail fast.
Why this happens under cron and systemd
When you run a script in a terminal, tools like git, ssh, apt, npx, gh, and sudo can ask you things. Under cron or a systemd timer, there is no terminal. What happens next depends on the tool and on how the job was started:
- It fails with a confusing error.
gitcan’t open/dev/tty, so you getcould not read Username for 'https://github.com': No such device or address. - It blocks forever. This happens if stdin is an open pipe that never closes, which is common when an agent runtime spawns a tool, or when a job was started from a tmux or ssh session. The prompt sits there until something kills it.
- It reads EOF and does something odd. It might take a default you didn’t want, or abort halfway through a package upgrade.
The common prompts are git asking for credentials over HTTPS, ssh asking to trust an unknown host key or unlock a key passphrase, dpkg asking about a changed config file, npx asking “Need to install the following packages… Ok to proceed?”, gh wanting you to log in, sudo asking for a password, and git opening $EDITOR for a merge or commit message.
Symptoms and how to see what it’s waiting on
Typical signs:
- The job never writes its “done” line.
- The overlap lock stays held, so the next run logs “already running” and exits.
- The process is still alive hours later, in state
S(sleeping), using no CPU.
Find it and look at it:
ps -eo pid,stat,etime,tty,cmd | grep -E '[g]it|[s]sh|[a]pt|[n]px|[g]h |[s]udo'
etime tells you how long it has been running. A tty of ? means no terminal, as expected under cron.
Then check what its stdin is and where it’s sleeping:
PID=12345
ls -l /proc/$PID/fd/0 # /dev/null, a pipe:[...], or /dev/pts/N
cat /proc/$PID/wchan; echo # kernel function it's blocked in
If fd 0 is a pipe or a pts device and wchan shows something like pipe_read, n_tty_read, or wait_woken, it is almost certainly sitting on a read. If strace is installed, you can watch it directly. You will probably need sudo:
sudo strace -p "$PID" -e trace=read,openat
A process parked on read(0, ...) is waiting for an answer.
Reproduce it by hand, with no TTY and closed stdin
Don’t wait for 3 a.m. to find out. Run the job the way the scheduler does: no controlling terminal, and stdin closed.
setsid -w ./job.sh </dev/null
setsid detaches it from your terminal, so tools can’t open /dev/tty. </dev/null means any read gets EOF immediately. -w waits so you see the exit code.
To also strip the environment down to roughly what cron gives you:
env -i HOME="$HOME" PATH=/usr/bin:/bin LANG=C.UTF-8 \
setsid -w ./job.sh </dev/null; echo "exit=$?"
Cron’s environment is very different from your login shell, and that is its own source of surprises. We covered it in why cron gives your agent a different environment.
If the job runs as a systemd user service, test it under systemd itself:
systemd-run --user --wait --pipe --quiet "$PWD/job.sh" </dev/null
If it hangs here, it will hang in production. If it fails here with a prompt-related error, you just saved yourself a morning.
Per-tool settings that stop the questions
Put these in the job wrapper or an env file. These are the exact settings for each tool.
git (HTTPS credentials): never let git prompt. Configure a credential helper or token for the account that runs the job, and turn prompting off so a missing credential fails instead of waiting:
export GIT_TERMINAL_PROMPT=0
You’ll sometimes see GIT_ASKPASS=/bin/true suggested. It works, but it hands git an empty password and can produce muddier errors. GIT_TERMINAL_PROMPT=0 plus a real credential helper is cleaner.
We’ve seen this one ourselves. MarketMai’s own unattended publish jobs have hit fatal: could not read Username for 'https://github.com': No such device or address on a machine with no stored GitHub credential. With no terminal to prompt on, it failed in under a second instead of hanging. That is the behavior you want. (If your publish job is blocked for a different reason, see the dirty git tree publish block.)
git (editor): merges, rebases, and commits without -m open an editor. Stop that:
export GIT_EDITOR=true
git merge --no-edit origin/main
git commit -m "Automated update"
ssh (host keys and passphrases): BatchMode=yes disables password and passphrase prompts. StrictHostKeyChecking=yes refuses unknown hosts instead of asking:
export GIT_SSH_COMMAND="ssh -o BatchMode=yes -o StrictHostKeyChecking=yes"
Pre-populate known_hosts once, and check the fingerprint against the one the host publishes before trusting it:
ssh-keyscan -t ed25519 github.com > /tmp/gh_hostkey
ssh-keygen -lf /tmp/gh_hostkey # compare to GitHub's published fingerprint
cat /tmp/gh_hostkey >> ~/.ssh/known_hosts
StrictHostKeyChecking=accept-new is fine for a single, supervised first contact. Don’t leave it on forever, because it silently trusts any new host you point it at. Use a key without a passphrase for the job user, or a running agent, since BatchMode won’t ask.
apt/dpkg: stop the conffile prompts and keep your existing config:
sudo -n env DEBIAN_FRONTEND=noninteractive apt-get -y \
-o Dpkg::Options::=--force-confdef \
-o Dpkg::Options::=--force-confold \
upgrade
Using env after sudo makes sure the variable actually reaches apt.
npx: npx --yes some-tool@1.4.2 skips the “Ok to proceed?” question. A better approach is to npm install a pinned version once and call ./node_modules/.bin/some-tool, so the job never downloads anything at run time. For npm generally, export npm_config_yes=true.
gh: disable prompts and supply the token from an env file the job loads:
export GH_PROMPT_DISABLED=1
# GH_TOKEN comes from the job's env file, not the script
pip: export PIP_NO_INPUT=1.
sudo: always sudo -n. With -n, sudo fails immediately if it would need a password, instead of waiting.
Everything else that opens an editor: set the editor to something that fails fast, so nothing silently waits on an editor that never closes:
export EDITOR=/bin/false VISUAL=/bin/false
GIT_EDITOR=true still wins for git, because git checks it before VISUAL and EDITOR.
Belt and braces in the wrapper
Settings cover the tools you know about. The wrapper covers the ones you forgot.
#!/usr/bin/env bash
set -uo pipefail
exec </dev/null # any read anywhere in this job gets EOF immediately
export GIT_TERMINAL_PROMPT=0 GIT_EDITOR=true
export GIT_SSH_COMMAND="ssh -o BatchMode=yes -o StrictHostKeyChecking=yes"
export GH_PROMPT_DISABLED=1 PIP_NO_INPUT=1 npm_config_yes=true
export DEBIAN_FRONTEND=noninteractive EDITOR=/bin/false VISUAL=/bin/false
"$HOME/agent/preflight.sh" || exit $?
exec timeout --kill-after=30s 20m "$HOME/agent/run-job.sh"
exec </dev/null turns “waits forever” into “fails now.” timeout is the backstop for anything that still hangs, such as a slow network call. It sends TERM at 20 minutes and KILL 30 seconds later. Pair it with an overlap lock; see run timeouts and overlap protection.
Under systemd, stdin is already /dev/null by default (StandardInput=null), but say so explicitly so nobody changes it later:
[Service]
Type=oneshot
StandardInput=null
EnvironmentFile=%h/.config/agent/job.env
ExecStart=%h/agent/job-wrapper.sh
TimeoutStartSec=25min
For Type=oneshot, TimeoutStartSec is the limit that applies. RuntimeMaxSec is for long-running simple or exec services and has no effect on oneshot units.
A preflight that fails before the job starts work
It’s better to learn that credentials are missing before the agent has spent tokens and written half a result. This script checks access without prompting and exits 75 (temporary failure) with a clear message:
#!/usr/bin/env bash
set -uo pipefail
exec </dev/null
export GIT_TERMINAL_PROMPT=0 GH_PROMPT_DISABLED=1
REPO_DIR="${REPO_DIR:-$HOME/site}"
DEPLOY_HOST="${DEPLOY_HOST:-}"
problems=()
if ! timeout 20 git -C "$REPO_DIR" ls-remote origin HEAD >/dev/null 2>&1; then
problems+=("git cannot reach origin without a prompt (check credential helper or deploy key)")
fi
if [[ -n "$DEPLOY_HOST" ]] && ! timeout 20 ssh -o BatchMode=yes \
-o StrictHostKeyChecking=yes -o ConnectTimeout=10 "$DEPLOY_HOST" true >/dev/null 2>&1; then
problems+=("ssh to $DEPLOY_HOST needs a prompt or is unreachable (key, passphrase, or known_hosts)")
fi
if command -v gh >/dev/null 2>&1 && ! timeout 20 gh auth status >/dev/null 2>&1; then
problems+=("gh is not authenticated (is GH_TOKEN in the env file?)")
fi
if (( ${#problems[@]} > 0 )); then
msg="$(hostname): job blocked in preflight:"
for p in "${problems[@]}"; do msg+=" [$p]"; done
echo "$msg" >&2
if [[ -n "${DISCORD_WEBHOOK_URL:-}" ]]; then
safe=${msg//\\/}; safe=${safe//\"/}
curl -fsS -m 10 -H 'Content-Type: application/json' \
-d "{\"content\":\"$safe\"}" "$DISCORD_WEBHOOK_URL" >/dev/null 2>&1 \
|| echo "preflight: Discord alert failed" >&2
fi
exit 75
fi
The webhook URL is never echoed, and curl’s own errors are discarded so they can’t leak it into logs. Quotes and backslashes are stripped from the message so the JSON stays valid.
The one-line habit
Every time you add or change a job script, run this:
grep -nE '\b(git|ssh|scp|apt(-get)?|npx|npm|pip3?|gh|sudo)\b' ~/agent/*.sh
For every hit, ask: can this prompt? If yes, add the flag or variable from above.
Make it a standing check
Prompt-proofing isn’t a one-time fix. New tools and new scripts bring new prompts with them. Grab our free prompts and turn one into a “review this wrapper for anything that can prompt” routine that runs whenever a job script changes.
If you want the wrappers, preflight checks, lock handling, and alerting already built, the OpenClaw Agent Ops Toolkit packages them for exactly this setup. Or get the Ultimate Bundle, which includes everything for $49.
More from the build log
Suggested
Want the full MarketMai stack?
Get the core MarketMai guides and operator playbooks in one premium bundle for $49.
View Bundle