Your Agent Can Ping the Internet. It Just Can't Find Anything by Name.
You check in over coffee. The Discord channel is quiet, which is usually good news. The uptime monitor is green. systemctl status says active (running). The Pi answers SSH instantly.
Then you notice the thing that should have happened overnight didn’t. No commit, no deploy, no post. The job ran, and every outbound call died before it left the house, because the box forgot how to turn api.github.com into an address.
This is the failure that hides best on a self-hosted agent box. The network is up. Only DNS is broken. Most of your checks never ask a DNS question, so all of them pass.
The errors you’ll actually see
Same root cause, different costumes depending on what made the call:
curl: (6) Could not resolve host: api.github.com
fatal: unable to access 'https://github.com/you/site.git/': Could not resolve host: github.com
ssh: Could not resolve hostname github.com: Temporary failure in name resolution
socket.gaierror: [Errno -3] Temporary failure in name resolution
Error: getaddrinfo EAI_AGAIN api.anthropic.com
Error: getaddrinfo ENOTFOUND api.github.com
The distinction worth learning: EAI_AGAIN and “Temporary failure” mean the resolver didn’t answer at all. ENOTFOUND (Python’s [Errno -2] Name or service not known) means something answered “no such name.” The first points at a dead or missing nameserver. The second points at a search domain, a filtering resolver, or a typo.
Why it hides
Three reasons, and they stack:
- Your healthcheck is local.
curl http://localhost:8080/healthreturns 200 without resolving a single external name. - Your connectivity check uses an IP.
ping 1.1.1.1proves routing, not resolution. - Your job swallowed the error. A wrapper caught the exception, logged it somewhere nobody reads, and exited 0. If that sounds familiar, the stalled automation alarm post covers catching “process up, work not happening.”
And After=network-online.target doesn’t save you. It’s necessary, but “online” can mean the interface has a carrier and an address. It doesn’t promise a nameserver that answers. It certainly says nothing about 3 a.m., when the Wi-Fi napped and came back with stale resolver state.
Step 1: Confirm it’s DNS, not “the internet is down”
ping -c 2 1.1.1.1
ping -c 2 github.com
If the first succeeds and the second prints ping: github.com: Temporary failure in name resolution, you’ve found it. Read the error text carefully: GitHub drops ICMP, so 100% packet loss on the second line means DNS worked and only the ping was ignored. The name resolution error is the one that matters.
Ask the system resolver directly, the same way your agent does:
getent ahosts github.com
Success prints a list of addresses. Failure prints nothing and exits non-zero.
Now look at what the box thinks its nameservers are:
ls -l /etc/resolv.conf
cat /etc/resolv.conf
What you’re looking for:
- An empty file, or no
nameserverlines. Something rewrote it and had nothing to write. - A symlink to
/run/systemd/resolve/stub-resolv.confwithnameserver 127.0.0.53. That’s systemd-resolved’s stub, which only works if resolved is running. - A single
nameserver 192.168.1.1(or your Pi-hole’s address). One upstream, one point of failure. - A
search lanordomain homeline. Keep that in mind for Step 2.
If systemd-resolved is in play:
systemctl is-active systemd-resolved
resolvectl status
resolvectl query github.com
resolvectl status shows which DNS servers each link actually has. A link with no “Current DNS Server” is your answer.
Finally, check whether anything is listening on port 53 locally:
sudo ss -lntup 'sport = :53'
If resolv.conf says 127.0.0.53 or 127.0.0.1 and this prints only a header, your resolver is pointed at an empty room.
Step 2: Find the cause
These are the usual suspects on a Pi or Debian agent box.
resolv.conf got overwritten. dhcpcd (older Raspberry Pi OS images) and NetworkManager (newer ones) both rewrite /etc/resolv.conf. A failed DHCP renew can leave it empty or missing its nameserver lines.
The symlink outlived the service. resolv.conf points at the resolved stub, but systemd-resolved is stopped, masked, or crashed.
Your only nameserver is down. The router rebooted for a firmware update, or your Pi-hole is mid-restart. If Pi-hole runs on the same Pi as your agent, its restarts are now your agent’s outages.
Boot ordering won the race, DNS lost. network-online.target fired when the interface came up, a beat before DHCP handed out DNS servers. Your service started into a resolver with nothing configured.
Wi-Fi power save. The Pi’s Wi-Fi chip naps when idle. Overnight it can drop off, reassociate, and leave resolver state stale, so the first job of the morning fails.
Search domain surprises. If a lookup fails, the resolver retries with your search domain appended. Some home routers answer anything under .lan with their own address, so api.example.com.lan resolves to the router. Now your agent is attempting TLS with your router and the error looks like a certificate problem. Test with a trailing dot to bypass the search list:
getent ahosts github.com.
grep -E '^(search|domain)' /etc/resolv.conf
Broken IPv6. The resolver returns AAAA records, the box tries IPv6 first, and IPv6 routing goes nowhere. Lookups seem to hang, then fall back slowly or time out. Compare:
getent ahosts api.github.com
curl -4 -sS -o /dev/null -w 'v4 %{http_code} %{time_total}s\n' https://api.github.com
curl -6 -sS -o /dev/null -w 'v6 %{http_code} %{time_total}s\n' https://api.github.com
If v4 is quick and v6 times out, that’s your hang.
Step 3: Fix it
Prove a working nameserver fixes it
Temporarily, with systemd-resolved (runtime only, gone on reboot):
sudo resolvectl dns wlan0 1.1.1.1 8.8.8.8
resolvectl query github.com
Without resolved, and only if /etc/resolv.conf is a regular file:
sudo cp /etc/resolv.conf /etc/resolv.conf.bak
printf 'nameserver 1.1.1.1\nnameserver 8.8.8.8\n' | sudo tee /etc/resolv.conf
getent ahosts github.com
If that works, you know the cause. Now make it stick through whatever tool owns resolv.conf on your box, or it will be overwritten at the next DHCP renew.
Make it persist with the tool you already use
NetworkManager:
nmcli -t -f NAME,DEVICE con show --active
sudo nmcli con mod "YourConnectionName" ipv4.dns "192.168.1.1 1.1.1.1"
sudo nmcli con up "YourConnectionName"
That adds those servers alongside whatever DHCP provides. Keeping your router first preserves local names.
dhcpcd, in /etc/dhcpcd.conf:
interface wlan0
static domain_name_servers=192.168.1.1 1.1.1.1
Then sudo systemctl restart dhcpcd.
systemd-resolved, as a drop-in at /etc/systemd/resolved.conf.d/fallback.conf:
[Resolve]
FallbackDNS=1.1.1.1 9.9.9.9
Then sudo systemctl restart systemd-resolved. FallbackDNS is used only when no link has DNS configured, which is exactly the empty-DHCP case. If resolved itself was stopped, also run sudo systemctl enable --now systemd-resolved.
Leave DNSSEC alone. Turning it off isn’t the fix here, and a single random public resolver as your permanent setup just trades one point of failure for another.
For the IPv6 hang, prefer IPv4 for outbound connections by adding this line to /etc/gai.conf:
precedence ::ffff:0:0/96 100
For search domain trouble, remove the search domain at the source (router DHCP options, or ipv4.dns-search "" in NetworkManager).
Turn off Wi-Fi power save
Check it, then turn it off now:
iw dev wlan0 get power_save
sudo iw dev wlan0 set power_save off
To make it durable on NetworkManager (2 means disabled):
sudo nmcli con mod "YourConnectionName" 802-11-wireless.powersave 2
sudo nmcli con up "YourConnectionName"
On a dhcpcd box, use a hook so it’s reapplied every time the interface comes back. Add this to /etc/dhcpcd.exit-hook:
if [ "$interface" = "wlan0" ]; then
/usr/sbin/iw dev wlan0 set power_save off
fi
Or use a small systemd oneshot at /etc/systemd/system/wifi-powersave-off.service:
[Unit]
Description=Disable Wi-Fi power save on wlan0
After=sys-subsystem-net-devices-wlan0.device
Wants=sys-subsystem-net-devices-wlan0.device
[Service]
Type=oneshot
ExecStart=/usr/sbin/iw dev wlan0 set power_save off
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
Then sudo systemctl enable --now wifi-powersave-off.service. The oneshot covers boot. The dhcpcd hook also covers reconnects.
Wait for a real answer, not a target
Save this as /usr/local/bin/wait-for-dns and run chmod +x on it:
#!/usr/bin/env bash
set -euo pipefail
host="${1:?usage: wait-for-dns HOST [TIMEOUT_SECONDS]}"
limit="${2:-90}"
deadline=$(( SECONDS + limit ))
until timeout 5 getent ahosts "$host" >/dev/null; do
if (( SECONDS >= deadline )); then
echo "wait-for-dns: $host did not resolve within ${limit}s" >&2
exit 1
fi
sleep 3
done
Then add it to your agent’s unit:
[Unit]
Wants=network-online.target
After=network-online.target
[Service]
ExecStartPre=/usr/local/bin/wait-for-dns github.com 90
TimeoutStartSec=150
TimeoutStartSec has to be longer than the wait, because the default start timeout is also 90 seconds. And if DNS flaps while Restart=on-failure is set, repeated ExecStartPre failures can push the unit into its start limit, where systemd quietly stops trying. The restart loop post covers how to spot and set that.
Step 4: A fail-closed preflight for cron jobs
This preflight checks whether the box can reach an IP and whether it can resolve the hostnames your job needs. It tells those two failures apart, alerts Discord once per outage, and exits non-zero so the real job never starts.
The twist: you can’t post to discord.com if DNS is broken. The script tries normal resolution first, then falls back to curl’s --doh-url to resolve through 1.1.1.1 over HTTPS. If both fail, it logs locally and retries the alert on the next run.
Create the marker directory on every boot with /etc/tmpfiles.d/agent-preflight.conf, replacing agent with the user cron runs as:
d /run/agent-preflight 0755 agent agent -
Run sudo systemd-tmpfiles --create once. Put the webhook in /etc/agent/alerts.env as DISCORD_WEBHOOK_URL=..., owned by that user, with chmod 600.
Then save this as /usr/local/bin/agent-dns-preflight:
#!/usr/bin/env bash
set -euo pipefail
REQUIRED_HOSTS=(github.com api.cloudflare.com discord.com)
PROBE_IP="1.1.1.1"
MARKER="/run/agent-preflight/dns-alert-sent"
ENV_FILE="/etc/agent/alerts.env"
if [[ -r "$ENV_FILE" ]]; then
# shellcheck disable=SC1090
source "$ENV_FILE"
fi
DISCORD_WEBHOOK_URL="${DISCORD_WEBHOOK_URL:-}"
log() { printf '%s agent-dns-preflight: %s\n' "$(date -Is)" "$*" >&2; }
post_webhook() {
# URL goes in via stdin config so it never appears in ps or logs.
printf 'url = "%s"\n' "$DISCORD_WEBHOOK_URL" |
curl -sS -o /dev/null --fail --max-time 10 \
-H 'Content-Type: application/json' \
-d "$PAYLOAD" "$@" -K -
}
send_discord() {
if [[ -z "$DISCORD_WEBHOOK_URL" ]]; then
log "no webhook configured; alert not sent"
return 1
fi
PAYLOAD=$(printf '{"content":"%s"}' "$1")
post_webhook 2>/dev/null && return 0
post_webhook --doh-url https://1.1.1.1/dns-query 2>/dev/null && return 0
return 1
}
route_ok=true
if ! timeout 5 bash -c "exec 3<>/dev/tcp/$PROBE_IP/443" 2>/dev/null; then
route_ok=false
fi
failed=()
for host in "${REQUIRED_HOSTS[@]}"; do
if ! timeout 5 getent ahosts "$host" >/dev/null; then
failed+=("$host")
fi
done
if [[ ${#failed[@]} -eq 0 && "$route_ok" == true ]]; then
if [[ -e "$MARKER" ]]; then
send_discord "Preflight on $(hostname): network and DNS OK again." ||
log "recovery notice not delivered"
rm -f "$MARKER"
fi
exit 0
fi
if [[ "$route_ok" == false ]]; then
reason="NO ROUTE: cannot reach $PROBE_IP:443. Uplink or router is down, not just DNS."
else
reason="DNS BROKEN: $PROBE_IP:443 reachable but cannot resolve: ${failed[*]}"
fi
log "$reason"
if [[ ! -e "$MARKER" ]]; then
if send_discord "Preflight on $(hostname) failed. $reason Job skipped."; then
: > "$MARKER"
else
log "alert not delivered; will retry next run"
fi
fi
exit 1
Make it executable with sudo chmod 755 /usr/local/bin/agent-dns-preflight, then chain it in front of the real job:
*/30 * * * * { /usr/local/bin/agent-dns-preflight && /home/agent/bin/run-publish-job; } >> /home/agent/logs/publish.log 2>&1
On success, the preflight is silent and the job runs. On failure, the log gets one clear line, Discord gets one message per outage, and the job doesn’t run half-blind. If your job also depends on variables cron doesn’t load, the cron environment post covers that separately.
Step 5: Teach your other checks to ask a DNS question
External healthcheck. If your monitor hits a health endpoint, have that endpoint run the equivalent of getent ahosts github.com and return non-200 when it fails. Green should mean “can do the work,” not “process exists.”
Reboot test. Boot is where the DHCP race lives. Add these lines to your reboot and power-loss test, right after you confirm the service came back:
getent ahosts github.com && echo "DNS OK after boot"
/usr/local/bin/agent-dns-preflight && echo "preflight OK after boot"
Standing routine
- Every job: the preflight runs first, chained with
&&. No preflight, no job. - Every agent unit:
Wants=andAfter=network-online.target, plusExecStartPre=/usr/local/bin/wait-for-dnswith a timeout shorter thanTimeoutStartSec. - Every Pi on Wi-Fi: power save off, set through NetworkManager or a dhcpcd hook so it survives reconnects.
- Monthly: run
ls -l /etc/resolv.conf,cat /etc/resolv.conf, andresolvectl statusif you use resolved. Confirm there’s more than one nameserver and that the file is owned by the tool you think owns it. - After any router, Pi-hole, or network change: run the reboot test, including the DNS lines.
- When an alert fires: read the first word.
NO ROUTEmeans go look at the router.DNS BROKENmeans start at Step 1.
If you want ready-made prompts for walking an agent through this kind of diagnosis, grab the free prompts. The preflight, wait script, and alerting patterns are packaged with more runbooks in the OpenClaw Agent Ops Toolkit, and everything ships together in the Ultimate Bundle.
More from the build log
Suggested
Want the full MarketMai stack?
Get the core MarketMai guides and operator playbooks in one premium bundle for $49.
View Bundle