Your Pi Thinks It Is 1970: Clock Checks for Self-Hosted Agents
Most people running an agent on a Raspberry Pi or a small home server never think about what time the box thinks it is. Why would they? The clock in the corner of the desktop is right, the cron jobs mostly run, and nothing has ever said “your clock is wrong.”
That last part is the problem. A box with the wrong time almost never tells you so. It tells you something else instead:
certificate is not yet validorcertificate has expiredwhen the agent calls an API401 Unauthorizedfrom a provider whose tokens are signed with a timestamp- a webhook receiver that rejects every request as a replay
gitcommits dated in the future or the past, so your history looks out of order- cron jobs that fire at the wrong hour, or not at all, right after a restart
- logs from the agent and logs from the cloud service that refuse to line up
You spend an hour debugging the API key, the certificate store, and the webhook secret. The real fault was a clock that was 4 minutes, 3 hours, or 56 years off.
This post is a 15-minute check that rules the clock in or out, plus the three fixes that make it stay right.
Why a home agent box gets the time wrong
A laptop or desktop has a small battery on the motherboard that keeps the hardware clock running when the power is off. When it boots, it already knows roughly what time it is.
Most Raspberry Pi models do not. The Pi 4 and older have no real-time clock at all. The Pi 5 has one, but it only keeps time across a power cut if you add the small battery it is designed for, and many people never do. So when power comes back, the box starts with no idea what time it is. Raspberry Pi OS papers over this with a saved timestamp from the last clean shutdown, so after a power cut the clock usually starts at “whenever it was last shut down cleanly,” which can be days or weeks ago.
From there, the box depends on the network to fix itself. A time sync service (on most current Linux systems that is systemd-timesyncd or chrony) reaches out to an NTP server, gets the real time, and jumps the clock forward.
That works most of the time. Here is where it does not:
- The network is not up yet. After a power cut, the router reboots too. The Pi is often ready a minute before the router is. The agent and its jobs start with a stale clock, and some of them fail before sync happens.
- Something blocks NTP. Outbound UDP port 123 is blocked by a strict router, a guest network, a VPN, or a Pi-hole style setup that cannot resolve the pool hostnames.
- Sync is turned off or fighting itself. Two time services are installed, or someone disabled the one that came with the OS while testing.
- The clock drifts between syncs. A cheap crystal under heat can drift by seconds a day. That is fine with sync running, and a slow-growing problem without it.
- A container has its own idea. Containers share the host kernel’s clock, so they are usually fine, but a container that ships without zone data shows UTC while everything else shows local time. That looks like a clock problem when it is a time zone problem.
None of these send an alert. The first symptom is usually an agent run that failed for a reason that makes no sense.
The 15-minute check
Do this on the box itself, over SSH. You need nothing installed beyond what ships with the OS.
1. Compare the box to a trusted clock
date -u
Then compare against an outside source. One simple trick is to ask a web server you trust for its Date header:
curl -sI https://www.google.com | grep -i '^date:'
Both are in UTC. If they differ by more than a couple of seconds, you have found something worth fixing. If they differ by minutes or more, you have likely found the cause of whatever weird failure sent you here.
2. Ask the system whether it is syncing
timedatectl
You are looking for three lines:
System clock synchronized: yesNTP service: activeTime zone:set to the zone you actually schedule in
If synchronized says no, the clock might still be right today, but nothing is keeping it right. If NTP service says inactive, turn it on:
sudo timedatectl set-ntp true
3. Find out which time service is actually running
systemctl status systemd-timesyncd --no-pager
systemctl status chrony --no-pager
You want exactly one of them active. If both are installed and both are trying, pick one and disable the other. On a small agent box, systemd-timesyncd is fine. If you want better behavior after long power cuts and on flaky networks, chrony handles big jumps and slow corrections more gracefully.
To see when the last successful sync happened:
timedatectl timesync-status
or, for chrony:
chronyc tracking
A last sync from weeks ago, or no server listed at all, means sync is failing quietly.
4. Check that NTP can get out
If sync is enabled but never succeeds, the network is the usual suspect. Check DNS for the pool name first:
getent hosts pool.ntp.org
No answer means your local DNS is the problem. An answer but still no sync usually means outbound UDP 123 is blocked. Check the router’s firewall rules, any VPN that captures all traffic, and any DNS filter on your network.
5. Reboot and watch the first two minutes
This is the step that catches the real-world failure. Reboot, log back in as soon as you can, and run:
journalctl -b -u systemd-timesyncd --no-pager | head -20
(Use -u chrony if that is your service.) Note the time of the first successful sync. Then look at when your agent runtime and scheduler started:
journalctl -b --no-pager | grep -iE 'openclaw|cron|agent' | head -20
If your agent started before the clock was synced, every call it made in that window ran on the wrong time. That is the gap to close.
If you have not done a deliberate reboot test on this box yet, the reboot test walkthrough covers the rest of what should come back on its own. Fold this clock check into it.
Three fixes that make it stay right
Fix 1: Make the agent wait for the clock
Systemd has a target built for this: time-sync.target. If your agent runs as a systemd service, tell it to start only after time is synced:
[Unit]
After=network-online.target time-sync.target
Wants=network-online.target time-sync.target
On systemd-timesyncd systems, that target is only truly honored if the wait service is enabled:
sudo systemctl enable systemd-time-wait-sync
Now, after a power cut, the agent sits and waits until the clock is real instead of starting at last Tuesday’s time. For cron jobs that call outside APIs, the simplest guard is a check at the top of the script:
if ! timedatectl show -p NTPSynchronized --value | grep -q yes; then
echo "clock not synced, skipping run" >&2
exit 75
fi
A skipped run with a clear reason beats a failed run with a misleading one.
Fix 2: Add a real clock if you lose power often
If your area has storms or your Pi shares a power strip with things people unplug, a battery-backed clock is cheap insurance. The Pi 5 has a connector for an official rechargeable RTC battery. Older models can take a small RTC add-on board. With one in place, the box boots with the right time even before the network is up, and NTP only has to correct seconds instead of days.
Fix 3: Alarm on drift from somewhere else
A box cannot reliably tell you its own clock is wrong, because the alert’s timestamp and any expiry checks use the same broken clock. Put one check outside the box. A free uptime monitor, a hosted scheduler, or another machine can call a tiny endpoint on the Pi that returns its current UTC time, and alert if it is more than 30 seconds away from real time. Even simpler: have the agent include date -u in its daily heartbeat message, and have whatever receives the heartbeat compare it to its own time.
The time zone trap that looks like a clock bug
Half of the “my clock is wrong” reports on home agent boxes are really “my time zone is wrong.” The clock is correct in UTC, but:
- the OS was never set to your zone, so
dateshows UTC - a container has no zone data and logs in UTC
- the agent runtime has its own zone setting that differs from the host
Check each one and write down which zone it uses. If you schedule anything in local time, this matters twice a year. US clocks fall back this Sunday, November 1, and the daylight saving audit for agent cron jobs walks through which of your jobs will shift.
Why this matters more for agents than for a normal server
A plain file server with a wrong clock mostly just has odd file dates. An agent is different because nearly everything it does is a timed conversation with something else:
- Tokens and sessions are signed with issue and expiry times. A box that is behind can be told a fresh token is not valid yet. A box that is ahead treats a good token as expired and keeps asking for new ones.
- Webhooks from payment, chat, and form services often include a timestamp, and the receiver rejects anything outside a few minutes to prevent replays.
- Scheduling is the whole point. “Send the morning brief at 7” means nothing if the box disagrees about what 7 is.
- Evidence matters. When something goes wrong, you line up the agent’s log against the provider’s log. If the clocks disagree by four minutes, you will blame the wrong step.
If you keep a credential expiry calendar for your agent’s keys, add a line for the clock itself: check sync status at the same time you review expiring tokens.
The short version
- Compare
date -uagainst an outside source. - Run
timedatectland confirm the clock is synchronized and NTP is active. - Make sure exactly one time service is running and that it has synced recently.
- Confirm DNS resolves the NTP pool and UDP 123 can get out.
- Reboot and check that the clock synced before the agent started.
- Make the agent wait for
time-sync.target, guard API-calling cron jobs, and check drift from outside the box.
It is a boring fix. That is the point. Once the clock is reliable, a whole category of mysterious token, certificate, and webhook failures stops showing up, and the failures that are left are real ones.
Turn the check into a prompt
If you want your agent to run this check for you every week and only speak up when something is off, start with MarketMai’s free AI prompt pack and adapt one into a “report clock sync status, last sync time, and drift against an outside source” check. When you are ready to run more of your business on agents, the MarketMai Ultimate Bundle puts 13 prompt, automation, and agent-operations products in one $49 download.
More from the build log
Suggested
Want the full MarketMai stack?
Get the core MarketMai guides and operator playbooks in one premium bundle for $49.
View Bundle