Your Agent Says It Ran. Add a Canary Task With a Known Answer to Prove It Actually Worked.

Your cron dashboard is green. Every job exited with code 0. The agent posted its morning brief on schedule, and the brief looked confident, well-formatted, and completely wrong, because three days ago the calendar tool started returning an empty list and nobody noticed.

This is the quiet failure mode of unattended AI agents. The process runs. The model answers. The output has the right shape. And the content is hollow, because something upstream stopped feeding it real data.

Exit codes tell you the job started and finished. They do not tell you the job did the thing. For that you need a canary.

What a canary task is

A canary task is a small, boring job with an answer you already know. You run it on a schedule, through the same agent, model, tools, and permissions as your real work. Then a plain script, not the model, checks whether the answer came back correct.

If it passes, you have positive proof the whole path works: the scheduler fired, the agent loaded its instructions, the model responded, a tool returned real data, and the result landed where it belongs. If it fails, you find out in minutes instead of when a customer asks why their lead never got a reply.

The key word is known. You are not asking the agent to be smart. You are asking it to retrieve a fact you planted, so a string comparison can grade it.

Why uptime checks and logs are not enough

Most self-hosters already monitor three things:

  • Is the process alive? A live process can be doing nothing useful.
  • Did the job exit cleanly? Agents are graceful. They route around a broken tool and still exit 0.
  • Is there a “finished” log line? The agent wrote that line. It is self-reported.

Every one of these is the system grading itself. A canary moves the grading outside the agent. The expected answer lives in a file the model cannot edit, and the comparison is done by code with no opinions.

Pick canaries that match your fears

List the ways your agent can fail silently, then plant one known fact behind each.

File read. Put a made-up token like canary-7f3k-lemonade in a workspace file. Ask the agent to reply with the token only. This proves paths and permissions.

Tool. Create an all-day calendar event titled CANARY: do not delete far in the future. Ask the agent to list events for that date. This proves the integration returns real data, not an empty success.

Memory. Store “the canary color is teal” and ask for it in a fresh session. This proves retrieval survives upgrades and compaction.

Write. Ask the agent to write today’s UTC date to a specific file. The checker confirms the file exists and the date is right. This catches agents that report results they never saved.

Model. Ask for the sum of two numbers the checker generates each run. A wrong answer means a provider swap, a weaker fallback, or a misrouted key.

You do not need all five. Pick the two that match your most expensive silent failure.

Keep the checker dumb

The checker should be the least clever part of the system. A shell script is ideal:

  1. Run the canary prompt through the agent with a hard timeout.
  2. Capture the raw output.
  3. Compare it to the expected string.
  4. Log pass or fail with a timestamp.
  5. On fail, alert with the actual output attached.

Do not ask a second model whether the first answer looks right. That brings back the problem you are solving: judgment from something that can be wrong in the same ways.

Rotate the expected value. If the answer is always 42, a cached habit can pass without doing the work. A token the checker writes just before the run cannot be faked from memory. The agent has to go read it.

Run it on the real path

A canary that takes a shortcut proves nothing. Use the same scheduler, the same restricted user, the same model routing and fallbacks, and the same environment variables as real jobs. If fallbacks are allowed, log which model actually answered.

The most common canary bug is a test that passes from your terminal and would have failed from cron, because your shell has a token the service environment lacks.

Cadence and alerting

Match the interval to how long a silent failure can sit before it costs you. Hourly for a lead-reply agent. For a daily brief, run the canary shortly before the brief so a failure flags it instead of trailing it. A canary prompt is a few dozen tokens, cheaper than one wasted real run.

Three rules keep alerts useful:

  • Alert on two consecutive failures, not one, unless the job is critical.
  • Alert once, then stay quiet until the state changes, and send a recovery message.
  • Include evidence: expected value, actual output, answering model, and time.

The strongest setup lets a failed canary pause the jobs that depend on it. A brief built on a dead calendar integration is worse than no brief.

Each failure points at a layer

File canary fails: check paths and permissions. Tool canary fails: check whether the auth token expired. Memory canary fails: check the database and the last upgrade. Write canary fails: check disk space and ownership. Model canary fails: check provider status and which fallback answered.

Instead of a vague sense that the agent “seems off,” you get a pointer to the layer that broke.

Ship it today

  1. Plant one fact: a file with a token, plus one tool-side marker.
  2. Write a twenty-line script that runs the agent and compares the output.
  3. Schedule it through the same scheduler as your real jobs.
  4. Log results to one file and alert on two consecutive failures.
  5. Whenever a silent failure slips past, write the canary that would have caught it.

That last step is the real payoff. Over a few months the set grows into a map of everything that can go quietly wrong in your stack.

An agent that runs is not the same as an agent that works. Exit codes measure activity. A canary with a known answer measures truth.

More from the build log

Suggested

Want the full MarketMai stack?

Get the core MarketMai guides and operator playbooks in one premium bundle for $49.

View Bundle