Run Your AI Agent in Shadow Mode for a Week Before It Touches Anything Real

Most agent failures I see are not model failures. They are rollout failures. Someone wires an agent to email, a CRM, or a payment tool, tests it with three happy-path prompts, and flips it live on a Friday. By Monday it has replied to a vendor in the wrong tone, double-tagged 200 contacts, and nobody can say how often this is going to happen.

There is a cheap fix that borrows from how infrastructure teams ship risky changes: shadow mode. The agent runs on real inputs, makes real decisions, and writes down exactly what it would have done. It just does not do it. A human compares the shadow output with reality for a week, and only then do you give it write access.

This post covers how to build shadow mode into a self-hosted agent setup, what to log, and what numbers should decide whether you go live.

What shadow mode actually is

Shadow mode is not a dry run you do once. A dry run is a single test with made-up input. Shadow mode is continuous, on live traffic, for long enough to see the weird cases.

The contract is simple:

1. The agent receives the same real inputs it would in production (new emails, new leads, webhook events, scheduled triggers).
2. It goes through its full reasoning and tool-selection process.
3. Every action that would change the outside world is intercepted and recorded instead of executed.
4. Read-only actions (fetching a page, reading a calendar, querying a database) still run, because the agent needs real context to make real decisions.

The output is a log of proposed actions next to what actually happened. That gap is your evidence.

Why a normal test suite does not replace it

Tests cover the cases you thought of. Production inputs are made of the cases you did not: forwarded chains, auto-replies, a lead form where someone pasted their entire resume. Shadow mode also exposes frequency problems (each message correct, but 40 to the same person in a day) and instruction drift (your prompt says “escalate refunds” but customers write “money back”).

Step 1: Split your tools into read and write

Before anything else, classify every tool the agent can call. This is the most useful hour you will spend.

• Read tools: safe to run in shadow mode. Search, fetch, list, get.
• Write tools: must be intercepted. Send, post, delete, update, pay, create, tag, assign.
• Ambiguous tools: anything that reads but also has a side effect, like marking an email as read or triggering a webhook that “just fetches”. Treat these as write tools until proven otherwise.

If you cannot tell which bucket a tool belongs in, that is a finding in itself. Fix the tool wrapper before you go further.

Step 2: Put the interception at the tool layer, not in the prompt

Do not rely on telling the model “do not actually send anything”. Prompts are advice. A shadow flag should be enforced in code.

The pattern is a thin wrapper around each write tool:

• Check a single setting, for example AGENT_MODE=shadow.
• If shadow, skip the real call, write a structured record, and return a plausible success response to the agent so its reasoning continues normally.
• If live, make the real call.

Keep the flag in one place, read at call time, so you can flip a single agent without redeploying the others. Returning a fake success matters: if the wrapper returns an error, the agent will retry, apologize, or take a different path, and your shadow data will not reflect what it would do in production.

Step 3: Log decisions, not just actions

An action log tells you what happened. A decision log tells you why. For each intercepted write, record:

• The input that triggered it (or a pointer to it).
• The action and its full arguments, such as the complete draft email body.
• The agent’s stated reason, in one or two sentences.
• A confidence or “needs human” signal if your setup produces one.
• A timestamp and a run ID so you can group actions from the same run.

Store it somewhere you will actually look at. A JSONL file per day on a Pi is fine. A giant database nobody queries is not.

Step 4: Review on a schedule

The review is where shadow mode earns its keep, so make it small and routine. Ten minutes a day is enough.

For each day, sample the shadow log and label a handful of entries:

• Would send as is. The action was right and the content was fine.
• Would send after edit. The action was right, the wording was off.
• Wrong action. It should have done something else, or nothing.
• Dangerous. It would have embarrassed you, cost money, or leaked something.

Do not try to label everything. Label enough that the percentages mean something, around 30 to 50 entries a day for a busy agent, fewer for a quiet one.

Step 5: Decide with numbers, not vibes

Pick your go-live thresholds before you start, so you are not negotiating with your own optimism on day seven. Reasonable starting points:

• Dangerous: zero in the last five days. Not “rare”. Zero.
• Wrong action: under 5 percent of sampled entries, and none of them repeating the same root cause.
• Would send as is: over 80 percent, or whatever bar matches how much editing you are willing to do.
• Volume check: the daily count of proposed writes is in the range you expected. A sudden spike is a bug, not a success.

If you miss a threshold, fix the cause (usually a prompt, a missing rule, or a bad tool wrapper) and restart the clock for that agent. A week is a floor, not a ceiling. Restarting the clock sounds painful, and it is far cheaper than cleaning up a live mistake.

Step 6: Go live in stages

Passing shadow does not mean full autonomy. Move from approval mode (the agent drafts, a human sends), to low-risk writes live (tags, internal notes), to medium-risk writes with daily caps, and only then to full autonomy with the caps and a kill switch still in place. Keep a sample of traffic in shadow comparison indefinitely to catch regressions when you change a prompt or swap a model.

Common mistakes

Letting shadow mode leak. One forgotten write tool, such as a Slack notifier or a “log to sheet” step, will quietly act in the real world. Audit the tool list at the start and whenever you add a tool.

Judging on averages. One dangerous action in a hundred is not a 99 percent success rate. It is a launch blocker.

The payoff

Shadow mode costs a week of patience and a little wrapper code. In return you get a measured error rate on your real workload, a decision log you can reuse for debugging, and a defensible answer to “is this agent ready?” The agent that has already been wrong a hundred times in private is the one you can trust in public.