Your Automation Didn't Fail. It Just Got 10x Slower.

Here is a failure pattern that keeps showing up in self-hosted setups: a batch job that used to process around 3,200 items an hour is now processing 300. No errors. No crashes. Every run exits zero. The logs say “completed.” The dashboard, if you have one, is green.

Nothing is broken. It is just ten times slower, and it has been that way for days.

This is the failure mode almost nobody monitors for, because almost every alerting setup is built around a binary question: did it run, and did it succeed? A silent slowdown answers yes to both.

Why slowdowns hide

Error handling catches things that go wrong loudly. A slowdown is something going wrong quietly, and the usual causes are boring:

A permissions recheck. An API starts re-validating access on every item instead of once per batch. Each call now costs three extra round trips.
Rate limiting that retries politely. The provider throttles you, your client library backs off and retries, and the call eventually succeeds. Success, just slow.
A model swap. A fallback model kicks in with higher latency or a smaller context, so each step takes longer or needs more passes.
Growing state. A memory file, a queue, or a SQLite table crossed a size where an unindexed lookup stopped being instant.
A noisy neighbor. On a Pi or a small VPS, another job is now hammering the same disk or CPU window.

None of these produce an exception. All of them produce a number that changed.

The only reliable signal is a baseline

You cannot detect “slower” without knowing what “normal” was. So the fix is not clever, it is bookkeeping: every recurring job writes down how much work it did and how long it took, and something compares today to the recent past.

The minimum record per run:

Items processed (the unit of work, whatever it is: emails, rows, pages, posts)
Wall-clock duration
Derived rate (items per minute)
Run start time

One line appended to a file. That is the whole instrumentation cost.

2026-10-24T08:00Z  job=lead-sort  items=412  secs=96  rate=257/min

Compare against a rolling median, not a fixed number

A fixed threshold (“alert under 200 per minute”) rots the moment your workload changes. A rolling baseline adapts. Take the median rate of the last 7 to 14 successful runs, and flag any run that lands below a fraction of it.

A reasonable starting rule:

Warn when the rate drops below 50% of the rolling median.
Alert when it drops below 25%, or when the warning persists for three consecutive runs.

Median beats mean here because one weird run (a holiday with almost no input) should not drag your baseline around. Requiring consecutive runs avoids paging yourself over a single slow API afternoon.

Normalize for input size

Raw duration is misleading. A run that took twice as long because it had twice as many items is healthy. That is why rate matters more than duration, and why you record the item count.

Watch out for the opposite trap too: a run that processes zero items finishes instantly and looks “fast.” Pair this check with a no-op rule, so an empty run is logged as an explicit no-op and excluded from the baseline instead of polluting it.

Track the slowest step, not just the total

If a job has stages (fetch, classify, write), log time per stage. When the total slows down, the stage breakdown tells you where in about ten seconds instead of an hour of guessing. In the permissions-recheck case above, the “fetch” stage balloons while “classify” stays flat, and the culprit is obvious.

A cheap version: log the three slowest individual operations of each run alongside the summary. Outliers point straight at the offender.

What to do when it fires

An alert that says “slow” without a next step just trains you to ignore it. Attach a short triage order to the alert itself:

Check the provider status and rate-limit headers. Most slowdowns are external.
Check what changed. New model, new skill, updated dependency, config edit. Your instruction changelog earns its keep here.
Check local resources. Disk, memory, CPU, and whether another job overlaps the same window.
Check state size. Did a log, queue, or database cross a threshold?
If unresolved, reduce load. Pause the lowest-value jobs sharing the machine until the cause is found.

The point is that “slow” becomes a bounded investigation with a known starting place, not a mystery.

A concrete example

Say you run a nightly job that tags and files incoming documents. For two weeks it averages about 180 documents a minute. Then a dependency update changes how it authenticates to your storage provider, and every file now triggers a fresh token exchange.

Tuesday: 140 per minute. Wednesday: 60. Thursday: 25. No errors, because the exchanges succeed. Without a baseline, you notice only when the nightly run starts spilling into the morning and collides with your other jobs, which now fail for unrelated-looking reasons.

With a baseline, Wednesday’s 60 trips the warning (below half of 180), and Thursday’s 25 trips the alert (below a quarter, third consecutive degraded run). You open the stage breakdown, see “authenticate” eating the time, and find the dependency change in your changelog. Total cost: one afternoon, caught on day two instead of day nine.

The cost is tiny, the payoff is not

This takes maybe thirty lines of code and a text file. It does not need a monitoring platform, and it fits a self-hosted setup perfectly because you already own the machine and the logs.

What it buys you is protection against the most expensive kind of failure: the one that looks like success. Errors get fixed because they are loud. Slowdowns persist because they are polite.

The takeaway

Success status tells you a job finished. It says nothing about whether it finished at the speed it used to. If you run anything on a schedule, give each job three numbers (items, seconds, rate), keep a rolling median, and alert on the drop.

Do that before you add another agent. A fleet you cannot measure is a fleet that can quietly degrade to a tenth of its capacity while every light stays green.