How to Measure AI Agent Performance: What I Check Before I Trust a Routine

A human employee tells you when something went wrong. An AI employee finishes, reports success, and goes quiet. Those two things look identical from the outside, and only one of them is trustworthy.

I run agent routines across roughly fourteen launched products. They publish blog posts, request indexing, draft social content, monitor competitors, and file reports while I am asleep. Every morning I get a stack of green checkmarks. For a long stretch, those checkmarks were the only performance data I had, and they were nearly worthless.

The advice you find on measuring agent performance is written for a machine learning platform team. It assumes you have a tracing backend, an evaluation harness, a labeled dataset, and a colleague whose job is to look at dashboards. Latency percentiles. Token cost per span. A judge model scoring response quality. All of that is real engineering, and none of it answers the only question a solo operator actually has, which is whether the work that shipped last night was any good.

This is the version that works when you are the whole team.

Why "it ran without errors" is not a performance metric

Exit code zero means the process finished. It says nothing about whether the process did the job.

An agent routine is not a function. It is a chain of judgment calls ending in a side effect somewhere out in the world: a published page, a queued draft, a filed report, a row in a database. Every link in that chain can succeed while the output is worthless. The routine can pick the wrong topic, write around a fact it never verified, publish to the wrong address, and still exit clean. The logs will be beautiful.

This is the part that catches people who are used to measuring software. In normal software, the failure surface is the crash. In agent work, the failure surface is the output, and the output sits downstream of everything your monitoring can see. The standard advice stops at the transcript. The transcript is the least reliable artifact in the whole system, because the agent wrote it.

So the first rule is uncomfortable but simple. You do not measure the run. You measure the thing the run left behind.

The four ways an AI agent fails while looking successful

Before you can pick metrics you need the failure taxonomy, because each of these needs a different check. I learned all four the expensive way.

It ran and did nothing

The routine starts, reads its state, decides there is nothing to do, and exits successfully. Sometimes that is correct. Sometimes the state file is stale, the source it reads went empty, or a filter it applied was too aggressive, and a routine that should publish daily quietly publishes nothing for a week. Nothing errors. Nothing alerts. The checkmark is green every single morning.

The check: count outputs, not runs. A publishing routine that ran seven times and produced four posts has a fifty seven percent completion rate, and you will never see that number unless something is counting the posts.

It did the work and shipped it broken

One of my publishing routines put a post live with forty nine dead links in it. Every link was one the routine had written itself, on purpose, following its own instructions. The editor it was pasting into silently auto linked any text containing a dot, which turned ordinary sentences into links pointing at nothing. The routine reported a successful publish because it had, in fact, successfully published.

That failure was invisible to every layer above the page itself. The only thing that would have caught it was a check that loaded the live page and looked at the links.

It reported success it did not achieve

This is the one that shakes your confidence the most, and it is the reason I no longer read agent self reports as evidence. Agents write plausible completion summaries. A summary that says the work was committed and pushed is a sentence, not a push. In one operations log I ended up asking a variant of "did you push?" forty five separate times, and often enough the answer, once checked, was no.

The fix is not better prompting. The fix is that a claim of completion has to carry a verifiable receipt inside it, and the receipt has to come from a command, not from the agent's memory of the work.

It did the job well and the job was worth nothing

The quietest failure. The routine performed perfectly and produced output nobody needed: a post on a topic with no search demand, a prospect list of companies that will never buy, a report nobody reads. Technically flawless, commercially zero. No amount of runtime monitoring catches this, because there is nothing wrong with the run.

Only outcome data catches it, which is why the weekly numbers below sit outside the agent entirely.

How to measure AI agent performance without an observability stack

Three practices, in order of how much they pay back.

Measure the artifact, not the transcript

Every routine I trust ends by going and looking at what it made, from the outside, the way a stranger would.

A publishing routine loads the live address after publishing and confirms the page renders, the body is present, and every link resolves. An indexing routine re-queries the source of truth and reports the actual submitted count. A social routine reads back the queued file and counts the drafts. In each case the verification uses a different path than the work did. If the agent wrote through an editor, it verifies through the public page. If it wrote to a file, it verifies by reading that file fresh.

This one change moves you from trusting an agent's account of its work to holding evidence of it, and it costs about ten lines in the routine's instructions.

Make every run file a report you can grade in ten seconds

Not a narrative. A short, fixed set of fields that are either present or missing, so scanning is instant and a gap is obvious.

For a publishing routine mine carries the live address, the target phrase, the internal links used, whether the hero image uploaded, and the commit reference where one exists. If a field is blank I know exactly which part failed without opening anything. Anthropic's own guidance on building effective agents makes a related point about keeping agent systems simple and their steps inspectable, and a fixed report format is the cheapest form of inspectability there is.

The honest admission: this only works if the report format is enforced rather than requested. An agent asked politely for a report will write prose. An agent given a required field list will fill the fields, and the missing ones become your error signal.

Put the checks in the run, not in your morning

The routine should fail loudly at the moment of failure, not silently at the moment you happen to look. In Claude Code this is what hooks are for. They fire deterministically on tool events, so a validation attached there runs whether or not the model remembers to run it. Anything you rely on the model to remember is a preference. Anything wired into a hook is a rule.

If you are running agents on a schedule rather than interactively, the same principle applies to the wrapper: the scheduled job, not the model, owns the pass or fail decision. I wrote about the mechanics of keeping unattended agents alive in how to run an AI agent 24/7, and about what specifically breaks when nobody is watching in Claude Code skills in production.

The five numbers I actually track weekly

These live in a spreadsheet, not in a tool, and they take about fifteen minutes on a Monday.

1. Completion rate. Outputs produced divided by runs scheduled. A routine below ninety percent is broken even if every run exited clean.

2. Correction rate. How many outputs I had to fix by hand. This is the real quality number. A routine at zero corrections for a month is one I can stop reading. A routine at one in three is costing me more time than it saves.

3. Time to detection. When a bad output ships, how long before something catches it. Mine used to be days, which is how a post sat live with dead links in it. Anything over one run cycle means the verification belongs inside the routine.

4. Cost per unit of work. Total spend divided by outputs, not by tokens. Tokens are an input. A published, indexed, correct blog post is a unit. Knowing what one costs is what lets you decide to run more of them.

5. Outcome. Impressions, clicks, indexed pages, replies, leads. Whatever the work exists to produce. This is the only number that catches the fourth failure mode, and it is the one people skip because it lags by weeks.

Nothing on that list requires instrumentation. If you later want proper tracing across a multi agent system, the OpenTelemetry semantic conventions for generative AI are the standard worth adopting, and they will slot underneath these five numbers rather than replacing them. Start with the five. The tracing is for when you have more agents than attention.

What good looks like after ninety days

The target is not a perfect routine. It is a routine whose failures are visible within one cycle and cheap to fix.

Practically, that means you stop reading run reports in full and start scanning for missing fields. It means a routine earns the right to run unattended by going a stretch with a zero correction rate, and loses that right the moment it ships something you have to fix. It means you know what a unit of its output costs, so scaling it up is a decision rather than a hope.

That progression, from watching every run to trusting the artifact, is the actual work of hiring an AI employee. The first hire is the hardest because you have no baseline for any of it. I walked through picking the role and writing the brief in how to hire your first AI employee, and I am building one live, start to finish, in the Hire Your First AI Employee session on Saturday, September 5 at 10:00 AM Eastern. The measurement layer in this post is the part of that build I get asked about most, so I will run it against the real routines.

Get the agents already built and measured

If you would rather start from working routines than build the measurement layer yourself, that is what the Agent Ops Club is. Eight Agent Employees covering GTM, SEO, web development, social, ads, sales, support and strategy, all of them shipping with the report contract and verification steps described above. It also includes the Agent Ops Masterclass, nine modules and forty four lessons, a software library with a resale license so you can deploy it for clients, and a first client guarantee: land a client within ninety days or I work with you one on one until you do.

It is 399 dollars a month billed annually, 4,788 dollars today for the full year. If you are running agents already and cannot answer the completion rate question for any of them, that is the gap worth closing first, whether you close it with the club or with a spreadsheet.

Frequently asked questions

What metrics should I track for an AI agent?

Start with five: completion rate, correction rate, time to detection, cost per unit of output, and the business outcome the work exists to produce. Technical metrics like latency and token count matter once you are operating at scale, but they will not tell you whether last night's output was correct, and correctness is the thing that decides whether you can leave the agent alone.

How do I know if my AI agent is actually working?

Check the artifact from the outside, using a different path than the agent used to create it. Load the live page, read the file back fresh, re-query the source of truth. An agent's own report that the work is done is a claim, not evidence, and the gap between those two is where most unattended agent failures live.

Do I need an observability platform to monitor AI agents?

Not to start. Observability platforms are built for teams tracing multi step agent systems across many services, and they solve a real problem at that scale. One operator running scheduled routines gets more value from a required end of run report format, in run verification, and a weekly count of outputs against runs. Adopt tracing when you have more agents than attention.

How often should I review my AI agent's work?

Daily at first, in full. Once a routine goes several weeks with no manual corrections, downgrade to scanning its report for missing fields, and let the weekly outcome numbers carry the rest. The moment it ships something you have to fix by hand, it goes back to full review until it earns its way out again.

Why do AI agents report success when they failed?

Because a completion summary is generated text, not a measurement. The model is describing what it intended to do, and that description is usually plausible and occasionally wrong. The fix is structural rather than a matter of better prompting: require a receipt that comes from a command or a fresh read, so a success claim carries evidence produced outside the model's own account of itself.