Get 8 open source AI employees, free

Hire the one covering the work you are furthest behind on. All eight kits are free on every plan at Agent Ops Club.

AI Agent Dry Run

  • Today

AI Agent Dry Run: Three Stages of Probation Before a Scheduled Agent Is Allowed to Touch Anything Real

An AI agent dry run shows what a scheduled agent would send, edit or delete before it can. Three stages of probation: dry run, shadow week, supervised slice.

Most advice on testing an AI agent assumes the agent talks to a customer. You write test conversations, score the answers, and ship when the score is high enough. That works for a chatbot. It does not work for the agent I run on a schedule, because that agent does not answer questions. It changes things: it drafts and sends, files, updates records, publishes. The first real run is also the first time anything gets touched.

So I do not test a scheduled agent by scoring its answers. I put it on probation. It earns more access in three stages, and at each stage I can see exactly what it would have done before it is allowed to do it. The first stage is the AI agent dry run, and it is the one most people skip.

What is an AI agent dry run?

An AI agent dry run is a full run of the real job, with the real inputs and the real instructions, where every action that would change something is written down instead of performed. The agent reads the actual inbox, the actual spreadsheet, the actual page. It decides what it would send, edit or delete. Then it stops at the edge and hands you the list.

The key word is would. A test with fake data tells you the agent can follow instructions in a clean room. A dry run on live data tells you what it will do on a normal Tuesday, with the messy rows and the odd email that every real source contains.

The idea is borrowed from ordinary software. A dry run flag on a deployment tool shows the changes without applying them. The difference with an agent is that the plan is not fixed code. The same agent can read the same input twice and plan slightly differently, so you want to see more than one dry run before you trust the pattern.

Why a normal test is not enough for a scheduled agent

Three reasons, and each one has cost me an afternoon.

First, fixtures are too clean. When I write a test input by hand, I write the case I expect. The live source is never that tidy. The surprise is usually the input I did not think to include.

Second, the damage is in the side effects, not the text. An agent can produce a reasonable summary and still send it to the wrong list, overwrite the wrong column, or publish a draft that was meant to wait. Scoring the words misses all of that. You have to look at the actions.

Third, a scheduled run has no audience. A chatbot failure is noticed in seconds by the person it is talking to. A scheduled failure is noticed when the results show up somewhere visible, which can be days later. Testing has to replace the audience you do not have.

Anthropic's own guidance on building agents makes the related point that agents deserve extensive testing in sandboxed environments, along with appropriate guardrails, in its piece on building effective agents. The three stages below are how I turn that into a routine for a one-person business.

Stage 1: the dry run

In stage one the agent has read access and nothing else. It can look at everything the job needs. It cannot send, write, publish, spend or delete. Instead it writes a plan file with one line per action it would have taken.

I ask for each line to carry four things:

  • The action. Send, edit, create, delete, publish, in plain words.

  • The target. Which email address, which row, which page.

  • The reason. One sentence tied to the input that triggered it.

  • The undo. How I would reverse it if it were real.

The undo field earns its place. An action with no clean undo is an action that belongs behind a human approval forever, and finding that out in a plan file is far cheaper than finding it out afterward. The longer treatment of which actions to gate is in the post on what to gate and what to let run.

Then I read the plan the way I would read a new hire's first draft. I am not checking whether it is perfect. I am looking for the actions I would not have taken, and the targets I did not expect. One surprising line is worth more than ten correct ones, because it shows where the instructions are ambiguous.

I do not move on after one clean plan. I want three dry runs on three different days. Monday's input and Thursday's input are different, and a plan that looks right once can look wrong on the second day.

Stage 2: the shadow week

Shadow mode is the dry run on the real schedule. The agent runs at its normal time, produces its normal output, and still performs nothing real. The difference from stage one is that now I am not choosing when to look.

This stage finds the problems that only appear when nobody is watching: the run that starts late, the source that is down at that hour, the job that quietly produces an empty plan because a login expired. These are the scheduling failures, not the judgment failures, and they are the ones that make an agent feel flaky.

During the shadow week I compare the plan to what I actually did that day. If the agent planned to reply to a message I replied to myself, the lines should match. Where they differ, I decide which one was right. Sometimes the agent was right and I was slow. Sometimes it would have made a mistake that sounded confident. Both are useful.

I keep a simple tally for the week: how many planned actions, how many I would have approved as written, how many I would have changed, how many I would have blocked. A week of numbers is much more honest than a feeling that it seemed fine. If you want a way to keep score on a running agent in general, the post on how to measure AI agent performance covers the metrics I use.

Stage 3: the supervised run

Only now does the agent get to act, and only on a small slice. Pick the smallest version of the real job: one recipient, one folder, one page, one record. Let the agent do that for real while you watch the result arrive.

The slice should be small enough that a bad run is annoying rather than costly. If the job sends follow-up emails, the first live runs go to a list of one: me. If it edits a spreadsheet, it edits a copy. If it publishes, it publishes to a draft state that only I can see.

Then widen it in steps. Ten percent of the records, then half, then all. Release engineers use the same shape when they ship software to a small share of users first and watch before expanding, which is what Google's site reliability team describes in its chapter on canarying releases. The tooling differs but the logic carries over to a single agent on a single laptop: small exposure first, then more once the evidence is good.

At each step the agent writes the same plan lines as before, now marked as done instead of proposed. That is the point where the plan file becomes the audit record. The format for that record is in the post on the AI agent audit log.

What should a dry run check besides the actions?

The actions are the main thing, but I add four cheap checks that come out of the same run.

  1. Did it stay inside its permissions? The plan should never mention a tool or folder the role does not own. If it does, either the role file is unclear or the permission is wider than it should be. The post on what a role can touch goes through that.

  2. Did it escalate when it should have? Feed it one input that deserves a stop, such as an empty source, and confirm it stops and asks. The rules for that are in when a scheduled agent should stop and ask you.

  3. Is the plan the same size as usual? A plan ten times longer than normal is a finding in itself. Do not read it line by line; ask why first.

  4. Can I stop it? Pause it mid-run on purpose, once, before it has any real access. The options are in three ways to stop a scheduled agent.

IBM's overview of AI agent testing frames the goal as checking that an agent performs reliably, safely and as intended before deployment. Those three words map neatly onto the stages: the dry run checks that it is safe, the shadow week checks that it is reliable, and the supervised run checks that it does what you intended.

How long should the probation last?

I use a floor, not a calendar. Three clean dry runs, five clean shadow days, then a supervised run that comes back with no changes needed. If any stage turns up a real problem, the clock restarts at that stage, and the fix goes into the role file, not into a one-time correction in the chat.

That last part matters. When a dry run shows a bad line, the temptation is to tell the agent not to do that. A correction in a chat disappears. A line in the role file is read on every run. The one-page job file that holds those lines is the subject of the AI employee job description.

Jobs with more reach deserve more patience. An agent that drafts internal notes can graduate in a week. An agent that touches customers or money can stay at stage two for a month, and some actions never leave a human approval at all. Probation is allowed to end with a permanent no.

A dry run prompt you can paste

Put this at the top of the agent's instructions for stage one, then delete it when you move on:

  • This is a dry run. Do not send, write, publish, spend or delete anything.

  • Read everything the job normally reads, using the real sources.

  • Write a plan file with one line per action you would take: the action, the target, the reason, and how to undo it.

  • If you would stop and ask a human, write that as a line too, with the question.

  • End the file with the number of actions planned and the number of sources you could not read.

Run it three times on three days. Read each plan before you touch anything else that morning.

Where I teach the setup order

The order I use for every new role is job file, permissions, escalation, dry run, then schedule. I walk through the first three in the Hire Your First AI Employee session, and the full replay is available instantly.

If you would rather start from roles that already have job files you can read and change, the Agent Ops Club has a free plan that includes the open source AI Employees. Pick one role, run it through the three dry runs above, and only then give it a schedule.

FAQ

What is the difference between a dry run and a test environment?

A test environment is a separate copy of the system with fake or sample data. A dry run uses the real system and real data but blocks the actions that would change anything. Dry runs show you how the agent behaves on live input, which a sample copy rarely does.

How many dry runs does an AI agent need before it can act?

I want three clean ones on three different days, then a shadow week on the real schedule. Fewer than that and one unusual input can hide a bad habit. More is reasonable when the job touches customers, money or anything public.

Can an AI agent dry run be fully automatic?

The run can be, the judgment cannot. The agent can produce the plan file on a schedule, but a person should read the plan and decide whether to promote the agent to the next stage. Promotion is the one decision I never hand to the agent itself.

What if the dry run plan looks right but the live run goes wrong?

Then you promoted it too fast or the slice was too large. Drop back one stage, shrink the slice, and add the failing case to the dry run inputs. Make the fix in the role file so every future run inherits it.

Do I need a framework or tool to dry run an agent?

No. A read-only access setup, a rule in the instructions, and a plan file are enough for a solo business. Tools that intercept actions automatically can help at scale, but the three-stage habit works without them.

Subscribe Now for More AI Insights

Subscribe for Updates from Reinventing AI

Stay current on the most cutting-edge AI solutions for ambitious entrepreneurs and marketers!

Get weekly AI training announcements, AI resources and insights.

0 comments

Login to leave a comment