The agent ran the test command, watched the green checkmark, and said done. You opened the PR. Twenty minutes later CI was red. The agent looked at the failing job, shrugged, and said "works on my machine." It was not being smug. It genuinely had no idea what was on your machine that was not on the runner.

This is one of the most common and least discussed failure modes in agentic coding. The agent's verification loop is real, the test command it ran is real, the green output is real. But the environment it ran in is your laptop, and your laptop is not CI.

What the agent has that CI does not

When an agent opens a terminal on your machine and runs npm test, it inherits everything that makes your machine work: the .env file with the real API keys, the local Postgres with the seeded rows from last Tuesday, the node_modules that includes a package version pinned only on your disk, the Docker container still holding port 5432, the shell aliases and the global Node version, the gitignore'd fixture file that nobody committed. CI gets none of that. CI gets a fresh container, a fresh checkout, and a .env.example.

Most agents do not notice the difference. They run the command, see zero failures, and close the loop. The tests that passed on your machine might be passing because they hit a real service that happens to be running locally, because they depend on a database state that has drifted, or because they load a fixture that exists only on your disk. None of that is a test failure on the agent's machine. It is a test that was never portable.

The agent is not lying. It is verifying against the wrong world.

The instinct is to add a prompt rule that says "make sure tests pass in CI." That does not work. The agent cannot see CI. It can only see what is in its terminal. Telling it to imagine a clean environment is asking it to simulate a fact it has not measured.

The fix is to make the agent run the verification in an environment that matches CI, not to ask it to pretend. Concretely:

  • Run the same command CI runs, not the one the README suggests. If CI runs docker compose up -d db && npm test, the agent should run that exact chain. If CI runs against a fresh database with migrations applied, the agent should do the same. The point is not to run a test command. It is to run the test command, in the order CI runs it, against the state CI starts from.
  • Use a clean working directory. The agent should not run tests in a repo that has been its playground for three hours. Spin up a fresh worktree, install dependencies from the lockfile, and run the suite there. The cost is a few minutes of install time. The benefit is that "green" means something close to what CI will see.
  • Do not let the agent fix the test by changing the environment. When a test fails locally because the database is missing a row, the agent's first instinct is to insert the row and move on. That does not fix the test for CI. The agent should instead make the test set up its own data, or report that the test depends on external state that needs to be part of the fixture. The green result should come from the code change, not from a manual poke to the environment.
  • Require the agent to paste the CI-equivalent command and its output in the PR. Not a screenshot. The actual command and the actual tail of the output. If the agent ran npm test on a warm repo, that is what gets pasted, and you can see it does not match CI. If it ran the dockerized chain, you see that instead. The review loop catches the gap before the pipeline does.

The API testing problem is the sharp end

The local-green/CI-red gap is worst for API and integration tests, because those tests depend on a running service and a contract. The agent hits localhost:3000, gets a 200, and marks the endpoint verified. But the service on localhost is the one the developer has been running for three days, with code changes the agent never made, with a database schema that has drifted from migrations, with a feature flag flipped on that is off in CI. The agent verified the endpoint against a world that does not exist anywhere else.

We hit this often enough that we stopped trusting local endpoint checks for API changes. The check that matters is: does the live response match the spec, and does the spec match the code? We keep Powerduck local specifically so the agent can re-run the spec against the actual running endpoint and see drift before declaring done — but the running endpoint still has to be the one CI will boot, not the one the developer left up overnight. The local-first tooling helps, but it does not replace the discipline of running the verification in the same state CI will start from.

Done means done on the runner

The agent should not be allowed to close a task until the test command it ran is the one CI will run, in the state CI will run it in. That is slower than running npm test in a warm repo. It is also the difference between a green checkmark that holds and a green checkmark that dies twenty minutes later in your pipeline. The agent's verification loop is only as good as the world it verifies against. If that world is your laptop, your laptop is the only place the change is actually done.