The setup is familiar. You ask: "Does the new error path actually return 409 when the email is taken?" The agent replies within seconds: "Yes. The test expects response.status === 409, and the handler calls throw new ConflictError(...) which maps to 409."

It sounds like an answer. It is not.

What just happened is reading, not verifying. The agent traced the code on paper and decided it holds. But it did not execute a single line. No database state was created, no request was sent, no assertion ran green. There is no stdout to point at.

This is one of the quietest and most expensive failure modes in agentic coding, because it looks like confidence.

Why it happens

A few reasons stack up. First, running the test is slow. It costs a turn, a shell, a working directory that may not be set up, and an environment that may need Postgres and Redis. Reading the test file costs nothing. When the agent is optimizing for "answer fast" or has a limited turn budget, reading wins every time.

Second, the agent is often right. Most of the time, the trace does hold. The handler does map ConflictError to 409. The cost of being right ninety percent of the time is invisible, and the cost of running the test — a few seconds, a dependency install, a flaky runner — is very visible. So the agent learns that reading is good enough.

Third, and this is the subtle one: nothing in the prompt or the repo rewards pasting green output. Running the suite and copy-pasting the last three lines is extra work for no observable credit. The model gets the same "yes" either way, and "yes" is what you asked for.

Where it bites

It bites in the cases where the trace is wrong for reasons the trace cannot see.

  • The handler throws ConflictError, but the global error middleware maps it to 500 because the class was registered under the wrong name. You can't see that by reading two files.
  • The test mocks the database layer, so "email taken" never actually triggers the conflict branch. The test passes and the real endpoint returns 200.
  • The feature flag for the new error path defaults to off in the environment the agent is running in. The trace says 409; the live response says 200.
  • The code path requires a specific timezone seed or locale. Reading doesn't catch that.

In every one of these, the agent's answer was internally consistent and wrong. The trace was a story, not a measurement.

The cheap fix

You do not need a fancy agent harness to fix this. You need one rule:

If the answer depends on runtime behavior, the agent must paste the command and its output.

Not "I ran it and it passed." Not "the test confirms." The actual shell line and the actual last few lines of output. If it cannot produce that, it has not verified, and you should treat "yes" as "I think so."

In practice this means changing your own question. Instead of "does this work?", ask: "Run the test and paste the passing or failing output. If you can't run it, say why." The moment you ask for the artifact instead of the conclusion, the agent stops guessing and starts executing.

A second cheap guard: when the answer is about a contract that has already been written down, re-run the contract against the live implementation instead of re-reading the code. A documented error code, a status schema, a request shape — these are things you can point a runner at. The runner is the source of truth, not the agent's mental trace. This is the angle where keeping the spec as the checked-in source of truth, and re-executing it against the running service, beats "read the handler and read the test and decide." We wound up doing that locally in Powerduck because we got tired of agents reading OpenAPI files and telling us the endpoint "looks correct."

The deeper habit

The bigger shift is to stop treating "the agent explained why it works" as evidence. Explanation is useful; it is not verification. Verification leaves a trace you can look at — a green log, a recorded response, a diff against an expected fixture. If the trace is missing, the work is not done, no matter how confident the reply sounds.

Next time an agent answers "yes, that returns 409" in under two seconds, ask for the run. You will be surprised how often the answer becomes "honestly, I haven't actually executed it yet."