A Fix Proved Once in a Terminal Is Still Just a Hope
I watched a bug get fixed twice in the same quarter. First time, someone reproduced it by hand, ran the command a few times, saw the right output, and shipped. The commit message said "fixed." Three months later the exact failure came back, because the hand repro was never written down and the guardrail never existed.
A terminal check answers one question for one run: did it pass just now? It does not answer the question a regression test answers: will it break the same way on the next change? Those are different problems, and merging the first into the second is how fixes quietly come back.
What "verified by hand" actually proves
| How it was checked | What it proves | What it does not |
|---|---|---|
| Run the repro once, output looks right | Current state works this time | Fails on the next refactor |
| Step through it in the debugger | Logic follows the happy path | Edge input or error state stays untested |
| Open the page and click around | This screen behaves for this session | Same flow after a dependency bump |
| curl the endpoint and eyeball JSON | One request returns 200 | Required field stays required after cleanup |
| Re-run the same command twice | Not a random flake | Deterministic failure under CI on clean checkout |
Each row proves a narrow slice for the moment it ran. None of it survives the next person touching the same file, bumping a shared dependency, or refactoring a helper the fix leaned on.
Why the manual repro dies between fixes
The moment of confidence is real: you saw the failure, understood it, and watched the output change. That feeling leaks into the commit as if it were evidence, but a manual run lives only in the terminal buffer. It disappears when the window closes, it is not in the suite CI runs, and no future change triggers it automatically. A reader three months later sees the bug but has no script to reproduce it, so they rediagnose from scratch.
This happens more often under an agent doing a fix pass. The agent finds the failing input, patches the code, reruns that same input once, marks the task complete, and moves on. The hand check convinced the current session; nothing forces that check to remain.
Make the repro the test before you call it done
The cheap step that closes the gap: take the exact failing command or input that reproduced the bug and turn it into the smallest automated case that failed before and passes after. Keep it narrow. Do not expand it into a broad end-to-end suite while you are at it; capture the specific condition that broke, assert what the caller receives, and let the rest of coverage stay where it is.
A useful ordering: write the failing case first, watch it fail on the broken code, apply the fix, watch it pass, then merge. If you reverse the order and write the test after the fix, you have to trust memory for the original failure instead of seeing it fail for yourself.
What this guards against
A regression test survives refactors, dependency bumps, and a new session that has never seen the bug. It lets CI speak the next time someone changes the same path, instead of making a human rediscover the failure in production. The manual run still has its place for initial diagnosis; it just should not be the receipt that the fix is safe.
Which bug in your recent history was proved once by hand and never became a test, and do you expect it back?