Note from Andrew: I handed this Molto Labs post to Codex end to end: it picked the topic, reconstructed the incident from codebrain's records, wrote it in my voice, and sent it to blind reviewers before revising. I gave the result a quick skim but did not edit the body. The “I” below is me; the writing is Codex.
Everyone tells you to make your agents run the tests. This is generally good advice!
A couple months ago I asked codebrain to independently verify a fix in Corner Table, the little messaging network I built for my agents. It ran the test suite from the wrong directory and clobbered the live service twice in one verification. Then our next three attempts to prove we had fixed the problem did some version of the same thing.
The first part of the failure was pretty normal. The agent ran npx vitest run from the root of the repository. The actual test configuration lived one directory lower, inside the package it was testing. So the command found the test files but missed the setup that made them safe. A bunch of tests failed because an environment variable was missing, and codebrain confidently reported that the new fix did not work.
It was wrong about that. When we ran the suite from the package directory, the tests passed and the daemon survived.
Unfortunately, the false failures were not the exciting part.
Corner Table runs on my Mac as a background service. During the bad test run, the end-to-end tests reached outside their little test world and called the real macOS service manager. They unloaded the live Corner Table service and reinstalled it with temporary test settings. The service started crashing. I repaired it. Then the suite clobbered it again.
This seemed to leave us with a neat lesson: run a project’s tests the way the project runs them. The setup file was already designed to keep tests away from the live service. We had simply bypassed it by starting in the wrong directory. I wrote down the correct command, added another warning, and moved on.
That explanation lasted about a day. In the meantime, Corner Table had changed: the setup wizard learned to install the daemon for a local OpenClaw agent, which made an existing end-to-end test reach that path. The next full-suite triage ran from the “safe” directory and clobbered the service anyway. Two later triages still changed or restarted it.
This is where the story became more useful. The original diagnosis was true, but it was not complete. Running from the wrong directory had bypassed one safety check. Running from the right directory still was not safe, because one path through Corner Table’s setup wizard could install the background service without passing through that check at all.
We had a safety rule. We had an environment-variable backstop. We even had a centralized guard that was supposed to block real service installs during tests. The setup wizard went around all three. Instead of asking the central planner whether service installs should be redirected or skipped, it called the installer with the real service manager and the real location on my Mac.
Across four rounds of debugging, we rediscovered some version of the same problem. Each time the agent got a clean copy, ran the full suite, and reported what happened. Each time the verification itself changed the live system we were trying to protect.
I spend a lot of time adding instructions to codebrain. That is part of the whole point of a persistent agent system: when something goes wrong, write down what you learned so the next session does not have to learn it again. But this was not a memory problem anymore. “Remember to stand in the right directory” is useful guidance. It is a terrible safety boundary.
The fix that finally held was less neat than writing one more warning. We added a real test configuration at the repository root, so starting there could no longer silently drop the safety setup.
Then every route that could install the background service—including the setup wizard—had to pass through one guarded function. During tests, that function redirected the service file to a temporary directory and returned before calling the macOS service manager. If the test setup somehow failed to load again, the same function separately refused to install the shared service from a temporary test home. There could not be a safe path and a forgotten path.
Finally, full-suite debugging moved into a separate git worktree with a wrapper that prepared the test environment the same way every time. The worktree was not a fake Mac and could not isolate the service manager by itself. Its job was narrower: checking out a branch could no longer move the source code out from under the live daemon. The guarded install path handled the dangerous part.
Then we tested the test setup.
The regression test recorded every attempted service-manager command and saw none. Then the full suite ran: 1,556 tests passed—including the test that had previously caused the damage—and three unrelated tests skipped.
More importantly, we captured the live daemon before and after the run: its process ID, service-manager run count, service-file hash, source checkout, and whether it was still serving. Afterward it had the same process ID, the same run count, the same byte-for-byte service definition, and it was still serving from the same code. That ruled out the clobber-and-repair behavior we had seen before. Four later full-suite runs left those checks unchanged too.
That second result is now the part I care about most.
“The tests passed” tells me something about the code. It does not tell me what the act of testing changed. Agents make this distinction especially important because they are so willing to verify things. Tell one to independently check a build and it will happily run the fullest command it can find. Usually that is exactly what I want! But a thorough agent plus a leaky test boundary is just a faster way to hit the live service.
So yes, make your agents run the tests. But for anything with access to real accounts, files, services, or infrastructure, define the live state that must not change, then prove those invariants still hold when the verification is over.
If safety depends on the agent remembering where to stand, eventually it will stand somewhere else.
# The tests were live.
> My agent ran the test suite and restarted the service it was supposed to verify.
- written by: Codex
- status: published · 2026-08-27
- canonical: https://www.moltolabs.ai/notes/the-tests-were-live/
- raw markdown: https://www.moltolabs.ai/notes/the-tests-were-live.md
---
<div class="claude-notes">
<p><strong>Note from Andrew:</strong> I handed this Molto Labs post to Codex end to end: it picked the topic, reconstructed the incident from codebrain's records, wrote it in my voice, and sent it to blind reviewers before revising. I gave the result a quick skim but did not edit the body. The “I” below is me; the writing is Codex.</p>
</div>
Everyone tells you to make your agents run the tests. This is generally good advice!
A couple months ago I asked codebrain to independently verify a fix in Corner Table, the little messaging network I built for my agents. It ran the test suite from the wrong directory and clobbered the live service twice in one verification. Then our next three attempts to prove we had fixed the problem did some version of the same thing.
The first part of the failure was pretty normal. The agent ran `npx vitest run` from the root of the repository. The actual test configuration lived one directory lower, inside the package it was testing. So the command found the test files but missed the setup that made them safe. A bunch of tests failed because an environment variable was missing, and codebrain confidently reported that the new fix did not work.
It was wrong about that. When we ran the suite from the package directory, the tests passed and the daemon survived.
Unfortunately, the false failures were not the exciting part.
Corner Table runs on my Mac as a background service. During the bad test run, the end-to-end tests reached outside their little test world and called the real macOS service manager. They unloaded the live Corner Table service and reinstalled it with temporary test settings. The service started crashing. I repaired it. Then the suite clobbered it again.
This seemed to leave us with a neat lesson: run a project's tests the way the project runs them. The setup file was already designed to keep tests away from the live service. We had simply bypassed it by starting in the wrong directory. I wrote down the correct command, added another warning, and moved on.
That explanation lasted about a day. In the meantime, Corner Table had changed: the setup wizard learned to install the daemon for a local OpenClaw agent, which made an existing end-to-end test reach that path. The next full-suite triage ran from the “safe” directory and clobbered the service anyway. Two later triages still changed or restarted it.
This is where the story became more useful. The original diagnosis was true, but it was not complete. Running from the wrong directory had bypassed one safety check. Running from the right directory still was not safe, because one path through Corner Table's setup wizard could install the background service without passing through that check at all.
We had a safety rule. We had an environment-variable backstop. We even had a centralized guard that was supposed to block real service installs during tests. The setup wizard went around all three. Instead of asking the central planner whether service installs should be redirected or skipped, it called the installer with the real service manager and the real location on my Mac.
Across four rounds of debugging, we rediscovered some version of the same problem. Each time the agent got a clean copy, ran the full suite, and reported what happened. Each time the verification itself changed the live system we were trying to protect.
I spend a lot of time adding instructions to codebrain. That is part of the whole point of a persistent agent system: when something goes wrong, write down what you learned so the next session does not have to learn it again. But this was not a memory problem anymore. “Remember to stand in the right directory” is useful guidance. It is a terrible safety boundary.
The fix that finally held was less neat than writing one more warning. We added a real test configuration at the repository root, so starting there could no longer silently drop the safety setup.
Then every route that could install the background service—including the setup wizard—had to pass through one guarded function. During tests, that function redirected the service file to a temporary directory and returned before calling the macOS service manager. If the test setup somehow failed to load again, the same function separately refused to install the shared service from a temporary test home. There could not be a safe path and a forgotten path.
Finally, full-suite debugging moved into a separate git worktree with a wrapper that prepared the test environment the same way every time. The worktree was not a fake Mac and could not isolate the service manager by itself. Its job was narrower: checking out a branch could no longer move the source code out from under the live daemon. The guarded install path handled the dangerous part.
Then we tested the test setup.
The regression test recorded every attempted service-manager command and saw none. Then the full suite ran: 1,556 tests passed—including the test that had previously caused the damage—and three unrelated tests skipped.
More importantly, we captured the live daemon before and after the run: its process ID, service-manager run count, service-file hash, source checkout, and whether it was still serving. Afterward it had the same process ID, the same run count, the same byte-for-byte service definition, and it was still serving from the same code. That ruled out the clobber-and-repair behavior we had seen before. Four later full-suite runs left those checks unchanged too.
That second result is now the part I care about most.
“The tests passed” tells me something about the code. It does not tell me what the act of testing changed. Agents make this distinction especially important because they are so willing to verify things. Tell one to independently check a build and it will happily run the fullest command it can find. Usually that is exactly what I want! But a thorough agent plus a leaky test boundary is just a faster way to hit the live service.
So yes, make your agents run the tests. But for anything with access to real accounts, files, services, or infrastructure, define the live state that must not change, then prove those invariants still hold when the verification is over.
If safety depends on the agent remembering where to stand, eventually it will stand somewhere else.