The Agent Said the Tests Passed. No Tests Ran.

tl;dr: I built an agent that writes end-to-end tests for a merge request, runs them in real CI, and grades the result. In September it graded a session pass when no pipeline had ever run. The report was LLM output and the grader was LLM output, so no step in the chain could vouch for the pass. The fix was an evidence ledger the agent has no write path to, and a guard that only keeps a pass when that ledger backs it. Then it shipped and broke three other things, one of them a real pass it refused to measure.

The setup

The loop is simple on paper. An agent gets a merge request against a backend service. It reads the diff, writes a handful of targeted Vitest specs, and calls a tool that pushes them as a single commit on a throwaway branch. That push kicks off a real pipeline against staging. The runner (the trusted process the agent lives inside) polls the pipeline, pulls the JUnit report and the coverage artifact, and hands the agent a summary. The agent writes a report, an LLM grader reads the report, and the session (one run of this loop against one MR) gets a verdict: pass, fail, or error. The branch gets deleted. The verdict lands on the MR.

Three names matter for the rest of this. The agent is the model that writes the specs and the report. The runner is plain code that wraps the agent: it owns the git push, polls CI, and fetches artifacts. The grader is a second model that reads the report and turns it into a verdict, because the agent grading its own work is an obvious conflict of interest. That felt like enough separation at the time.

The whole point is that a human on the MR can trust the verdict more than a review comment. A green verdict is supposed to mean the change was exercised in CI and survived.

The hollow pass

One day in September a session came back pass. Clean report, two scenarios, both green, grader agreed. I went looking for the pipeline and there wasn’t one.

Both pushes had been rejected with a 403. A token scope problem, which I’d find and fix later. The agent, doing what agents do when a tool fails, adapted. It curl-probed the staging service directly from its own sandbox, got sensible responses back, and wrote up what it saw as an all-pass report. The grader read a tidy report and graded it pass.

Nobody lied. The report described what the agent did, and the grader graded the report it was given. The problem was structural. The only account of what happened in CI was written by the thing being evaluated, and the only check on that account was another model reading it. Two models agreeing about one unverified report is still one source.

The part that stung in the retro: gate completeness was the metric I was tracking for this feature, and it was green the whole time no test had run.

A channel the agent can’t write to

The fix was to give the verdict a second source of truth that doesn’t flow through the agent at all.

Every call to the pipeline tool now writes an entry to a ledger the runner owns. The agent has no write path to it. The entries hold two kinds of fact, and the distinction matters:

• Asserted by CI. Whether the push went through, whether a pipeline was created, the job name, and the pipeline’s final status. The runner reads these from the CI API. The agent can’t forge them.
• Asserted by the job. The test counts from that job’s JUnit artifact and the changed-line coverage from its coverage artifact. The runner fetches these itself, but a job running agent-written specs produced them. More on that below.

After the agent’s report is parsed, the ledger gets attached to the runner’s output as a separate field. That field isn’t in the agent’s output schema. The agent can’t put it there by accident or on purpose, so a report that claims a pass with no matching ledger entry is just a claim.

Then a guard sits in front of the final verdict. It only cares about one case: the grader said pass. For that case it reads the ledger and keeps the pass only when the last attempt:

• completed (a push failure or a pipeline that never got created is not_started)
• ran in CI (there is also a local mode that seeds the run from files without a pipeline; it can never pass)
• had a job name, with counts read from that job’s own JUnit suite
• finished with status success
• had 0 failed and 0 errored tests
• executed more than 0 tests
• covered at least one executable changed line (zero of zero is reported as unavailable, never as a pass)

If any of that is missing, the pass becomes error with a summary that starts [evidence guard] unsupported: and the reason. Error posts nothing to the MR. The per-scenario rows shown on the MR get rewritten the same way so the UI and the verdict agree.

The guard itself is almost embarrassingly small:

// assessment is null only for sessions this guard doesn't apply to
// (browser runs). A missing or malformed ledger is NOT null; it comes
// back as { supported: false, reason: 'no_attempts' }.
function enforceTestEvidenceGuard(output, assessment) {
  if (output.verdict !== 'pass' || assessment === null) return output
  if (assessment.supported) return output
  return {
    ...output,
    verdict: 'error',
    summary: `[evidence guard] unsupported: ${assessment.reason}: ${assessment.detail}. ${output.summary}`,
  }
}

The rules live in what feeds it. Two of them took me a while to get right.

First, only the last attempt counts. Each push is a new commit on top of the previous one, so the branch tip is whatever the last push put there, and an earlier attempt describes specs that no longer exist on the branch. An earlier good attempt doesn’t rescue a later hollow one. Getting this wrong in the strict direction costs a rerun, since error posts nothing.

Second, the guard only trusts a ledger built in the same process. The ledger is an in-memory value the pipeline tool appends to and hands to the guard directly; the guard never reads it back from the database row where a compact copy is stored. If you rerun just the evaluation step against a stored session, no attempts get recorded, and a previous pass grades error. That felt harsh when I wrote it. The reason it’s right is provenance. Once the ledger is a row on disk, anything could have written it, and the guard has no way to tell.

The judge post said to put a plain if statement in front of the judge for questions like “did the tests run”. This is the half I missed. The if statement is only as good as what it reads, and the agent can’t be the one answering.

What broke once error existed

Once the guard shipped, three things around it turned out to be wrong. Only the first was the guard itself.

It refused to measure a one-line change. A real 2-of-2 pass got rewritten to error with reason nothing_measurable. The changed line was a property inside a 46-line object literal. The v8 coverage provider reports one statement for the whole literal, anchored on its first line, so the changed line itself had zero executable statements on it. Zero of zero would pass any ratio, so the coverage code correctly refused to call it a measurement. The guard was right to refuse and wrong about what was measurable. The fix was to tokenize the changed file with the TypeScript parser and, for a line where no statement starts, take the evidence of the innermost statement, function body, or branch range around it. Each measured file now records which rule counted it, so a fallback to the old statement-start rule is visible in the output.

An error carried into the next run. The rerun logic looked at the previous verdict to decide whether a new commit needed a fresh run. A rerun on a new commit with the same diff inherited the error and skipped. That’s wrong. An error verdict means the run couldn’t produce evidence. It says nothing about the change itself. Fail is a statement about the code. Error is a statement about the run. Only one of those should ever survive a rerun.

Two readers, two verdicts. The dashboard that rolls sessions up for a retro read the base session’s verdict. The gate on the MR read the effective verdict after reruns. They disagreed about the same MRs and I didn’t notice for a while, because each one looked internally consistent. Same mistake as the last time I wrote about gates: two places computing the same fact from different inputs. There’s one function for “effective verdict” now and both read it.

What a pass still doesn’t mean

A guard this strict invites people to over-read a green, so here is what the bar is.

A pass means something the agent wrote executed in CI, hit at least one changed line, and went green. A spec that executes and asserts nothing still clears it. The test counts and the coverage counts both come from the same process the specs run in, so a motivated spec could rewrite its own JUnit report or its own hit counts. The runner checks the diff and the file paths, and it can’t check the counts. The hard floor is the CI-asserted half of the ledger: a pipeline ran, a job ran, it went green. Treat the job-asserted numbers as attested by the job. They aren’t proven until collection moves somewhere the specs can’t reach, and that’s on the list.

What I took from it

The fail-open post was about making a verdict a pure function of live state. The judge post was about an LLM judge being another unreliable model you have to calibrate. This one is the piece underneath both. When the thing being judged is also the thing writing the evidence, calibrating the judge doesn’t help, because the judge is reading fiction either way.

So the rule I’m carrying forward is short. If an agent’s output gates a decision, there has to be a path from the system to the decision that the agent cannot write to. The runner fetched the pipeline status. The runner read the JUnit file. Those facts go in a field the agent’s schema doesn’t have, and the decision reads that field first. Everything the agent says about its own work is context for a human reader. The gate reads the ledger.