# The Fail-Open I Shipped Three Times
> A completeness check that had to be safe over time, three state machines that each leaked a fail-open, and the pure function that finally fixed it.
Published: 2026-08-10
Tags: Agents
Canonical: https://alexmlewis.com/writing/the-fail-open-i-shipped-three-times
---
**tl;dr:** I was building a gate that auto-approves a merge request once every required reviewer has reported and passed. The hard part was one tiny sub-question: how do you know every reviewer that _will_ report _has_ reported? I answered it three separate times. Every answer was a state machine, and every state machine leaked a fail-open. The fix was to stop remembering anything and make the verdict a pure function of what the system already tells you.

## The setup

The gate is easy to describe. An MR picks up a handful of automated reviewers, each one a CI job that posts a review note, plus a separate quality check. When all of them pass, the gate marks the MR approvable. When any of them fails or hasn't shown up yet, it stays pending.

One rule mattered more than everything else: the gate can only ever downgrade. If it's unsure, it lands on pending. Approving an MR whose reviewers haven't all weighed in is the one outcome you can't ship, because "approved" is the state that lets code merge without a human looking.

So the whole thing hangs on a single question. Right now, at this instant, has every reviewer that's going to report already reported? Two of three bots posted PASS and the third hasn't run? That's incomplete, and incomplete has to read as pending.

Sounds trivial. It was not.

## Attempt one: a settling window

First idea, the obvious one: wait a beat. Once review notes started landing, I'd hold for a short settling window before trusting the set. The bots post within a minute or two of each other, so if things went quiet for long enough, the set was probably complete.

"Probably complete" is carrying the entire sentence there. A slow-starting reviewer that hadn't posted yet looked exactly like a reviewer that was never going to post. The window expired, the gate counted the passes it had in hand, and it greened an MR that was still missing a voice. Fail-open number one.

## Attempt two: settle, plus a shrink guard

Okay, naive. I added memory: remember the biggest reviewer set I'd ever seen for this MR, and refuse to green if the current set was smaller than that. If reviewer C ever showed up, the gate should never again pretend C doesn't exist.

That caught the shrinking case and did nothing for the case that actually bit me: C simply hadn't shown up yet the first time either. On the very first evaluation, before C had ever posted, the remembered set already looked complete, so the gate greened. The guard protected me from forgetting a reviewer. It had no opinion about a reviewer I'd never met.

Fail-open number two.

## Attempt three: per-head roster memory

Third swing, more memory, because apparently I'm a slow learner. For each commit head I persisted the roster of reviewers I expected, and only greened once every name on that roster had reported for that head.

The tell that I'd wandered off a cliff: the test I wrote for this asserted the fail-open as _correct behavior_. There was a path where the persisted roster came back stale or empty, and instead of treating that as "I don't know, stay pending," my own test said "roster looks satisfied, green it." I had built the bug into the design and then written a passing test to bless it. That's the software equivalent of signing your own permission slip.

```ts
// Completeness as a state machine.
// The decision at T depends on what an earlier evaluation wrote.
function isComplete(mr: MR): boolean {
  const roster = store.get(mr.head) // read persisted memory
  const reported = mr.reviewNotes.map((n) => n.reviewer)

  if (!roster) {
    store.set(mr.head, reported) // first sighting becomes the roster
    return covers(reported, reported) // ...trivially "complete" -> fail-open
  }
  return covers(reported, roster)
}
```

## The thing I kept doing

Three attempts, one mistake sitting underneath all of them.

Every version modeled completeness as a state machine. Each evaluation read some persisted memory (a window timestamp, a high-water set, a roster), made a call, and wrote memory back. The decision at time T leaned on what an _earlier_ observation had recorded. And every single time, some sequence of observations could walk the machine into a state where "green" was reachable while a reviewer was still missing.

That's the trap with a downgrade-only guarantee. Downgrade-only has to hold across the whole sequence of evaluations, not just inside one of them. The moment one evaluation's stored output feeds the next one's decision, you owe a proof that the invariant survives every path through that memory. I couldn't write that proof, because it wasn't true.

![Top lane: a state machine where each evaluation is fed by the previous evaluation's stored memory, and a reviewer that reports late slips through as a wrong green. Bottom lane: a pure function where each evaluation reads only live state, carries nothing forward, and stays correct.](../../assets/writing/the-fail-open-memory-vs-live.webp)

## The fix: delete the state

Here's the part that stung. The reviewer set was never actually unknown. It was sitting in plain sight the whole time.

The reviewers are CI jobs. The list of CI jobs on the commit under test _is_ the roster. I never needed to remember which reviewers I'd seen across time, because at any instant the pipeline already tells me exactly which reviewers exist and which have reported. I'd been persisting a shaky shadow copy of a fact the system was handing me for free.

So the completeness check became a pure function of the state you can observe right now:

```ts
// The reviewer set was never unknown. The CI jobs on the head ARE the roster.
// Nothing persisted is read back to decide a verdict.
function isComplete(mr: MR): boolean {
  const expected = new Set(
    mr.pipeline.jobs.filter((j) => j.name.startsWith('review')).map((j) => j.reviewer),
  )
  const reported = new Set(mr.reviewNotes.map((n) => n.reviewer))

  return [...expected].every((reviewer) => reported.has(reviewer))
}
```

Which is really just this: `verdict(head) = f(review_notes, head, pipelines, jobs)`.

No stored window. No high-water set. No roster memory. Nothing written during one evaluation is ever read back to decide a later one. If the pipeline shows a review job with no note yet, the set is incomplete, full stop, pending. If every review job has reported, it's complete, and the gate one layer up decides pass or fail. That's the whole logic.

Once the verdict depends only on the current observable state, downgrade-only stops being something I have to defend across a sequence. It just falls out. Two evaluations of the same head see the same inputs and give the same answer. A new commit resets the world, because the job list resets with it. There's no accumulated memory left to rot.

## What I took from it

When a rule has to hold across a series of checks instead of within one, reach for persisted state as a last resort. Every scrap of memory I added was me trying to reconstruct, from history, a fact the live system was already broadcasting. All three heuristics were fine on their own. The mistake was deciding to remember at all.

If a check has to stay safe over time, the cleanest version is usually the one where each evaluation can't see the last one. Make it a pure function of what's observable now, and the property you're chasing often stops being a property you have to guard.

Took me three tries to believe that. Writing it down so the fourth version of me doesn't add a fourth cache.