Bisecting a flaky failure

Bisection assumes each commit gives a truthful answer. A flaky reproduction breaks that assumption in one direction only: a commit where the bug is present can still pass, so it gets marked good, and the search eliminates the half of the range containing the actual culprit. The result is a confident answer pointing at an innocent commit, and no indication anything went wrong. The fix is statistical rather than clever — repeat the reproduction enough times that a false good becomes unlikely. This recipe does that, within bisect and history forensics.

When to use this approach

  • The failure reproduces sometimes rather than always.
  • A previous bisect produced a commit that clearly cannot be responsible.
  • The bug involves timing, concurrency, ordering or external state.
  • You can run the reproduction many times cheaply.
  • If the reproduction fails less than about one run in twenty, make it more reliable first — the repetition cost becomes prohibitive.

Step 1 — Measure the failure probability

You cannot choose a repeat count without a number.

# Run the reproduction thirty times at the known-bad commit
fail=0
for i in $(seq 1 30); do node /tmp/repro.mjs || fail=$((fail + 1)); done
echo "$fail / 30 runs failed"
# And confirm it never fails at the known-good commit
git switch --detach v2.6.0 && npm ci --silent >/dev/null 2>&1
ok=0; for i in $(seq 1 30); do node /tmp/repro.mjs && ok=$((ok + 1)); done
echo "$ok / 30 runs passed at the good endpoint"
# Verification: a good endpoint that ever fails is not a good endpoint
How many repeats a given flake rate needsWith a failure probability of one run in two, three repeats reduce the chance of a false good to one in eight. At one run in five, ten repeats are needed for the same confidence, and at one in twenty the cost becomes prohibitive.repeats needed for 95% confidencefails 1 in 25fails 1 in 411fails 1 in 1029fails 1 in 2059each repeat multiplies the total bisect time — the red bar usually means fixing the test first

The arithmetic is straightforward: with failure probability p, the chance that n runs all pass at a bad commit is (1−p)^n. Pick n so that number is small — under 5% is a reasonable bar.

Step 2 — Build repetition into the script 

#!/usr/bin/env sh
# bisect-flaky.sh — any failure across N runs means the bug is present.
set -eu
RUNS=${RUNS:-11}

npm ci --silent --ignore-scripts >/dev/null 2>&1 || exit 125
npm run --silent build >/dev/null 2>&1 || exit 125

i=0
while [ "$i" -lt "$RUNS" ]; do
  if ! node /tmp/repro.mjs >/dev/null 2>&1; then
    echo "failed on run $((i + 1)) of $RUNS" >&2
    exit 1                       # bad: one failure is enough
  fi
  i=$((i + 1))
done
exit 0                           # good: survived every run
# Verification: the script is bad at the known-bad commit, reliably
for i in 1 2 3; do sh bisect-flaky.sh; echo "attempt $i → $?"; done

The asymmetry is deliberate. A single failure proves the bug is present; no failures across many runs is evidence — not proof — that it is absent, which is exactly the confidence structure the repeat count is chosen to bound.

Step 3 — Cut the cost per repeat

Repetition multiplies everything, so this is where optimisation pays.

# Run the reproduction directly rather than through a test runner
time npm test -- --testNamePattern 'refund' >/dev/null 2>&1
time node /tmp/repro.mjs
# Reuse the build across repeats — build once, run many
npm run --silent build >/dev/null 2>&1
i=0; while [ "$i" -lt 11 ]; do node /tmp/repro.mjs || break; i=$((i + 1)); done
# Verification: total script time should be dominated by the repeats, not the build
time sh bisect-flaky.sh
Where the time goes in a flaky bisectThe install and build happen once per commit; the reproduction happens once per repeat. Moving anything expensive out of the repeat loop is what makes a high repeat count affordable.Installonce per commitBuildonce per commitRepeat loopN times per commitkeep this tinyVerdictany failure → badan expensive step inside the loop multiplies by N and by the number of bisect steps

Step 4 — Run the search, then verify the answer twice

A flaky bisect earns more scepticism than a deterministic one.

git worktree add --detach ../app-bisect main && cd ../app-bisect
git bisect start main v2.6.0
RUNS=11 git bisect run sh ../app/bisect-flaky.sh
# Verify at the result and its parent, with a higher repeat count
result=$(git rev-parse HEAD)
git switch --detach "$result"    && RUNS=30 sh ../app/bisect-flaky.sh || echo "bad, confirmed"
git switch --detach "$result^"   && RUNS=30 sh ../app/bisect-flaky.sh && echo "parent good, confirmed"
# And read the commit — does it plausibly cause this?
git show --stat "$result"

SAFETY WARNING — a bisect over a flaky test can produce a plausible-looking commit that is entirely innocent, and acting on it means reverting working code and continuing to carry the real bug. Never treat a flaky bisect result as final without the higher-repeat confirmation at the commit and its parent, and without reading the commit to check that it could cause the symptom.

Step 5 — Fix the flakiness, not just the bug

The investigation has just demonstrated that the test is unreliable. That is worth acting on while the evidence is fresh.

# Quarantine it out of the blocking path, with an owner and a date
mkdir -p test/quarantine && git mv test/refund.test.ts test/quarantine/
git commit -m 'test: quarantine the refund window test

Flake rate measured at ~25%. Owner: payments. Review by 2026-10-02.'
# Verification: the gate no longer depends on it
npx jest --testPathIgnorePatterns 'quarantine/' --listTests | grep -c refund
A naive bisect against a repeated oneWith a single run per commit, a flaky test marks bad commits as good roughly a quarter of the time, and the search eliminates the wrong half. Repeating eleven times reduces that to a few per cent, at the cost of eleven times the runtime.One run per commitEleven runs per commitfalse good probability25%under 5%result trustworthynowith confirmationtime per step1 unit11 unitsanswer usablemisleadingactionableeleven times the runtime for an answer that is true is a good trade

The quarantine mechanics, including keeping the list from becoming permanent, are in handling flaky tests in a merge queue.

Validation checklist

Frequently Asked Questions

What if the flake rate differs across the range?

That is a strong signal the flakiness itself was introduced by a commit in the range, which makes it the more interesting thing to bisect. Search for the change in flake rate rather than for the failure: have the script run a fixed number of repeats and exit bad when the failure count exceeds a threshold.

Can I reduce repeats by running them in parallel?

Yes, where the reproduction is independent of shared state — which for a flaky test is exactly what is in question. Parallel runs that contend for the same port, file or database change the conditions and can mask or amplify the flakiness. Measure both ways before trusting parallel results.

Is git bisect skip useful here?

Not for flakiness. Skip is for commits that cannot be judged at all; a flaky commit can be judged, just unreliably. Skipping on a failure would discard exactly the evidence the search needs.