Handling flaky tests in a merge queue

On a pull request, a flaky test costs its author one re-run. In a merge queue it costs a whole batch: the candidate tree goes red, an innocent entry is ejected, the batch is rebuilt, and everyone behind it waits. The same flake rate that was an annoyance becomes a throughput ceiling, and the usual reaction — retry everything automatically — quietly converts the gate into decoration. This recipe keeps the gate honest, building on merge queues and required checks.

When to use this approach

  • Entries are being ejected for failures that pass on the next run.
  • Your queue throughput is lower than your pipeline duration suggests it should be.
  • You already know which tests are unreliable but have nowhere to put them.
  • Someone has proposed “just retry the whole job three times” and you would like a better answer.
  • If your suite is genuinely deterministic, skip this and spend the effort on batching pull requests to cut CI cost.

Step 1 — Measure the flake rate before changing anything

A flake rate is the probability that a run fails for reasons unrelated to the change. You can estimate it cheaply by running the suite repeatedly against a commit that is known good — no code changes, so every failure is noise.

# Re-run the suite ten times against an unchanged tree
for i in $(seq 1 10); do
  npm test >/dev/null 2>&1 && echo "run $i pass" || echo "run $i FAIL"
done
# The same measurement in CI, where the environment differs from a laptop
gh workflow run ci.yml --ref main
gh run list --workflow ci.yml --branch main --limit 20 \
  --json conclusion --jq 'group_by(.conclusion)[] | {result: .[0].conclusion, count: length}'

What this gives you: a number to reason with. A 2% per-run flake rate sounds tolerable until you notice a batch of five inherits roughly a 10% chance of going red for nothing.

How a small flake rate scales with batch sizeA two per cent chance that any one entry fails spuriously compounds across a batch, because a single spurious failure reddens the whole candidate tree. At a batch of ten, one run in five is wasted on a failure nobody caused.chance a batch fails for no reason (2% per-entry flake rate)batch of 12%batch of 36%batch of 510%batch of 1018%flakiness and batching multiply — fix the first before raising the second

Step 2 — Quarantine rather than delete

A test that fails at random is still testing something. Move it out of the blocking path into a quarantined suite that runs on the same commit, reports separately, and is visible enough that the quarantine list does not become a graveyard.

# Tag the offender so the runner can exclude it from the gate
# (Jest: a custom test path pattern; pytest: a marker; go: a build tag)
npx jest --testPathIgnorePatterns 'quarantine/' --ci
npx jest --testPathPattern 'quarantine/' --ci || true    # reported, not enforced
# .github/workflows/ci.yml — the quarantined run informs without gating.
  quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx jest --testPathPattern 'quarantine/' --ci
# Verification: the gate no longer includes quarantined tests
npx jest --testPathIgnorePatterns 'quarantine/' --listTests | wc -l

SAFETY WARNING — quarantine is a loan, not a write-off. A quarantined test covers behaviour that is now unguarded, so every entry needs an owner and a date. A quarantine list with no expiry is indistinguishable from deleting the tests, except that it still costs machine time.

Step 3 — Retry at the test level, never the job level

Retrying a whole job hides which test was unreliable and re-runs everything to mask one failure. Retrying a single test preserves the signal: the run is green, and the retry is recorded so the flake rate stays measurable.

Job-level retry against test-level retryRe-running an entire job masks which test failed and multiplies machine time. Retrying only the failing test keeps the run fast, records the retry so flakiness stays visible, and leaves a genuine regression failing.Retry the whole jobRetry the single testsignal keptlost — all greenrecorded per testmachine timefull suite againone testreal regressionretried until greenstill failsconfig effortone linea runner flaga job-level retry is how a gate stops gating without anyone deciding to remove it 
# Most runners support a bounded per-test retry
npx jest --retryTimes 2 --ci                 # Jest
pytest --reruns 2 --reruns-delay 1           # pytest-rerunfailures
go test -count=1 ./...                       # Go: no retry — fix or skip explicitly

Cap retries at two. Three or more means the test is not flaky, it is broken, and the number is being chosen to make it pass.

Step 4 — Make the queue eject the right entry

When a batch is red, the queue must attribute the failure. Most implementations bisect by rebuilding smaller batches; you can make that far cheaper by ensuring test output names the failing test in a machine-readable form.

      - run: npx jest --ci --reporters=default --reporters=jest-junit
        env:
          JEST_JUNIT_OUTPUT_FILE: reports/junit.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: junit, path: reports/junit.xml }
# Verification: the failing test name is extractable without reading logs
xmllint --xpath '//testcase[failure]/@name' reports/junit.xml 2>/dev/null

What changed: an ejection now comes with the name of the test that caused it, which is the difference between an author fixing something and an author pressing re-enqueue.

From a red batch to a correct ejectionA red candidate tree produces a structured test report. The queue extracts the failing test, maps it to the entry that touched the relevant code, ejects only that entry, and rebuilds the batch from the remainder.Red batchcandidate tree failsStructured reportjunit.xmlfailing test namedAttributionwhich entry touched itEject onebatch rebuiltothers continuewithout the middle two steps the queue guesses, and guessing ejects innocent entries

Step 5 — Report the flake rate where the team sees it

Flakiness is a budget. Publish it next to the pipeline duration, because the two together decide the maximum sensible batch size and, eventually, whether the queue is worth running at all.

# Weekly: how many merge-group runs failed and were re-run green unchanged
gh run list --event merge_group --limit 200 --json conclusion,attempt \
  --jq '{runs: length,
         failed: ([.[] | select(.conclusion=="failure")] | length),
         retried: ([.[] | select(.attempt > 1)] | length)}'

Validation checklist

Frequently Asked Questions

Why not retry the job automatically and move on?

Because the gate then reports “eventually green”, which is true of a broken build as well as a flaky one. Every genuine regression that fails intermittently — a race condition, a timezone bug, a resource leak under load — is exactly the class of defect that a blanket retry converts into a silent merge.

How long should a test stay in quarantine?

Two weeks is a workable default: long enough to schedule the fix, short enough that the list cannot grow unnoticed. When the date passes, the choice is deliberate — fix it, or delete it and accept that the behaviour is untested. Letting it sit is the option that quietly loses coverage.

Does a merge queue make flakiness worse?

It makes the cost visible rather than larger. The same tests failed at the same rate before; the failures were absorbed by individual authors instead of by the shared queue. That visibility is useful, because it converts a chronic annoyance into a number someone can be asked to reduce.