Hotfix branches that do not drift from main

The classic hotfix failure is not the fix — it is what happens two weeks later, when a routine release overwrites it and the incident recurs. It happens because the fix was applied where the problem was visible, directly onto the production line, and never travelled back to the branch everything else comes from. The prevention is a fixed order of operations that takes about ninety extra seconds during an incident, and this recipe establishes it within environment and deployment branches.

When to use this approach

  • Production is broken and the fix cannot wait for the normal release path.
  • The default branch has moved on and contains changes you cannot ship yet.
  • Your release model has a production reference that trails the default branch.
  • A previous hotfix has already been lost once.
  • If the default branch is safe to deploy right now, do not branch at all — fix forward and promote normally.

Step 1 — Branch from what is deployed, not from the default branch

The fix must be minimal, and minimal means starting from the exact commit that is running.

git fetch origin

# The deployed commit, taken from the deployment record rather than assumed
deployed=$(git rev-parse "$(git tag -l 'deploy/production/*' --sort=-creatordate | head -1)^{commit}")
git switch -c hotfix/PAY-931 "$deployed"
# Verification: the branch point matches what is actually running
img=$(kubectl get deploy app -n production -o jsonpath='{.spec.template.spec.containers[0].image}')
echo "$img" | grep -q "$(git rev-parse --short "$deployed")" && echo "branching from the live commit"
Where a hotfix branch starts and where it landsThe fix branches from the commit currently deployed, so it contains nothing unreleased. It is merged into the default branch first, which is what guarantees the fix survives, and production is then fast-forwarded to a commit that already exists on the default branch.branch from C, merge into main, then promotemainABCDMhotfixCHproductionCMthe fix reaches production only after it exists on the default branch

Step 2 — Keep the fix as small as the incident allows 

# Make the change, then check what it actually touches
git diff --stat "$deployed"
git diff "$deployed" | head -40
# Verification: a hotfix touching many files is a warning sign
git diff --name-only "$deployed" | wc -l

A hotfix that changes eleven files is usually a hotfix carrying unrelated improvements along with it, and each of those is an untested change entering production during an incident. Land the minimum; everything else goes through the normal path afterwards.

Step 3 — Merge into the default branch first

This is the step that gets skipped under pressure and the one that matters.

git switch main
git merge --no-ff hotfix/PAY-931 -m 'fix(billing): clamp refund window to the configured maximum

Incident INC-4471. Fix applied to production out of band; this merge is what
keeps it from being lost at the next release.'
git push origin main
# Verification: the fix is on the default branch before it is promoted
git merge-base --is-ancestor hotfix/PAY-931 origin/main && echo "safe to promote"
# If CI on main is slow, this is the moment to wait — not after production has it
gh run watch "$(gh run list --branch main --limit 1 --json databaseId --jq '.[0].databaseId')"
The order of operations, and what each step protectsMerging into the default branch before promoting is what prevents the fix being lost. Promoting first and merging later leaves a window in which a routine release can overwrite production with a branch that does not contain the fix.hotfix branchmainproductionnext releasemerge firstfast-forward promotecontains the fixpromote firstoverwrites the fixthe two red arrows are the same incident, two weeks apart

Step 4 — Promote the merge commit, not the hotfix branch 

git switch production
git merge --ff-only origin/main
git push origin production
# Verification: the invariant still holds afterwards
git merge-base --is-ancestor origin/production origin/main && echo "no divergence"
git log --oneline origin/production -1

If the default branch contains work that cannot ship, this fast-forward is not available — and that situation is the real argument for keeping the default branch releasable at all times. Where it is unavoidable, cherry-pick the merge onto a release branch and record that you did, using the approach in cherry-picking hotfixes across release branches.

Step 5 — Verify the fix exists everywhere it should 

# The fix commit must be reachable from every long-lived branch
fix=$(git rev-parse hotfix/PAY-931)
for ref in main production staging release/2.8; do
  printf '%-14s ' "$ref"
  git merge-base --is-ancestor "$fix" "origin/$ref" 2>/dev/null \
    && echo "has the fix" || echo "MISSING"
done
# And record the incident link in the history itself
git tag -a "incident/INC-4471" -m 'Hotfix for INC-4471; see the incident review.' "$fix"
git push origin "incident/INC-4471"

SAFETY WARNING — do not delete the hotfix branch until the verification above passes for every branch that needs it. Release branches are the usual omission: the fix lands on the default branch and production, the release branch for the version shipping next week does not get it, and the bug returns on schedule. Check every long-lived ref, not just the two you were looking at during the incident.

Where lost hotfixes actually goReviewing recurrences of previously fixed bugs shows a consistent pattern: most were applied to production without ever reaching the default branch, and the rest reached the default branch but missed a release branch that was already cut.recurrences by cause, over two yearsnever merged to main9missed a release branch4reverted by a bad rollback2genuinely a new bug1thirteen of sixteen were process failures, not engineering ones

Validation checklist

Frequently Asked Questions

What if the default branch cannot be released right now?

Then production is being kept on an older line, and the fix has to be cherry-picked onto that line as well as merged into the default branch. Both are required: the cherry-pick ships it, the merge keeps it. Record the cherry-pick’s source commit so the relationship is traceable later.

Can we skip review during an incident?

Review can be compressed, not skipped — a second person reading a four-line diff takes ninety seconds and catches the class of mistake people make under pressure. Where policy allows an emergency bypass, use the documented break-glass path and complete the review afterwards, rather than removing protection.

Should hotfix branches be long-lived?

No. They exist for the duration of one incident and are deleted after verification. A long-lived hotfix branch becomes a second line of development, which is the divergence this whole procedure exists to prevent.