Rolling back a deployment with Git Jump to heading#

There are two rollbacks and they are not interchangeable. Redeploying the previous artefact restores service in seconds and changes nothing about your source history; reverting the change removes it from the line of development so it cannot come back at the next release. During an incident you almost always want the first, and almost always also need the second — and doing only the first is how the same outage happens twice. This recipe covers both and the order to do them in, within environment and deployment branches.

When to use this approach Jump to heading#

• A deployment has caused an incident and service needs restoring now.
• The cause is known well enough to identify the change responsible.
• Your deployment system can redeploy a previous artefact by reference.
• You need the history to reflect what happened afterwards.
• If the fix is smaller than the rollback, fix forward instead — a one-line correction ships faster than a revert and a redeploy.

Step 1 — Record where things are before moving anything Jump to heading#

Thirty seconds now saves an hour of reconstruction later.

stamp=$(date -u +%Y%m%dT%H%M%SZ)
git tag -a "incident/$stamp/pre-rollback" -m 'State at the start of the rollback' origin/production
git push origin "incident/$stamp/pre-rollback"

# And what is actually running, which may differ from the reference
kubectl get deploy app -n production \
  -o jsonpath='{.spec.template.spec.containers[0].image}' | tee /tmp/running-image.txt

# Verification: both records exist before anything changes
git tag -l "incident/$stamp/*"
cat /tmp/running-image.txt

Step 2 — Restore service first, by redeploying the previous artefact Jump to heading#

This is not a Git operation and should not wait for one.

# The previous production deployment, from the deployment tags
prev=$(git tag -l 'deploy/production/*' --sort=-creatordate | sed -n '2p')
git tag -l "$prev" --format='%(contents)' | grep '^digest:'

# Redeploy that digest; source history is untouched
kubectl set image deploy/app app="registry.example.com/app@$digest" -n production
kubectl rollout status deploy/app -n production --timeout=5m

# Verification: service is restored and running the previous artefact
kubectl get deploy app -n production -o jsonpath='{.spec.template.spec.containers[0].image}'

Step 3 — Then revert the change in history Jump to heading#

Service is back; now stop the change returning.

# A squashed pull request is one commit
git switch main
git revert --no-edit <sha>

# A merge commit needs the mainline specified
git revert -m 1 --no-edit <merge-sha>

# Verification: the revert removes exactly what the original added
git show --stat HEAD
git diff <sha>~1 HEAD -- $(git show --name-only --format= <sha>) && echo "content restored"

-m 1 means “keep the first parent’s side” — the branch you merged into. Getting it wrong reverts the wrong half of the merge and produces a commit that looks plausible and is not; the distinction is covered in when to use git revert vs git reset.

SAFETY WARNING — never roll back a shared branch with git reset and a force-push. Reset discards commits that other people already have, so their next push reintroduces the change, and the history diverges in a way that is hard to untangle during an incident. Revert adds a commit that says what happened, which is both safer and more honest.

Step 4 — Promote the revert so the reference matches reality Jump to heading#

git push origin main
git switch production && git merge --ff-only origin/main && git push origin production

# Verification: the invariant holds and production matches the revert
git merge-base --is-ancestor origin/production origin/main && echo "no divergence"
git log --oneline origin/production -2

# Record the rollback deployment like any other
git tag -a "deploy/production/$(date -u +%Y%m%dT%H%M%SZ)" \
  -m "Rollback of <sha>; incident INC-4471" origin/production
git push origin --tags

Step 5 — Reland deliberately Jump to heading#

A reverted change usually still needs to ship. Reverting the revert is the clean way to bring it back with its history intact.

# Bring the original work back as a starting point
git switch -c redo/PAY-931 origin/main
git revert --no-edit <revert-sha>          # revert the revert

# Then fix the actual problem on top, as a separate commit
# ... commit the fix ...
gh pr create --base main --title 'fix(billing): reland refund window with the clamp corrected'

# Verification: the branch contains the original change plus the fix
git log --oneline origin/main..HEAD

Validation checklist Jump to heading#

• The pre-rollback state was tagged before anything moved
• The running artefact was recorded, not assumed from the reference
• Service was restored by redeploying, not by waiting for a build
• The change was reverted on the default branch afterwards
• A merge revert specified the correct mainline parent
• No shared branch was reset or force-pushed
• The rollback is recorded as a deployment
• Relanding restores the work and fixes the cause separately

Frequently Asked Questions Jump to heading#

Should we always revert, or is fixing forward acceptable? Jump to heading#

Fix forward when the fix is small, understood and quick to ship — it is less disruptive and leaves a cleaner history. Revert when the cause is unclear, when the change is large, or when it is the middle of the night. The deciding question is whether you are confident enough in the fix to deploy it to a system that is currently broken.

What if several changes shipped together and we do not know which broke it? Jump to heading#

Roll back the whole deployment to restore service, then bisect the range against a reproduction to identify the culprit — the technique in automating git bisect with a test script. Reverting all of them permanently is rarely necessary once the specific change is known.

Does reverting a merge cause problems when the branch is merged again? Jump to heading#

Yes, and it is a well-known trap: Git considers the merged commits already present, so a second merge brings in nothing. Reverting the revert before merging again, as in Step 5, is the standard answer and keeps the history readable.

Related Jump to heading#

• Environment & Deployment Branches — the parent topic and the deployment record this depends on.
• When to Use git revert vs git reset — the distinction in detail.
• Promoting a Release From Staging to Production — the forward path this undoes.