Rotating CI signing identities without breaking verification

CI signing keys should rotate often — quarterly is a reasonable default — because unlike a person’s key, nobody notices if one leaks. In practice they rotate rarely, because the last rotation broke something: a release job that was mid-run when the secret changed, a deploy gate whose pinned trust file still had only the old key, an audit that went red for every commit signed before the switch. Each of those is a sequencing problem, and each is avoided by the same pattern: add the new key everywhere before using it, switch, and only then close the old key’s window. This page turns that pattern into a repeatable, mostly automated procedure, within signed commits in CI pipelines.

When to use this approach

Step 1 — Generate the new key and add it to trust first

The new key must be trusted by every verifier before anything signs with it. Add it with a valid-after date of today and leave the old key’s line untouched.

new=release-bot-$(date +%Y%m)
ssh-keygen -t ed25519 -N '' -C "signing:[email protected]:$(date +%Y-%m)" -f "$new"
printf '[email protected] namespaces="git",valid-after="%s" %s\n' \
  "$(date +%Y%m%d)" "$(cat "$new.pub")" >> trust/allowed_signers
git commit -am "Trust new release-bot signing key $(date +%Y-%m)"
A rotation with an overlap windowThe new key is added to the trust file first and distributed to every verifier. After a few days the CI secret is switched to the new key. A week later the old key's window is closed with a valid-before date, and its secret is deleted.Trust new keyvalid-after todayday 0Distributeevery verifier updatedday 0–3Switch secretjobs sign with new keyday 3Close old keyvalid-before setday 10Delete old secretnothing can use itday 10the order — trust, distribute, switch, close — is what makes it uneventful

Step 2 — Wait until every verifier has the new trust file

Distribution is the step rotations skip. Server hooks may copy the trust file on a schedule, deploy tooling may pin a revision, and users may have an old copy. Check each consumer before switching.

# Each verifier should report the new key's fingerprint
new_fp=$(ssh-keygen -lf "$new.pub" | awk '{print $2}')
for host in git-primary git-replica; do
  ssh "$host" "grep -c '$(cut -d' ' -f2 "$new.pub")' /etc/git/allowed_signers" | sed "s/^/$host: /"
done
grep -rn "SIGNERS_REV" .github/workflows/      # pinned revisions that need bumping

If a verifier pins the trust file by revision, bump the pin now and let that change merge before the switch.

Step 3 — Switch the secret, not the workflow

Replace the secret’s value in place. The workflow keeps referring to the same secret name, so no pipeline change is needed and in-flight runs are unaffected — a job that already loaded the old key finishes with it, which is fine because the old key is still trusted.

gh secret set RELEASE_BOT_KEY --env release-signing < "$new"
shred -u "$new"
gh secret list --env release-signing --json name,updatedAt
# Verification: the next signed object uses the new key's fingerprint
git fetch --tags && git log -1 --format='%GF' "$(git describe --tags --abbrev=0)"
Switching the secret versus switching the workflowReplacing the secret's value keeps the workflow unchanged and lets in-flight jobs finish with the old key. Pointing the workflow at a new secret name requires a code change, can race with running jobs, and leaves the old secret lying around.Replace secret valueNew secret nameworkflow changenoneedit + reviewin-flight jobsfinish on old keymay fail mid-runold secretoverwrittenlingers until deletedkeep the name stable and rotate the value

Step 4 — Close the old key’s window after the overlap

Once the new key has signed successfully and every verifier has had a week to catch up, add a valid-before date to the old key’s line. Do not delete the line: historical commits and tags signed with it must keep verifying.

# Close the old key as of today (edit the line for the previous key)
sed -i "/[email protected]/ { /$(date -d '-3 months' +%Y%m)/ s/namespaces=\"git\"/namespaces=\"git\",valid-before=\"$(date +%Y%m%d)\"/ }" trust/allowed_signers
git diff trust/allowed_signers
git commit -am "Close previous release-bot signing key"

The sed expression above depends on how you label lines; reviewing the diff before committing is the real safeguard.

# Verification: an old tag still verifies, and a test signature with the old key is rejected
git verify-tag v2.3.0 && echo "history ok"

Step 5 — Automate the schedule

A rotation that needs a person to remember it will not happen quarterly. Script steps one and four as two scheduled workflows that open pull requests, with step three run by an approved environment job in between.

A quarterly rotation as two pull requestsA scheduled workflow generates a key, stores it in a staging secret and opens a pull request adding it to the trust file. Merging that triggers the secret swap after a delay. A second scheduled workflow, ten days later, opens a pull request closing the old key's window.Scheduled jobgenerate keyPR 1trust new keySwapafter 3 daysPR 2close old windowDeleteold secret valuehumans review two small diffs a quarter; nothing else needs remembering

Validation checklist

Frequently Asked Questions

How long should the overlap be?

Long enough for the slowest verifier to pick up the new trust file — usually a few days — plus the longest job that might start with the old key. A week covers almost every setup. During a compromise there is no overlap: close the old key at once and accept a few failed runs.

Can I rotate keyless signing identities?

There is nothing to rotate: each signature uses a certificate that lives for minutes. What you manage instead is the identity policy verifiers check, such as the workflow path and repository, which changes only when the workflow moves.

What if a verifier was missed and starts rejecting new tags?

Add the new key to that verifier’s trust file; nothing else needs to change, because the old key is still valid during the overlap. That is the reason to close the old window last.