Mirroring third-party repositories for resilience

Every submodule, pinned Git dependency and vendored upstream remote is a promise from someone else: that the repository will keep existing, under the same name, with the commits you depend on still reachable. Upstreams break that promise more often than people expect. A maintainer force-pushes away a branch you pinned. A project is renamed, transferred or deleted. A hosting provider has an outage the morning of your release. A mirror — a copy of the upstream repository on infrastructure you control, refreshed on a schedule and never pruned — turns all of those into non-events. This page sets one up correctly, which mostly means making sure the mirror keeps what upstream deletes. It belongs with submodule and dependency integrity.

When to use this approach

  • Your builds fetch code from Git repositories you do not control, via submodules, Git dependencies or subtree remotes.
  • A build has failed because an upstream commit, tag or repository disappeared.
  • Releases must be reproducible years later, as in reproducing a release build from a tag.
  • You want a single place to scan third-party code before it reaches developers.

Step 1 — Create a mirror that never prunes

git clone --mirror copies every ref, and git remote update refreshes them. The default refresh also deletes refs that upstream deleted — exactly the commits you wanted to keep. Disable pruning, and keep a dated snapshot of upstream’s refs so nothing is lost when upstream rewrites a branch.

cd /srv/mirrors
git clone --mirror https://github.com/example/fast-parser.git fast-parser.git
cd fast-parser.git
git config remote.origin.prune false
git config remote.origin.pruneTags false
git config gc.reflogExpire never
git config gc.reflogExpireUnreachable never

Disabling reflog expiry keeps old values of each ref recoverable even after upstream force-pushes. Combined with no pruning, nothing upstream does can make a previously mirrored commit unreachable in your copy.

A default mirror against a preserving mirrorA default mirror refresh follows upstream exactly, including deletions and force-pushes, so commits disappear from the mirror when they disappear upstream. A preserving mirror disables pruning and reflog expiry and archives old ref values, so the mirror only ever grows.Default --mirror refreshPreserving mirrorupstream deletes a branchdeleted here tookeptupstream force-pushesold commits unreachablekept in archive refsupstream deletes reporefresh fails, copy intactcopy intactsize over timetracks upstreamonly growsa mirror that copies deletions is a cache, not a backup

Step 2 — Archive every ref value you have ever seen

Reflogs are per-repository state and easy to lose in a migration. A sturdier approach copies each upstream ref into a dated namespace on every refresh that changes it. Those refs are ordinary Git refs, so they survive clones of the mirror and keep commits reachable indefinitely.

#!/bin/sh
# refresh-mirror.sh <mirror.git>
set -eu
cd "$1"
stamp=$(date +%Y%m%d%H%M)
git for-each-ref --format='%(objectname) %(refname)' refs/heads refs/tags > /tmp/before
git fetch --quiet origin '+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*'
git for-each-ref --format='%(objectname) %(refname)' refs/heads refs/tags > /tmp/after
# Any "sha ref" line that is no longer present means the ref moved or vanished:
# keep its old value under refs/archive/
sort /tmp/before > /tmp/before.s; sort /tmp/after > /tmp/after.s
comm -23 /tmp/before.s /tmp/after.s | while read -r sha ref; do
  git update-ref "refs/archive/$stamp/${ref#refs/}" "$sha"
done
# Verification: simulate an upstream force-push and confirm the old commit is archived
git for-each-ref refs/archive --format='%(refname) %(objectname:short)' | tail

Step 3 — Refresh on a schedule and alert on rewrites

Run the refresh every hour or so. Any archive ref created means upstream rewrote or deleted something you had seen, which is worth a human glance: it might be routine branch cleanup, or it might be a maintainer removing a malicious commit.

# .github/workflows/refresh-mirrors.yml (on the mirror host's runner)
on: { schedule: [{ cron: "23 * * * *" }] }
jobs:
  refresh:
    runs-on: [self-hosted, mirror-host]
    steps:
      - run: |
          for m in /srv/mirrors/*.git; do
            before=$(git -C "$m" for-each-ref refs/archive | wc -l)
            ./refresh-mirror.sh "$m"
            after=$(git -C "$m" for-each-ref refs/archive | wc -l)
            [ "$after" -gt "$before" ] && echo "::warning::$(basename "$m"): upstream rewrote $((after-before)) ref(s)"
          done
An upstream incident, seen from the mirrorThe mirror copies a commit you later pin. Upstream force-pushes that branch and the refresh archives the old value. Upstream then deletes the repository. Builds that fetch from the mirror keep working throughout, and the archive shows exactly what changed.Commit mirroredyou pin itday 1Force-push upstreamold value archivedday 40Rewrite alertsomeone looksday 41Upstream deletedrefresh failsday 90Builds continueserved from mirrorday 90+the alert on day 41 is the early warning; the mirror on day 90 is the insurance

Step 4 — Point builds at the mirror

Change submodule URLs, Git dependency URLs and subtree remotes to the mirror. For submodules, update .gitmodules in a reviewed change and add the mirror URL to the allow-list described in detecting submodule URL tampering in pull requests.

git config -f .gitmodules submodule.vendor/fast-parser.url https://git.example.com/mirrors/fast-parser.git
git submodule sync --recursive
git commit -am "Fetch fast-parser from internal mirror"

For builds you cannot easily edit, a URL rewrite on CI runners redirects without touching manifests:

git config --global url."https://git.example.com/mirrors/".insteadOf "https://github.com/example/"

Rewrites are invisible in the repository, so prefer explicit URLs and use rewrites only as a transition, as discussed in rewriting remote URLs with insteadOf.

Step 5 — Treat the mirror as part of your supply chain

The mirror is now a single place every build trusts. Restrict who can push to it — only the refresh job — and protect it like any build input.

Who may change what in the mirrorOnly the refresh job writes to mirrored refs, and only by fetching from upstream. Archive refs are append-only. Builds and developers have read access only. Nobody pushes to the mirror by hand, so its content always traces to an upstream fetch.write paths into the mirrorRefresh jobfetches upstream, writes refsArchive refsappend-only, never deletedBuilds and developersread onlyManual pushesnot permittedif a person can push to the mirror, the mirror is no longer a faithful copy

Validation checklist

Frequently Asked Questions

Won’t the mirror grow forever?

Yes, slowly. Third-party libraries are usually small, and archived refs usually point at commits that share most of their objects with current history. Monitor disk use; if one mirror grows sharply, investigate why upstream is rewriting so much.

Is a package registry proxy a substitute?

For registry packages, a caching proxy does the same job. Git dependencies, submodules and subtree remotes bypass registries entirely, which is why they need Git mirrors.

Should we scan mirrored code for malware or secrets?

The mirror is a natural choke point for it: scan new commits on each refresh before builds can use them. Even a basic check — new binaries, new install scripts — catches a surprising amount.