Reusing a Git mirror on self-hosted runners

On a repository with a decade of history, a full clone can take longer than the build. Every job pays it, and the transfer is identical each time — the same objects, from the same server, to machines that already had them an hour ago. On hosted runners there is nothing to do about that beyond trimming what you fetch. On machines you own, a local bare mirror turns the fetch into a disk copy. This recipe sets one up safely, extending CI caching and runner performance.

When to use this approach

  • You run self-hosted runners, or runners on long-lived virtual machines.
  • The repository is large enough that fetch time is a visible share of the pipeline.
  • Several jobs on the same host clone the same repository.
  • Jobs need more history than a shallow clone provides.
  • If your runners are ephemeral containers on hosted infrastructure, there is no persistent disk to hold a mirror; use fetch trimming instead.

Step 1 — Create the mirror

A mirror is a bare clone that tracks every ref, which is what makes it usable as a reference for any branch a job might check out.

sudo install -d -o runner -g runner /srv/git
git clone --mirror https://github.com/acme/app.git /srv/git/app.git

# Size it, so you know what you are committing to on each runner host
du -sh /srv/git/app.git
# Verification: the mirror knows every branch and tag
git --git-dir=/srv/git/app.git for-each-ref --count=5 'refs/heads/*'
git --git-dir=/srv/git/app.git rev-parse --is-bare-repository
Where the objects come from with a mirror in placeThe mirror fetches from the remote once per refresh interval. Each job clones from the local mirror over the filesystem, then dissociates so its working copy owns a complete set of objects and does not depend on the mirror surviving.Remoteone fetch per refreshLocal mirror/srv/git/app.gitbare, all refsJob clone--referencefilesystem speedDissociatecopies borrowed objectsindependent working copythe network hop happens once an hour instead of once per job

Step 2 — Clone from it, then dissociate

--reference tells Git to borrow objects from the mirror rather than transferring them. --dissociate then copies the borrowed objects into the new clone, so the result is a normal, self-contained repository.

git clone --reference /srv/git/app.git --dissociate \
  "https://github.com/acme/app.git" "$WORKDIR"
# Verification: no alternates file remains, so nothing depends on the mirror
test -f "$WORKDIR/.git/objects/info/alternates" \
  && echo "STILL BORROWING — dissociate did not run" \
  || echo "self-contained clone"
git -C "$WORKDIR" fsck --connectivity-only --no-progress

SAFETY WARNING — omitting --dissociate leaves the clone borrowing objects from the mirror through an alternates file. If the mirror is then repacked, pruned or deleted while a job is running, the working copy loses objects mid-build and produces corruption errors that look like a hardware fault. Always dissociate for disposable job clones; reserve borrowing for long-lived developer clones where somebody is watching.

Step 3 — Refresh the mirror on a timer, not per job

Refreshing inside the job serialises every job behind a network fetch and reintroduces the cost you were removing.

# /etc/systemd/system/git-mirror.service
[Unit]
Description=Refresh the local Git mirror

[Service]
Type=oneshot
User=runner
ExecStart=/usr/bin/git --git-dir=/srv/git/app.git remote update --prune
# /etc/systemd/system/git-mirror.timer
[Unit]
Description=Refresh the local Git mirror every ten minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=10min

[Install]
WantedBy=timers.target
# Verification: the mirror is current and the timer is firing
systemctl enable --now git-mirror.timer
git --git-dir=/srv/git/app.git log -1 --format='%ci %h' origin/main 2>/dev/null \
  || git --git-dir=/srv/git/app.git log -1 --format='%ci %h' main

Step 4 — Handle the stale-mirror case

A mirror refreshed every ten minutes will sometimes lack the commit a job needs, because the push that triggered the job is newer than the last refresh. The clone must still succeed.

# Clone from the mirror, then fetch only what is missing from the real remote
git clone --reference /srv/git/app.git --dissociate "$REPO_URL" "$WORKDIR"
git -C "$WORKDIR" fetch --no-tags origin "$GIT_SHA" || git -C "$WORKDIR" fetch origin
git -C "$WORKDIR" checkout --detach "$GIT_SHA"
# Verification: the commit under test is present after the fetch
git -C "$WORKDIR" cat-file -e "$GIT_SHA^{commit}" && echo "commit present"

Because the mirror supplies almost everything, the top-up fetch transfers only the handful of new objects — typically a fraction of a second even on a large repository.

Clone time on a repository with 240,000 commitsA full network clone dominates the pipeline. Cloning from a local mirror and dissociating copies the same objects from disk, and the top-up fetch for commits newer than the last refresh adds almost nothing.seconds to produce a working copynetwork clone74 smirror + dissociate9 smirror + top-up fetch11 sthe saving repeats on every job on every runner

Step 5 — Keep the mirror healthy

A mirror that is never maintained grows unboundedly and eventually fetches slower than the network.

# Repack periodically; a mirror benefits from aggressive settings a working copy does not
git --git-dir=/srv/git/app.git gc --aggressive --prune=now

# Or let the scheduler handle it
git --git-dir=/srv/git/app.git maintenance start
git --git-dir=/srv/git/app.git config maintenance.gc.enabled true
# Verification: loose object count should stay low after maintenance
git --git-dir=/srv/git/app.git count-objects -vH | grep -E 'count|size-pack'

The scheduling options are covered in scheduling git maintenance for background repacking, and apply as much to a mirror as to a developer’s clone.

Choosing between a mirror, a cache and a shallow fetchA mirror needs persistent disk on the runner host. Where runners are ephemeral, trimming the fetch is the only lever. Where the repository is small, neither is worth the operational overhead.Do your runners keep a persistent disk between jobs?yes, long-lived hostsLocal mirrorfetch becomes a disk copyno, ephemeralTrim the fetchdepth and filtersrepository is smallLeave it alonethe fetch is not the costthe mirror is an operational commitment: it needs refreshing, repacking and monitoring

Validation checklist

Frequently Asked Questions

Can several runners share one mirror over a network filesystem?

They can, and it frequently performs worse than the network clone it replaced, because Git’s access pattern over a shared filesystem is latency-bound. Keep a mirror per host; the disk is cheaper than the debugging.

What happens if the mirror is corrupted?

Jobs that have already dissociated are unaffected. New clones fail at the reference step, which is loud rather than subtle. Guard it by running git fsck from the refresh timer occasionally and recreating the mirror on failure — recreating is a single command and takes one clone.

Does this work for private repositories?

Yes, provided the mirror is refreshed with credentials the runner user holds and the disk is protected accordingly. Remember that the mirror contains the entire repository in plain form on disk, so the host’s security posture is now part of your source-code protection.