Auditing a repository for large blobs

A repository that takes six minutes to clone is carrying something, and it is rarely the source code. Build artefacts committed once and deleted, a dependency vendored before a package manager existed, a dataset added for a demo four years ago — all still transfer on every clone, because history includes everything that was ever committed. Finding them takes one command; deciding what to do about them is the part that needs care. This recipe covers both, within bisect and history forensics.

When to use this approach

  • Cloning takes minutes and the working tree is small.
  • The .git directory is much larger than the checkout.
  • A binary was committed by accident and you want to know what it cost.
  • Before deciding whether to migrate to LFS or rewrite history.
  • If a full clone takes seconds, there is nothing here worth acting on.

Step 1 — Measure the gap between history and checkout 

du -sh .git
git count-objects -vH | grep -E 'size-pack|count'
du -sh --exclude=.git .
# The ratio is the signal: history much larger than the tree means deleted weight
python3 - <<'PY'
import subprocess
pack = subprocess.run(['git','count-objects','-v'],capture_output=True,text=True).stdout
size = int([l for l in pack.splitlines() if l.startswith('size-pack')][0].split()[1])
print(f'packed history: {size/1024:.0f} MB')
PY
# Verification: how long does a fresh clone actually take?
time git clone --quiet . /tmp/clone-timing && rm -rf /tmp/clone-timing

Step 2 — Rank every object ever committed

This is the command worth remembering.

git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
  | awk '$1 == "blob" {print $3, $4}' \
  | sort -rn | head -20 | numfmt --to=iec --field=1
# Grouped by path, which is more actionable than individual blobs
git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectsize) %(rest)' \
  | awk '$1 == "blob" && $3 != "" {s[$3] += $2} END {for (p in s) print s[p], p}' \
  | sort -rn | head -20 | numfmt --to=iec --field=1
# Verification: the top paths should be recognisable
What a first audit usually findsThe largest contributors are almost never source files. A vendored dependency directory, a set of design assets and a build output directory that was committed for a few months typically account for the majority of a repository's packed size.megabytes of history, by pathvendor/ (removed 2023)610 MBassets/design/340 MBdist/ (committed briefly)290 MBall source files41 MBtwo of the top three no longer exist in the working tree and still transfer on every clone

Step 3 — Attribute each blob to a commit and a date

Knowing which commit introduced something tells you whether it is still needed and who to ask.

# Find the commits that contain a given path
git log --all --oneline --name-only -- 'vendor/**' | head -20
# When was it added, and when removed?
git log --all --diff-filter=A --format='%h %ad %an added' --date=short -- vendor/ | tail -1
git log --all --diff-filter=D --format='%h %ad %an removed' --date=short -- vendor/ | tail -1
# Verification: is it still present anywhere?
git ls-files vendor/ | head -3 || echo "not in the working tree"

Step 4 — Choose between the three options

Each has a different cost, and the right answer differs per path.

What to do with a large pathA file still in use and still large belongs in LFS, which keeps it usable while removing it from the ordinary clone. A file no longer present is dead weight that only a history rewrite removes. Anything below a few tens of megabytes is usually not worth the disruption.Is the path still in the working tree?yes, and still largeMigrate to LFSkeeps it usableno, deleted long agoHistory rewritethe only way to reclaimunder ~50 MB totalAccept itnot worth the disruptionthe middle branch means everyone reclones and every signature breaks 
# Option A: migrate an active path to LFS
git lfs track 'assets/design/**'
git add .gitattributes && git commit -m 'chore: track design assets in LFS'
# Option B: remove a dead path from history (destructive, requires a reclone)
git clone --mirror . ../backup-before-rewrite.git      # ALWAYS first
git filter-repo --path vendor/ --invert-paths
# Verification after either: the audit should show the change
git count-objects -vH | grep size-pack

SAFETY WARNING — removing a path from history rewrites every commit that contained it, which changes every subsequent commit id. Signatures break, external references stop resolving, and everyone must reclone rather than pull. Take a mirror backup first, publish the commit-id mapping, and weigh the disruption honestly: a 300 MB saving on a repository nobody complains about is not worth a week of broken links.

Step 5 — Prevent the next one

An audit is a cleanup; a check is a policy.

# A pre-receive hook is the only enforcement that cannot be skipped
#!/usr/bin/env sh
limit=$((5 * 1024 * 1024))
while read -r old new ref; do
  git rev-list --objects "$old..$new" \
    | git cat-file --batch-check='%(objecttype) %(objectsize) %(rest)' \
    | awk -v lim="$limit" '$1=="blob" && $2 > lim {print "too large: " $3 " (" $2 " bytes)"; bad=1}
                           END {exit bad+0}' || exit 1
done
# And a local warning, so the rejection is not a surprise at push time
git config --local core.bigFileThreshold 5m
# Verification: a large file is refused
dd if=/dev/urandom of=/tmp/big.bin bs=1M count=10 2>/dev/null
git add -f /tmp/big.bin 2>/dev/null; git status --short | head -1
The audit cycleMeasure the gap between history and checkout, rank the paths, attribute each to when it entered, choose a remedy per path, and install a limit so the next one is rejected at push time rather than discovered in an audit three years later.Measurehistory vs checkoutRank by pathcumulative sizeAttributeadded, removed, by whomRemedyLFS, rewrite, acceptPreventserver-side limitthe last box is the only one that stops the audit being needed again

The server-side enforcement is covered in enforcing file size limits on the remote, and the LFS migration path in migrating large binaries to Git LFS.

Validation checklist

Frequently Asked Questions

Does garbage collection reclaim the space?

Not for anything still reachable from a branch or tag, which is almost everything in history. git gc repacks and removes genuinely unreachable objects, so it helps after a rewrite and does nothing before one. If a path is still in any commit anywhere, its blobs are reachable and stay.

Can partial clone avoid the problem without a rewrite?

Largely, yes, and it is the least disruptive option: --filter=blob:none defers blob transfer until something reads a file, so a clone stops paying for history it never touches. It does not shrink the server-side repository and it makes some operations slower, but it requires no rewrite and no reclone — the details are in speeding up clones with partial clone.

What size is worth acting on?

There is no universal number, but a useful rule is that a rewrite needs to remove a large fraction of the repository to justify its disruption. Reclaiming 15% is rarely worth it; reclaiming 70% usually is. Below that threshold, partial clone and a size limit achieve most of the benefit with none of the fallout.