Large Repository Performance Jump to heading#

A slow repository is rarely slow for the reason people assume. “It is too big” is a conclusion, not a diagnosis, and the three underlying causes — deep history, a wide working tree, and neglected object maintenance — have entirely different fixes. Applying the wrong one produces a lot of disruption and no improvement. This part of Git Workflow Architecture & Branching Strategies is about measuring first, then choosing the fix that matches what the measurement says.

The good news is that the effective fixes are all client-side and reversible. Partial clone, sparse checkout and background maintenance change how one machine talks to the remote; nobody else needs to agree, nothing about the repository’s contents changes, and a developer who wants a full clone keeps one. That makes the whole area unusually safe to experiment with — in contrast to history rewriting, which is the one intervention that genuinely shrinks a repository and the one that costs everybody a re-clone.

Prerequisites Jump to heading#

• Git v2.30 or newer; several features here are much better from v2.37 onward (git --version)
• A remote that supports partial clone filters — all major hosts do, self-managed instances may need uploadpack.allowFilter enabled
• Baseline numbers for clone time, git status time, and repository size, captured before any change
• For the monorepo case, agreement on which directories each team actually needs — see Monorepo Branch Topology
• A policy stopping new large files from arriving, or the problem returns — see Enforcing File Size Limits on the Remote

Step 1 — Measure Before Changing Anything Jump to heading#

# Object counts and on-disk size — the headline numbers
git count-objects -vH

# How many refs? Tens of thousands make every operation slow regardless of size.
git for-each-ref | wc -l

# The ten largest blobs anywhere in history, with their paths
git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectsize) %(rest)' \
  | awk '$1=="blob" {print $2, $3}' \
  | sort -rn | head -10

# How long does the most common operation take?
time git status

Write those four numbers down. Every change below should be re-measured against them, because the most common outcome of an unmeasured optimisation is a lot of disruption and no improvement.

Step 2 — Separate the History Problem From the Working-Tree Problem Jump to heading#

The third row is the one teams overlook. A repository of two hundred megabytes with a hundred thousand loose objects and no commit-graph feels far worse than a two-gigabyte repository that is well packed, because the cost lives in object lookup rather than bytes. Check git count-objects -vH for a large loose-object count before concluding that you have a size problem at all.

Step 3 — Shrink What Is Fetched With Partial Clone Jump to heading#

A partial clone downloads all commits and trees but no file contents, then fetches blobs on demand when a command actually needs them.

# Blobless clone: full history, no file contents until needed
git clone --filter=blob:none https://example.com/org/repo.git

# An existing clone can be converted in place
git config remote.origin.promisor true
git config remote.origin.partialclonefilter blob:none

Verify the filter took effect:

git config remote.origin.partialclonefilter    # expect: blob:none
git rev-list --objects --all --missing=print | grep -c '^?' # count of absent blobs

The trade is latency, not correctness. git log and git branch become dramatically faster because they never touch file contents; git checkout of an old commit pauses briefly to fetch the blobs it needs. For everyday work on recent commits, the fetches are rare and small. For a git log -p across the whole history, they are neither — which is the one workflow where a full clone still wins.

SAFETY WARNING — a partial clone depends on the remote staying reachable, because missing blobs are fetched lazily. It is therefore a poor choice for an archival copy, a disaster-recovery mirror, or a machine that works offline for long stretches. Keep at least one full clone of anything you rely on for recovery, and verify it with git fsck --full rather than assuming it is complete.

Step 4 — Shrink What Is Checked Out With Sparse Checkout Jump to heading#

Partial clone reduces what crosses the network; sparse checkout reduces what lands on disk. In a monorepo the second is usually the bigger win, because git status cost is driven by the number of files in the working tree.

# Cone mode: directory-level patterns, and the fast path
git sparse-checkout init --cone

# List the directories this developer actually works in
git sparse-checkout set services/payments libs/shared

# Inspect and adjust
git sparse-checkout list
git sparse-checkout add libs/telemetry

# Confirm the working tree shrank and status got faster
git ls-files | wc -l           # tracked paths in the index
find . -path ./.git -prune -o -type f -print | wc -l   # files actually on disk
time git status

Cone mode matters for performance, not just ergonomics: it restricts patterns to whole directories, which lets Git use a sparse index that does not enumerate excluded paths at all. Non-cone patterns are more expressive and much slower, and mixing the two is the usual reason a sparse checkout fails to deliver the expected speedup. The full workflow, including how to handle a build that expects paths outside the cone, is in Sparse Checkout for Large Monorepos.

Step 5 — Schedule Maintenance So the Gains Persist Jump to heading#

# Register the repository with the background scheduler
git maintenance start

# Inspect what was registered and when it runs
git maintenance register
git config --get-regexp '^maintenance\.'

# Disable the legacy blocking collector, which the scheduler replaces
git config maintenance.auto false

# Confirm the commit-graph exists and history queries use it
ls .git/objects/info/commit-graph*
time git log --oneline -5000 > /dev/null    # compare with the Step 1 baseline

The commit-graph is the least visible and most valuable part. It precomputes commit ancestry so that git log --graph, git merge-base, and every branch comparison in a review tool answer from a compact structure rather than by walking objects. On a repository with a hundred thousand commits it routinely turns multi-second operations into instant ones — and it is exactly the sort of thing that decays silently, which is why scheduling it matters more than running it once.

Integration With Adjacent Practice Jump to heading#

Boundary with branch topology. Sparse checkout only works if the repository’s directory structure matches how teams divide work. If every change touches files in five top-level directories, no sparse configuration will help. That is a layout question, answered in Monorepo Branch Topology and enforced through path-based CODEOWNERS.

Boundary with CI. A CI runner has different needs from a developer: it wants one commit, no history, and nothing cached between runs. There, --depth=1 --filter=blob:none --single-branch is close to optimal, whereas on a developer machine the same flags would break blame and bisect. Configure the two separately; the trigger-scoping techniques in CI/CD Pipeline Trigger Mapping reduce how often the checkout happens at all.

Boundary with prevention. Every technique here manages weight that already exists. Stopping new weight from arriving is a hook problem, handled by file size limits on the remote. Without that, the measurements from Step 1 regress within months.

Configuration Reference Jump to heading#

Common Failure Modes and Diagnostics Jump to heading#

A partial clone is slower, not faster. Symptom: everyday commands pause repeatedly. Root cause: a workflow that touches historical file contents — git log -p, a full-history search, or a tool that diffs every commit. Fix: keep a full clone for that workflow, or git fetch --refetch the blobs you need in one batch rather than thousands of individual fetches.

Sparse checkout did not speed up status. Symptom: the working tree is smaller but timings are unchanged. Root cause: non-cone patterns, so the sparse index is disabled. Fix: git sparse-checkout init --cone and re-express the patterns as directories; confirm with git config index.sparse.

The repository grows again after a cleanup. Symptom: size returns to its previous value within a quarter. Root cause: nothing prevents new large files. Fix: add a server-side size limit and route genuinely large assets to Git LFS.

git status is slow despite everything. Symptom: seconds of latency with a modest file count. Root cause: no filesystem monitor on a platform where directory scanning is expensive, or an antivirus scanner inspecting the working tree. Fix: enable core.fsmonitor, and exclude the repository from real-time scanning.

Fetches are slow but the repository is small. Symptom: git fetch takes far longer than the data transferred justifies. Root cause: tens of thousands of refs — usually abandoned branches or per-build tags — enumerated on every negotiation. Fix: prune merged branches on a schedule, and move build markers out of refs entirely.

Team Rollout Jump to heading#

• Capture the four baseline measurements from Step 1 on a representative machine
• Publish a recommended clone command in the contributing guide, with the filter flags
• Enable git maintenance start as part of onboarding, and turn off maintenance.auto
• Introduce sparse checkout with one willing team before recommending it broadly
• Add a size limit on the remote so the improvement is not temporary
• Re-measure after a month and publish the before-and-after numbers

Frequently Asked Questions Jump to heading#

Is a shallow clone the same as a partial clone? Jump to heading#

No, and the difference matters. A shallow clone truncates history to a depth, so commits simply do not exist locally and operations such as blame, bisect and merge-base either fail or give wrong answers. A partial clone keeps every commit but omits blobs until something needs them, fetching on demand. Partial clone is the right default for developer machines; shallow clone is for throwaway CI checkouts that never need history.

Will these changes break anything for developers who do nothing? Jump to heading#

No. Partial clone, sparse checkout and scheduled maintenance are all per-clone client-side settings — they change how one machine talks to the remote, not what the remote contains. A colleague who keeps a full clone continues to work exactly as before, and the two can push and pull to each other’s branches without noticing any difference.

How often should git maintenance run? Jump to heading#

The built-in schedule is a good default: hourly prefetch, daily loose-object packing and commit-graph writes, weekly incremental repack. What matters more than the frequency is that it runs in the background rather than interrupting a developer mid-command, which is exactly what git maintenance start arranges and what the old automatic gc did not.

Does rewriting history to remove large files help? Jump to heading#

It is the only thing that genuinely shrinks the repository rather than deferring the cost, but it invalidates every commit SHA after the rewrite point and forces every clone to be re-created. Reach for it when the repository is unusable and the alternatives have been exhausted — and treat it as a scheduled migration with a communication plan, not a Tuesday-afternoon cleanup.

Why is my repository slow even though it is only a few hundred megabytes? Jump to heading#

Size is often not the cause. A repository with a hundred thousand loose objects, no commit-graph, or tens of thousands of refs will feel slow at any size, because the cost is in object lookup and ref enumeration rather than bytes on disk. Run git count-objects -vH and count your refs before assuming the fix is to make the repository smaller.

Related Jump to heading#

• Speeding Up Clones with Partial Clone — the blobless-clone recipe end to end, including what gets slower and how to tell whether the trade is worth it.
• Scheduling git maintenance for Background Repacking — replace the blocking automatic collector with a scheduler that never interrupts a command.
• Migrating Large Binaries to Git LFS — move the assets that should never have been in the object store, without losing their history.
• Sparse Checkout for Large Monorepos — the working-tree half of the problem, including cone mode and build systems that expect the full tree.
• Enforcing File Size Limits on the Remote — the prevention that stops today’s cleanup from being repeated next year.