Verifying submodule commits are reachable upstream

A submodule is a pointer: the parent repository records a commit hash, and Git fetches that commit from the submodule’s remote when someone clones. Nothing checks, at commit time, that the remote actually has that commit. A developer commits inside the submodule, forgets to push it, updates the pointer in the parent and pushes the parent. Their own checkout works perfectly. Everyone else’s clone fails with “reference is not a tree”, CI goes red, and the only copy of the missing commit is on one laptop. A quieter version happens when the pointed-to commit lived on a branch that was later force-pushed or deleted. This page adds the checks that catch both, within submodule and dependency integrity.

When to use this approach

  • Your repository uses submodules, and their pointers change through pull requests.
  • You have seen “fatal: reference is not a tree” or “did not contain” errors in fresh clones.
  • Submodule repositories allow branch deletion or force-pushes, so previously reachable commits can disappear.
  • You want a pointer to be accepted only if it names a commit on a branch the submodule’s owners consider stable — for the update process itself, see pinning and updating Git submodules safely.

Step 1 — Make Git refuse to push an unpushed submodule commit

The cheapest fix is on the developer’s machine. push.recurseSubmodules=check makes git push in the parent refuse if any submodule commit it points to has not been pushed to the submodule’s remote.

git config --global push.recurseSubmodules check
# or push the submodule commits automatically before the parent
git config --global push.recurseSubmodules on-demand

check fails loudly and lets the developer decide. on-demand pushes the submodule for them, which is convenient but can push work-in-progress commits to a shared remote. Most teams prefer check.

# Verification: commit in the submodule, update the pointer, try to push the parent
git -C vendor/lib commit --allow-empty -m "local only"
git add vendor/lib && git commit -m "bump lib"
git push    # expect: "The following submodule paths contain changes that can not be found on any remote"

Step 2 — Check reachability in CI, against the real remote

Local configuration helps, but CI is the gate. For every submodule pointer that changed in the pull request, fetch the submodule from its remote and confirm the commit exists there.

#!/bin/sh
# ci/check-submodule-reachable.sh <base> <head>
set -eu
base=$1 head=$2
git diff --submodule=short "$base" "$head" --raw |
awk '$1 ~ /^:160000/ || $2 ~ /^160000/ {print $NF}' |      # mode 160000 = gitlink
while read -r path; do
  sha=$(git ls-tree "$head" "$path" | awk '{print $3}')
  url=$(git config -f .gitmodules --get "submodule.$path.url")
  tmp=$(mktemp -d)
  git -C "$tmp" init -q
  if git -C "$tmp" fetch -q --depth=1 "$url" "$sha" 2>/dev/null; then
    echo "ok       $path @ ${sha%${sha#???????}}"
  else
    echo "MISSING  $path @ $sha not fetchable from $url"; fail=1
  fi
  rm -rf "$tmp"
done

Fetching a specific commit by hash requires the server to allow it; most hosted forges do for reachable commits. A fetch that fails is exactly the signal you want.

Checking a changed submodule pointer in CIThe job finds gitlink entries that changed between base and head, reads each submodule's URL from .gitmodules, tries to fetch exactly the pointed-to commit into an empty repository, and fails the pull request if any fetch fails.Changed gitlinksmode 160000URL.gitmodulesFetch by hashempty repo, depth 1Passcommit existsFailname the pathan empty repository per check means nothing cached locally can mask a missing commit

Step 3 — Require the commit to be on a stable branch

Reachable is not enough if the commit sits on a feature branch that will be deleted next week. Check that the pointed-to commit is an ancestor of one of the submodule’s protected branches.

# Inside the loop, after a successful fetch
git -C "$tmp" fetch -q "$url" "+refs/heads/main:refs/remotes/origin/main" "+refs/heads/release/*:refs/remotes/origin/release/*"
if ! git -C "$tmp" for-each-ref --format='%(refname)' refs/remotes/origin |
     while read -r r; do git -C "$tmp" merge-base --is-ancestor "$sha" "$r" && exit 0; done; then
  echo "UNSTABLE $path @ $sha is not on main or a release branch"; fail=1
fi
Is this submodule pointer acceptable?A pointer to a commit that cannot be fetched is rejected immediately. A fetchable commit that is only on a feature branch is rejected as unstable. A commit that is an ancestor of the submodule's main or a release branch is accepted.Where does the pointed-to commit live?nowhere on the remoteRejectclones will failfeature branch onlyRejectmay vanish latermain or release branchAcceptstable and reachablethe middle case is the one that breaks builds months later

Step 4 — Audit existing pointers across history

New pointers are checked from now on, but old ones may already point at commits that have since disappeared — which breaks checking out old tags and bisecting. A periodic audit walks the pointers at every release tag.

for tag in $(git tag --list 'v*'); do
  git ls-tree -r "$tag" | awk '$1=="160000"{print $4, $3}' |
  while read -r path sha; do
    url=$(git show "$tag:.gitmodules" | git config -f - --get "submodule.$path.url" || true)
    git ls-remote "$url" >/dev/null 2>&1 || { echo "$tag $path: remote gone ($url)"; continue; }
    tmp=$(mktemp -d); git -C "$tmp" init -q
    git -C "$tmp" fetch -q --depth=1 "$url" "$sha" 2>/dev/null || echo "$tag $path: $sha missing"
    rm -rf "$tmp"
  done
done

Missing commits found this way can sometimes be recovered from a developer’s clone or a CI cache and pushed back to a dedicated refs/keep/* namespace in the submodule’s repository, so history becomes buildable again without polluting branches.

Step 5 — Protect the commits you depend on

The root cause of disappearing commits is a submodule repository that allows rewriting the branches you point at. Protect those branches against force-pushes and deletion, and consider a mirror you control, as described in mirroring third-party repositories for resilience.

Layers that keep submodule pointers validDeveloper configuration stops unpushed commits leaving the laptop, the CI check stops pointers to missing or unstable commits from merging, branch protection on the submodule stops reachable commits being rewritten away, and a periodic audit finds what slipped through in the past.from the developer's push to years laterpush.recurseSubmodules=checkno unpushed commits leaveCI reachability checkonly stable, fetchable pointers mergeSubmodule branch protectionpointed-to commits cannot vanishTag auditold releases stay buildableeach layer catches a failure the others cannot see

Validation checklist

Frequently Asked Questions

Why not just use git submodule update --init in CI as the check?

It does fail on a missing commit, but CI runners often have cached clones or mirrors that contain commits the real remote does not. Fetching into an empty repository from the URL in .gitmodules tests what a new contributor will experience.

Does this work when submodules use relative URLs?

Yes, if you resolve them first. A URL starting with ../ is relative to the parent’s remote URL; compute the absolute URL from git remote get-url origin before fetching.

What about submodules pointing at private repositories?

The CI job needs read credentials for them, which it already needs to clone recursively. Use a read-only deploy key or token scoped to exactly those repositories.