Handling API rate limits in Git automation

Forge APIs limit how many requests an identity can make. The primary limit is a budget per hour — a few thousand requests for a token, more for an app installation. Secondary limits apply to bursts: too many concurrent requests, too many writes per minute, too much CPU time on expensive queries. Scripts that loop over repositories, pull requests or commits hit these limits sooner than expected, and the failure modes are unpleasant: a job dies halfway through a bulk change, a bot stops responding during a busy afternoon, or the identity is temporarily blocked and every automation sharing it stops too. This page shows how to read the limits, back off correctly, and make fewer calls in the first place, within forge API and webhook automation.

When to use this approach

  • Scripts fail intermittently with 403 “rate limit exceeded” or 429 responses.
  • A bot slows down or stops responding at busy times.
  • You are about to run a loop over hundreds of repositories or thousands of pull requests, as in bulk updating repository settings with the API.
  • Several automations share one token and interfere with each other.

Step 1 — Read your budget before and during a run

Every response carries headers describing the budget: the limit, how much remains and when it resets. A dedicated endpoint reports the same without spending a request.

gh api rate_limit --jq '.resources | {core: .core, graphql: .graphql, search: .search}
  | to_entries[] | "\(.key): \(.value.remaining)/\(.value.limit), resets \(.value.reset | todate)"'

# Headers on any response
gh api -i "repos/$OWNER/$REPO" | grep -i '^x-ratelimit-'

Before a large run, estimate the calls it needs — repositories × calls per repository — and compare with the remaining budget. If it does not fit, plan to spread it or reduce the calls first.

Primary and secondary rate limitsThe primary limit is an hourly budget of requests per identity, reported in headers and reset on a schedule. Secondary limits protect the service from bursts — concurrency, rapid writes, expensive queries — and are signalled by 403 or 429 responses with a Retry-After header rather than a predictable budget.Primary limitSecondary limitswhat it capsrequests per hourbursts, writes, concurrencyvisible in advanceyes, headersnosignal403 + remaining 0403/429 + Retry-Afterfixspread, reduce callsserialise, slow downstaying under the hourly budget does not protect you from bursting

Step 2 — Back off correctly when limited

When a response says you are limited, wait — for the time Retry-After gives, or until the reset time for the primary limit — then retry. Never retry immediately in a loop; that extends secondary limits and can get the identity blocked.

#!/bin/sh
# api_call <args...> — gh api with rate-limit aware retries
api_call() {
  attempt=0
  while :; do
    out=$(gh api -i "$@" 2>&1) && { printf '%s\n' "$out" | sed '1,/^\r\{0,1\}$/d'; return 0; }
    status=$(printf '%s\n' "$out" | sed -n '1s/^HTTP[^ ]* \([0-9]*\).*/\1/p')
    retry=$(printf '%s\n' "$out" | sed -n 's/^[Rr]etry-[Aa]fter: *\([0-9]*\).*/\1/p')
    reset=$(printf '%s\n' "$out" | sed -n 's/^[Xx]-[Rr]ate[Ll]imit-[Rr]eset: *\([0-9]*\).*/\1/p')
    case "$status" in
      403|429) wait=${retry:-$(( ${reset:-$(date +%s)} - $(date +%s) + 5 ))}
               [ "$wait" -gt 0 ] || wait=$(( 2 ** attempt * 10 ))
               echo "rate limited, sleeping ${wait}s" >&2; sleep "$wait" ;;
      5??)     sleep $(( 2 ** attempt * 5 )) ;;
      *)       printf '%s\n' "$out" >&2; return 1 ;;
    esac
    attempt=$((attempt + 1)); [ "$attempt" -lt 6 ] || return 1
  done
}
What to do with a failed API callIf the response is 403 or 429 with a Retry-After header, wait that long and retry. If it is 403 with the primary budget exhausted, wait until the reset time. If it is a 5xx error, retry with exponential backoff. Any other error is a real failure and should stop the script.What did the API return?403/429 + Retry-AfterWait as toldthen retry403, remaining = 0Wait for resetX-RateLimit-Reset5xxExponential backoffmax a few tries404, 422 and other 4xx errors are not retryable — fix the request instead

Step 3 — Avoid secondary limits on writes

Secondary limits are most often hit by writes: creating many comments, labels, branches or pull requests quickly, or making concurrent requests. Serialise writes and add a short pause between them; do not parallelise mutating calls.

# Serial writes with a small gap — slower per item, far faster than being blocked
while read -r repo; do
  api_call -X PATCH "repos/$repo" -F delete_branch_on_merge=true >/dev/null
  sleep 1
done < repos.txt

Reads can be parallelised modestly — a handful at a time — but writes should be one at a time.

Step 4 — Make fewer calls

The best limit handling is not needing it. Three techniques cut call counts substantially.

# 1. GraphQL: fetch nested data in one request instead of one per item
gh api graphql -f query='
  query($org: String!) { organization(login: $org) {
    repositories(first: 100, isArchived: false) { nodes {
      nameWithOwner deleteBranchOnMerge squashMergeAllowed
      defaultBranchRef { name } } } } }' -f org="$ORG"

# 2. Conditional requests: unchanged resources return 304 and usually do not count
etag=$(gh api -i "repos/$OWNER/$REPO" | sed -n 's/^[Ee][Tt]ag: *//p' | tr -d '\r')
gh api -i -H "If-None-Match: $etag" "repos/$OWNER/$REPO" | head -1      # HTTP/2.0 304

# 3. Webhooks instead of polling: react to events rather than asking repeatedly
API calls to audit 400 repositoriesAn illustrative audit of merge settings and default branches across four hundred repositories. One REST call per setting per repository costs over a thousand requests. One REST call per repository costs four hundred. A paginated GraphQL query fetching the same fields costs four.requests for one audit of 400 repositories (illustrative)REST, per setting1200REST, per repo400GraphQL, paginated4GraphQL queries have their own cost budget, but simple field reads are cheap

Step 5 — Give each automation its own identity

When several automations share one token, one noisy job exhausts the budget for all of them. Separate identities give separate budgets and make it obvious which job is responsible for a spike. GitHub App installations also receive larger limits that scale with the organisation.

# Which identity is a token? Check before sharing it with another job
gh api user --jq .login 2>/dev/null || gh api installation/repositories --jq '.total_count'

App identities for automation are introduced in forge API and webhook automation and used in verified bot commits with a GitHub App identity.

Validation checklist

Frequently Asked Questions

Do GITHUB_TOKEN workflow tokens have their own limit?

Yes, per repository, and it is lower than an app installation’s. Workflows that make many API calls across repositories should use an app token instead.

Does GitLab have the same limits?

GitLab applies per-user and per-IP limits, configurable on self-managed instances, and returns 429 with Retry-After and RateLimit-* headers. The same wrapper logic applies with the header names adjusted.

Are search API limits different?

Search endpoints have a much smaller, separate budget. Avoid using search for things that can be listed directly, and cache search results within a run.