Skip to content

merge-base: do not keep commit buffers in memory - #2316

Draft
spkrka wants to merge 1 commit into
gitgitgadget:masterfrom
spkrka:merge-base-no-buffer
Draft

spkrka wants to merge 1 commit into
gitgitgadget:masterfrom
spkrka:merge-base-no-buffer

Conversation

@spkrka

@spkrka spkrka commented Nov 16, 2026 •

Copy link
Copy Markdown

Some context, since this patch comes a bit out of nowhere: Patrick's
recent work on reducing the memory used by "git repo structure" [1]
made me look for other places that keep many objects in memory for
the whole process. Looking at "git merge-base" without a
commit-graph, the memory used per commit was much higher than the
commit objects alone could explain. It turned out that the command
keeps the full buffer of every commit it parses, even though it never
looks at commit messages.

Without a commit-graph, this cuts the memory of a deep walk on
linux.git by about 4x.

[1] https://lore.kernel.org/git/20261116-pks-repo-count-objects-speedup-v2-0-9c8be218eaff@pks.im/

@spkrka
spkrka force-pushed the merge-base-no-buffer branch from 5b0a0a3 to 3b4e1dc Compare November 16, 2026 13:02
"git merge-base" only ever prints object IDs, but it keeps the default
save_commit_buffer = 1, so every commit that is parsed from the object
database during the walk keeps its full buffer cached until the
process exits.  Without a commit-graph, that is the bulk of the memory
used by a deep walk.

Turn it off like other commands that do not need commit messages,
such as "git describe" and "git rev-list" without a pretty format.
This only affects the builtin; callers of the merge-base machinery
that need the buffers keep them.

This is the same reasoning as in dd2e50a (commit-graph: turn off
save_commit_buffer, 2019-09-07), 359b01c (ref-filter: disable
save_commit_buffer while traversing, 2022-07-11) and 5f64279
(upload-pack: always turn off save_commit_buffer, 2024-02-28).

Measured on linux.git at 37e2f878a7a6 (1.46M commits), anonymous
memory at exit for

  git -c core.commitGraph=false merge-base --all 37e2f878a7a6 <base>

  <base>          before     after
  e2062df704de     607MB     130MB
  33d8881    1560MB     373MB

Wall time is unchanged within noise: each object is read and inflated
for parsing either way, and keeping the buffer only skips freeing it.
With a commit-graph covering the walk, commits are parsed from the
graph without reading their buffers, so there is no change.

Signed-off-by: Kristofer Karlsson <krka@spotify.com>
@spkrka
spkrka force-pushed the merge-base-no-buffer branch from 3b4e1dc to ca72623 Compare November 16, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant