Repository navigation
repack: write filtered-out objects as loose objects - #2314
Open
achakilam-nv wants to merge 2 commits into
Open
achakilam-nv wants to merge 2 commits into
achakilam-nv wants to merge 2 commits into
Conversation
…objects Some external object stores (like key-value) manage and retrieve git objects individually. Although repack's --filter-to option can separate objects from the main object store, it writes them into packs. Using those objects individually then requires an additional unpacking step. Add --filter-to-loose=<dir> to write filtered-out objects directly as loose objects into <dir>, treating it as an object directory. Stream the filtered pack into unpack-objects instead of writing an intermediate pack file to disk. The filtered pack is built with --stdin-packs, so this initially handles only objects already stored in packs. Loose objects in the source repository are left in place. Signed-off-by: Adithya Chakilam <achakilam@nvidia.com>
--filter-to-loose only writes objects that were in an existing pack, because the filtered pack is built with --stdin-packs. Loose objects that the filter excludes stay behind in the source repository, so the destination does not get the complete set of filtered-out objects. Copy such loose objects to the destination as well. An object already present in the destination is not rewritten. With -d, the source copies are removed once everything is written. Signed-off-by: Adithya Chakilam <achakilam@nvidia.com>
Author
|
/submit |
|
Submitted as pull.2314.git.1794607911.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
| @@ -12,7 +12,7 @@ SYNOPSIS | |||
| 'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] | |||
There was a problem hiding this comment.
Junio C Hamano wrote on the Git mailing list (how to reply to this email):
"Adithya Chakilam via GitGitGadget" <gitgitgadget@gmail.com> writes:
> From: Adithya Chakilam <achakilam@nvidia.com>
>
> Some external object stores (like key-value) manage and retrieve git
> objects individually. Although repack's --filter-to option can separate
> objects from the main object store, it writes them into packs. Using
> those objects individually then requires an additional unpacking step.
>
> Add --filter-to-loose=<dir> to write filtered-out objects directly as
> loose objects into <dir>, treating it as an object directory. Stream
> the filtered pack into unpack-objects instead of writing an intermediate
> pack file to disk.
>
> The filtered pack is built with --stdin-packs, so this initially handles
> only objects already stored in packs. Loose objects in the source
> repository are left in place.
I haven't decided whether this new feature makes sense or not, so I
won't comment on that aspect. Take the following review comments
with that in mind (i.e., pretend as if I said "if we assume doing
this makes sense, then..." before every sentence).
> +--filter-to-loose=<dir>::
> + Like `--filter-to`, but treat `<dir>` as an object directory
> + and write the filtered out objects there as loose objects
> + instead of as a pack. Cannot be combined with `--filter-to`
> + or `--max-pack-size`. Objects already present in `<dir>` are
> + not rewritten. All objects are written before any existing
> + pack is removed.
We talk about "already present in ...", so presumably we require the
directory to exist before we run. Have a test that ensures that
passing a non-existing directory will make the command error out
before doing any real work.
> diff --git a/builtin/repack.c b/builtin/repack.c
> index 21835453c6..140a992f1b 100644
> --- a/builtin/repack.c
> +++ b/builtin/repack.c
> @@ -41,7 +41,7 @@ static const char *const git_repack_usage[] = {
> N_("git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n"
> "[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n"
> "[--write-midx[=<mode>]] [--name-hash-version=<n>] [--path-walk]\n"
> - "[--filter=<filter-spec>] [--drop-filtered [--dry-run]]"),
> + "[--filter=<filter-spec> [--filter-to-loose=<dir>]] [--drop-filtered [--dry-run]]"),
The above documentation and help text consistently tell us that the
value passed is a directory (do we use the abbreviated <dir> so
commonly? If not, it would make sense to spell it out, i.e.,
<directory>, for clarity).
> + OPT_STRING(0, "filter-to-loose", &filter_to_loose, N_("dir"),
> + N_("object directory to write filtered out objects into as loose objects")),
Here, too.
This should use OPT_FILENAME so that a "git repack" started from a
subdirectory with a relative path to the external loose object store
will find the directory at the correct place, I think. Have a test
to validate the combination to see that such a repack that was given
a relative path to a directory find ...
mkdir extra-loose-objects &&
mkdir sub/dir &&
(
cd sub/dir &&
git repack --filter-to-loose=../../extra-loose-objects ...
)
the directory correct at the right place.
> diff --git a/repack-filtered.c b/repack-filtered.c
> index 869b9fc6e3..51c9dfeb13 100644
> --- a/repack-filtered.c
> +++ b/repack-filtered.c
> @@ -29,10 +50,18 @@ int write_filtered_pack(const struct write_pack_opts *opts,
> strvec_pushf(&cmd.args, "--keep-pack=%s", item->string);
>
> cmd.in = -1;
> + if (opts->loose)
> + cmd.out = unpack.in;
>
> ret = start_command(&cmd);
> - if (ret)
> + if (ret) {
> + if (opts->loose) {
> + close(unpack.in);
Please double check that this is not closing the same thing twice.
IIRC, a failing start_command() would close all file descriptors
associated with cmd so you may have already closed unpack.in when
start_command() gave us a failure, which presumably closed cmd.out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some key-value object stores manage and retrieve Git objects
individually. The existing --filter-to option writes filtered-out
objects into packs, so using it with such stores requires unpacking
the resulting packs before importing the objects.
This series adds --filter-to-loose=dir to git repack, allowing
filtered-out objects to be written directly as loose objects into
another object directory.
With --filter-to-loose, repack streams the filtered pack into
unpack-objects, avoiding that extra step and the intermediate pack
file. For example:
git repack -a --filter=blob=1m --filter-to-loose=/path/to/objects
The destination is treated as a Git object directory.
The first patch adds the option for objects already stored in packs.
The second extends it to include loose objects excluded by the
filter, so the destination also receives those objects. Objects
already present in the destination are not rewritten. With -d,
the source copies of those loose objects are removed after all
objects have been written successfully.
cc: Martin Fick mfick@nvidia.com
cc: Christian Couder christian.couder@gmail.com