Repository navigation
Conversation
5262dea to
4adc9a7
Compare
4adc9a7 to
169ca59
Compare
|
/submit |
|
Submitted as pull.2301.git.1794063565.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
|
Junio C Hamano wrote on the Git mailing list (how to reply to this email): "Kristofer Karlsson via GitGitGadget" <gitgitgadget@gmail.com>
writes:
> While looking for slow tests in CI, t4205 stood out on macOS: its EXPENSIVE
> tests take several times longer there than on Linux. A CI experiment showed
> that much of the time was spent in tr(1), which is very slow on macOS for
> the 2GB of data that t4205 generates.
>
> Patch 1 renames test-genzeros.c so that the actual changes in patch 2 show
> up as a regular diff. Patch 2 adds "genbytes" and uses it instead of
> "genzeros | tr", which is where the speedup comes from.
Just being curious. Would forcing C locale on it help? I.e.,
genzeros | LANG=C LC_ALL=C tr ...
|
| TEST_BUILTINS_OBJS += test-dump-split-index.o | ||
| TEST_BUILTINS_OBJS += test-dump-untracked-cache.o | ||
| TEST_BUILTINS_OBJS += test-env-helper.o | ||
| TEST_BUILTINS_OBJS += test-example-tap.o |
There was a problem hiding this comment.
Junio C Hamano wrote on the Git mailing list (how to reply to this email):
"Kristofer Karlsson via GitGitGadget" <gitgitgadget@gmail.com>
writes:
> From: Kristofer Karlsson <krka@spotify.com>
>
> The next commit adds a "genbytes" command to this file and turns
> "genzeros" into a thin wrapper around it. Rename the file in a
> separate step first, so that git can detect the rename and the actual
> changes show up as a regular diff, which is easier to review.
>
> No functional changes.
>
> Signed-off-by: Kristofer Karlsson <krka@spotify.com>
> ---
> Makefile | 2 +-
> t/helper/meson.build | 2 +-
> t/helper/{test-genzeros.c => test-genbytes.c} | 0
> 3 files changed, 2 insertions(+), 2 deletions(-)
> rename t/helper/{test-genzeros.c => test-genbytes.c} (100%)
Needless to say, with this patch applied, the following tests would
now fail, unable to find "test-tool genzeros".
t0613-reftable-write-options.sh
t1007-hash-object.sh
t1051-large-conversion.sh
t1450-fsck.sh
t4141-apply-too-large.sh
t4205-log-pretty-formats.sh
t5300-pack-object.sh
t5318-commit-graph.sh
t7006-pager.sh
t9850-shell.sh
FWIW, if you apply [1/3] and [2/3], squash them together, then give
a custom rename factor to the command, e.g.,
$ git format-patch -M3 -1
to format the resulting single patch, you will see a modification of
genzeros while the file gets renamed just fine, and you would not
have to break these tests in the middle of a series.There was a problem hiding this comment.
Junio C Hamano wrote on the Git mailing list (how to reply to this email):
Junio C Hamano <gitster@pobox.com> writes:
> Needless to say, with this patch applied, the following tests would
> now fail, unable to find "test-tool genzeros".
Not really. Even though the source file is renamed, the dispatcher
inside still recognises genzeros and calls cmd__genzeros. In fact,
at this stage, "test-tool genbytes" would not even work (and it does
not have to work as nobody calls it).
So, sorry for the false alarm, but my point about not split the
rename + modification into two separate steps still stands.
Thanks.
>
> t0613-reftable-write-options.sh
> t1007-hash-object.sh
> t1051-large-conversion.sh
> t1450-fsck.sh
> t4141-apply-too-large.sh
> t4205-log-pretty-formats.sh
> t5300-pack-object.sh
> t5318-commit-graph.sh
> t7006-pager.sh
> t9850-shell.sh
>
> FWIW, if you apply [1/3] and [2/3], squash them together, then give
> a custom rename factor to the command, e.g.,
>
> $ git format-patch -M3 -1
>
> to format the resulting single patch, you will see a modification of
> genzeros while the file gets renamed just fine, and you would not
> have to break these tests in the middle of a series.There was a problem hiding this comment.
Kristofer Karlsson wrote on the Git mailing list (how to reply to this email):
On Sun, 8 Nov 2026 at 05:44, Junio C Hamano <gitster@pobox.com> wrote:
>
> Junio C Hamano <gitster@pobox.com> writes:
>
> > Needless to say, with this patch applied, the following tests would
> > now fail, unable to find "test-tool genzeros".
>
> Not really. Even though the source file is renamed, the dispatcher
> inside still recognises genzeros and calls cmd__genzeros. In fact,
> at this stage, "test-tool genbytes" would not even work (and it does
> not have to work as nobody calls it).
>
> So, sorry for the false alarm, but my point about not split the
> rename + modification into two separate steps still stands.
>
[...]
> > FWIW, if you apply [1/3] and [2/3], squash them together, then give
> > a custom rename factor to the command, e.g.,
> >
> > $ git format-patch -M3 -1
> >
> > to format the resulting single patch, you will see a modification of
> > genzeros while the file gets renamed just fine, and you would not
> > have to break these tests in the middle of a series.
When using the GGG workflow I think that option is not available,
though I will see if I can try formatting and sending the patch
manually for this one instead. (Or try to submit a patch to GGG
to allow for configuring this too.)
I could squash it all into a single patch too, which I think
would make sense if we don't need the intermediate state
where both genzeros and genbytes exist at the same time.
What's your preference?
Thanks,
KristoferThere was a problem hiding this comment.
Junio C Hamano wrote on the Git mailing list (how to reply to this email):
Kristofer Karlsson <krka@spotify.com> writes:
> When using the GGG workflow I think that option is not available,
> though I will see if I can try formatting and sending the patch
> manually for this one instead. (Or try to submit a patch to GGG
> to allow for configuring this too.)
>
> I could squash it all into a single patch too, which I think
> would make sense if we don't need the intermediate state
> where both genzeros and genbytes exist at the same time.
>
> What's your preference?
I have no strong preference between keeping both genzeros() and
genbytes() or rewriting users of genzeros() to use genbytes(), and I
prefer to leave that out of the decision.
My preference is to optimize for those who consume the resulting
history the most over the longer term.
Your split between a "pure rename" and a "modification in place"
is optimizing for those who review the patches on the mailing list
(arguably due to tool limitations). While they are an important
audience of yours (after all, without their support, your change
would not end up in the final history), those who will read the
history via "git log" or "git show" to see what happened later are
the more important readers of your work in the long run. To them,
a single rename-and-modify commit is much easier to understand, I
would think.
There was a problem hiding this comment.
Kristofer Karlsson wrote on the Git mailing list (how to reply to this email):
On Sun, 8 Nov 2026 at 19:35, Junio C Hamano <gitster@pobox.com> wrote:
>
> > I could squash it all into a single patch too, which I think
> > would make sense if we don't need the intermediate state
> > where both genzeros and genbytes exist at the same time.
> >
> > What's your preference?
>
> I have no strong preference between keeping both genzeros() and
> genbytes() or rewriting users of genzeros() to use genbytes(), and I
> prefer to leave that out of the decision.
>
> My preference is to optimize for those who consume the resulting
> history the most over the longer term.
>
> Your split between a "pure rename" and a "modification in place"
> is optimizing for those who review the patches on the mailing list
> (arguably due to tool limitations). While they are an important
> audience of yours (after all, without their support, your change
> would not end up in the final history), those who will read the
> history via "git log" or "git show" to see what happened later are
> the more important readers of your work in the long run. To them,
> a single rename-and-modify commit is much easier to understand, I
> would think.
That's fair -- For v2 I will squash patches 1 and 2 and keep
patch 3. I'll also add the lowercase hex as input format.
The similarity seems to be ~29%, so running it with -M20% would
only mildly reduce the diff, though I think the diff is
manageable to consume anyway.
Thanks,
Kristofer|
Junio C Hamano wrote on the Git mailing list (how to reply to this email): "Kristofer Karlsson via GitGitGadget" <gitgitgadget@gmail.com>
writes:
> Open question: is patch 3 wanted? It folds "genzeros" into "genbytes",
> spelling the NUL byte as "NUL", so that there is a single command.
I would suggest genbytes to take the value of the byte in decimal or
hex, instead of av[1][0], so that you do not have to treat NUL any
specially. I would actually think it is even better to limit the
input format only to [0-9a-f][0-9a-f], i.e., two hexadecimal digits
that does not allow uppercase to represent values 0-255 (inclusive),
without support for any other input format like decimal.
After all, this is just a test helper, so ergonomics or flexibility
do not matter as much as simplicity.
> Some other decisions that may be worth pointing out:
>
> * The buffer is no longer static: it used to be static so that it was
> zero-initialized for free, but now that it is filled with the requested
> byte on every call, a local buffer avoids shared state. It keeps its size
> of 256KB. Smaller buffers made no measurable difference in my tests, but
> there was no reason to change it.
Shouldn't matter.
> * Invalid counts are still silently treated as 0, as before. That could be
> tightened, but it is unrelated to this series.
Shouldn't matter. That's ergonomics and flexibility that we do not
value as much as in a normal program.
Thanks. |
169ca59 to
cbf1d9e
Compare
| @@ -0,0 +1,59 @@ | |||
| #define DISABLE_SIGN_COMPARE_WARNINGS | |||
|
|
|||
| #include "test-tool.h" | |||
There was a problem hiding this comment.
Johannes Sixt wrote on the Git mailing list (how to reply to this email):
Am 07.11.26 um 15:59 schrieb Kristofer Karlsson via GitGitGadget:
> Several tests create fixture data consisting of a single repeated
> character with "test-tool genzeros <count> | tr '\000' <char>". This
> works well on platforms where tr(1) is implemented efficiently.
>
> The implementation of tr(1) on macOS supports multi-byte characters,
> and processes its input one character at a time through
> getwchar() and putwchar(), even in the C locale. For small inputs
> that does not matter, but t4205 generates more than 2GB of data this
> way, which makes that test much slower on macOS than on Linux: on the
> macOS CI runners, tr(1) alone spends 95 to 123 seconds of CPU time on
> it, compared to about 6 seconds on Linux.
>
> Fix this by adding "test-tool genbytes <byte> [<count>]", which writes
> the requested byte directly, and turn "genzeros" into a thin
> wrapper around the same code.
All cases that do not need a particular character can use
printf %0.500d 0
Though, on my Linux box,
printf %0.2147483649d 0
produces nothing, but
printf 000000000%0.2147483640d 0
has 2147483649 bytes of output.
If this isn't insanely slow on macOS, use this pattern instead of a
highly engineered helper program? ;-)
-- Hannes
There was a problem hiding this comment.
Kristofer Karlsson wrote on the Git mailing list (how to reply to this email):
On Sun, 8 Nov 2026 at 09:54, Johannes Sixt <j6t@kdbg.org> wrote:
>
> Am 07.11.26 um 15:59 schrieb Kristofer Karlsson via GitGitGadget:
> > Several tests create fixture data consisting of a single repeated
> > character with "test-tool genzeros <count> | tr '\000' <char>". This
> > works well on platforms where tr(1) is implemented efficiently.
> >
> > The implementation of tr(1) on macOS supports multi-byte characters,
> > and processes its input one character at a time through
> > getwchar() and putwchar(), even in the C locale. For small inputs
> > that does not matter, but t4205 generates more than 2GB of data this
> > way, which makes that test much slower on macOS than on Linux: on the
> > macOS CI runners, tr(1) alone spends 95 to 123 seconds of CPU time on
> > it, compared to about 6 seconds on Linux.
> >
> > Fix this by adding "test-tool genbytes <byte> [<count>]", which writes
> > the requested byte directly, and turn "genzeros" into a thin
> > wrapper around the same code.
>
> All cases that do not need a particular character can use
>
> printf %0.500d 0
>
> Though, on my Linux box,
>
> printf %0.2147483649d 0
>
> produces nothing, but
>
> printf 000000000%0.2147483640d 0
>
> has 2147483649 bytes of output.
>
> If this isn't insanely slow on macOS, use this pattern instead of a
> highly engineered helper program? ;-)
>
> -- Hannes
>
Nice trick, and it would work for t4205, which does not care about
the byte value. I tried it for the 2147483649 bytes there on Linux:
printf time peak memory
dash 6.5s 2.1GB
bash 6.0s 3MB
coreutils 3.7s 2.1GB
It is definitely fast enough (on the variants I tested).
The annoying thing (but not a real blocker) is the extra memory
usage required and I am not sure I fully trust that all
implementations will behave the same for absurdly large padding.
(Particularly for paddings larger than 2^31, as you noted too.)
It feels a bit like a hack, and tweaking the existing test-tool
feels like a more sane option that is easier to reason about.
A gotcha here is that the printf hack only works for producing
"0", so for the cases we truly need NUL characters it does not
help us, and I think that is also a point in favor of
simply generalizing test-tool genzeros instead.
Thanks,
KristoferThere was a problem hiding this comment.
Johannes Sixt wrote on the Git mailing list (how to reply to this email):
Am 08.11.26 um 10:18 schrieb Kristofer Karlsson:
> On Sun, 8 Nov 2026 at 09:54, Johannes Sixt <j6t@kdbg.org> wrote:
>> printf 000000000%0.2147483640d 0
>
> It is definitely fast enough (on the variants I tested).
> The annoying thing (but not a real blocker) is the extra memory
> usage required and I am not sure I fully trust that all
> implementations will behave the same for absurdly large padding.
>
> (Particularly for paddings larger than 2^31, as you noted too.)
I don't know how to measure memory requirements, but if it helps,
printf 000000000%02147483640d 0
(without the fullstop) works, too.
> It feels a bit like a hack, and tweaking the existing test-tool
> feels like a more sane option that is easier to reason about.
>
> A gotcha here is that the printf hack only works for producing
> "0", so for the cases we truly need NUL characters it does not
> help us, and I think that is also a point in favor of
> simply generalizing test-tool genzeros instead.
On the other hand, the only case you really care about is the one that
takes an insane amount of time. If you convert only that case to printf,
then we do not need all this churn in the first place.
-- Hannes
There was a problem hiding this comment.
Kristofer Karlsson wrote on the Git mailing list (how to reply to this email):
On Sun, 8 Nov 2026 at 10:35, Johannes Sixt <j6t@kdbg.org> wrote:
>
> On the other hand, the only case you really care about is the one that
> takes an insane amount of time. If you convert only that case to printf,
> then we do not need all this churn in the first place.
That's a good point -- if we want to make a minimal fix, we could
simply switch t4205 to use printf instead of "genzeros | tr".
- test-tool genzeros 2147483649 | tr "\000" "1" >expect &&
+ printf 000000000%02147483640d 0 >expect &&
I tried this out and it works on Linux but fails on macOS[1]. What
happens is that printf only outputs the nine literal zeros
("000000000") there, and silently ignores the very large padding.
[1] https://github.com/gitgitgadget/git/actions/runs/42475956442/job/128717309586
The setup and the next test still pass, since the latter compares
against the same file, and only the test expecting the exact size in
the error message notices. So I do not think we can rely on printf
for this, and I still think that generalizing the existing helper is
a reasonable way forward.
I think even if printf _would_ have worked, the end state of having
a uniform way of creating this fake data is nicer than having three
separate ways:
- genzeros when we only need NUL bytes
- genzeros | tr for small inputs that need some other byte
- printf for large inputs where "0" bytes are valid
(And if we ever need to support large inputs with a specific byte
value, we'd get stuck again, though anticipating such a need is
speculative.)
Thanks,
KristoferThere was a problem hiding this comment.
Johannes Sixt wrote on the Git mailing list (how to reply to this email):
Am 08.11.26 um 13:24 schrieb Kristofer Karlsson:
> - test-tool genzeros 2147483649 | tr "\000" "1" >expect &&
> + printf 000000000%02147483640d 0 >expect &&
>
> I tried this out and it works on Linux but fails on macOS[1]. What
> happens is that printf only outputs the nine literal zeros
> ("000000000") there, and silently ignores the very large padding.
Fair enough.
> [1] https://github.com/gitgitgadget/git/actions/runs/42475956442/job/128717309586
-- Hannes
There was a problem hiding this comment.
Johannes Sixt wrote on the Git mailing list (how to reply to this email):
Am 08.11.26 um 17:34 schrieb Johannes Sixt:
> Am 08.11.26 um 13:24 schrieb Kristofer Karlsson:
>> - test-tool genzeros 2147483649 | tr "\000" "1" >expect &&
>> + printf 000000000%02147483640d 0 >expect &&
>>
>> I tried this out and it works on Linux but fails on macOS[1]. What
>> happens is that printf only outputs the nine literal zeros
>> ("000000000") there, and silently ignores the very large padding.
>
> Fair enough.
>
>> [1] https://github.com/gitgitgadget/git/actions/runs/42475956442/job/128717309586
Well, this patch passed for me[2], but I have no idea whether the
particular tests were skipped due to the EXPENSIVE prerequisite or not.
[2] https://github.com/j6t/git/actions/runs/42583623620/job/129048078411
diff --git a/t/t4205-log-pretty-formats.sh b/t/t4205-log-pretty-formats.sh
index 01b97c8888..5dedb9054e 100755
--- a/t/t4205-log-pretty-formats.sh
+++ b/t/t4205-log-pretty-formats.sh
@@ -1182,7 +1182,8 @@ unfold () {
'
test_expect_success EXPENSIVE,SIZE_T_IS_64BIT 'set up huge commit' '
- test-tool genzeros 2147483649 | tr "\000" "1" >expect &&
+ # 3 * 715827883 == 2147483649
+ printf %0715827883d 0 0 0 >expect &&
huge_commit=$(git commit-tree -F expect HEAD^{tree})
'
There was a problem hiding this comment.
Kristofer Karlsson wrote on the Git mailing list (how to reply to this email):
On Mon, 9 Nov 2026 at 10:33, Johannes Sixt <j6t@kdbg.org> wrote:
>
> Am 08.11.26 um 17:34 schrieb Johannes Sixt:
> > Am 08.11.26 um 13:24 schrieb Kristofer Karlsson:
> >> - test-tool genzeros 2147483649 | tr "\000" "1" >expect &&
> >> + printf 000000000%02147483640d 0 >expect &&
> >>
> >> I tried this out and it works on Linux but fails on macOS[1]. What
> >> happens is that printf only outputs the nine literal zeros
> >> ("000000000") there, and silently ignores the very large padding.
> >
> > Fair enough.
> >
> >> [1] https://github.com/gitgitgadget/git/actions/runs/42475956442/job/128717309586
>
> Well, this patch passed for me[2], but I have no idea whether the
> particular tests were skipped due to the EXPENSIVE prerequisite or not.
>
> [2] https://github.com/j6t/git/actions/runs/42583623620/job/129048078411
>
>
>
> diff --git a/t/t4205-log-pretty-formats.sh b/t/t4205-log-pretty-formats.sh
> index 01b97c8888..5dedb9054e 100755
> --- a/t/t4205-log-pretty-formats.sh
> +++ b/t/t4205-log-pretty-formats.sh
> @@ -1182,7 +1182,8 @@ unfold () {
> '
>
> test_expect_success EXPENSIVE,SIZE_T_IS_64BIT 'set up huge commit' '
> - test-tool genzeros 2147483649 | tr "\000" "1" >expect &&
> + # 3 * 715827883 == 2147483649
> + printf %0715827883d 0 0 0 >expect &&
> huge_commit=$(git commit-tree -F expect HEAD^{tree})
I tested this and it passes on all the macOS jobs. t4205 goes
from 800-900s to 170-280s there, so your idea works well.
Two caveats: the printf builtin in bash 3.2 (/bin/sh on macOS)
silently prints nothing for a field wider than INT_MAX - 15,
and it builds each field in memory, so this peaks at about 2.1GB
RSS. Both go away by writing in 1MiB chunks, at the same speed:
gen_ascii_zeros () {
_n=$1
while test "$_n" -gt 0
do
_chunk=$(( _n > 1048576 ? 1048576 : _n ))
printf "%0${_chunk}d" 0 || return 1
_n=$(( _n - _chunk ))
done
}
as a local helper in t4205. That fixes the slow test without a
new test-tool, at the cost of yet another way of generating test
data alongside genzeros and "genzeros | tr".
I have it ready locally as a single patch, as an alternative to
this series, but perhaps it's best to wait for more people to
chime in first.
Thanks,
KristoferThere was a problem hiding this comment.
Johannes Sixt wrote on the Git mailing list (how to reply to this email):
Am 09.11.26 um 14:05 schrieb Kristofer Karlsson:
> On Mon, 9 Nov 2026 at 10:33, Johannes Sixt <j6t@kdbg.org> wrote:
>> test_expect_success EXPENSIVE,SIZE_T_IS_64BIT 'set up huge commit' '
>> - test-tool genzeros 2147483649 | tr "\000" "1" >expect &&
>> + # 3 * 715827883 == 2147483649
>> + printf %0715827883d 0 0 0 >expect &&
>> huge_commit=$(git commit-tree -F expect HEAD^{tree})
>
> I tested this and it passes on all the macOS jobs. t4205 goes
> from 800-900s to 170-280s there, so your idea works well.
>
> Two caveats: the printf builtin in bash 3.2 (/bin/sh on macOS)
> silently prints nothing for a field wider than INT_MAX - 15,
> and it builds each field in memory, so this peaks at about 2.1GB
> RSS. Both go away by writing in 1MiB chunks, at the same speed:
>
> gen_ascii_zeros () {
> _n=$1
> while test "$_n" -gt 0
> do
> _chunk=$(( _n > 1048576 ? 1048576 : _n ))
> printf "%0${_chunk}d" 0 || return 1
> _n=$(( _n - _chunk ))
> done
> }
I wouldn't do that (and pay the memory price instead), because the next
shell doesn't have printf as a builtin and spawns 2000 printf processes.
Now let's stop painting bikeshes. I'd be fine with your previous
suggestion to have both `test-tool genzeros` and `test-tool genbytes`.
-- Hannes
There was a problem hiding this comment.
Collin Funk wrote on the Git mailing list (how to reply to this email):
Kristofer Karlsson <krka@spotify.com> writes:
> Nice trick, and it would work for t4205, which does not care about
> the byte value. I tried it for the 2147483649 bytes there on Linux:
>
> printf time peak memory
> dash 6.5s 2.1GB
> bash 6.0s 3MB
> coreutils 3.7s 2.1GB
>
> It is definitely fast enough (on the variants I tested).
> The annoying thing (but not a real blocker) is the extra memory
> usage required and I am not sure I fully trust that all
> implementations will behave the same for absurdly large padding.
>
> (Particularly for paddings larger than 2^31, as you noted too.)
>
> It feels a bit like a hack, and tweaking the existing test-tool
> feels like a more sane option that is easier to reason about.
>
> A gotcha here is that the printf hack only works for producing
> "0", so for the cases we truly need NUL characters it does not
> help us, and I think that is also a point in favor of
> simply generalizing test-tool genzeros instead.
I've mentioned this idea on coreutils or gnulib's mailing lists before,
but I forget where and when. I've wanted to create a printf variant that
doesn't allocate much memory and will just use putc_unlocked for each
padding character, which would avoid the memory issue you bring up.
Most platforms will be limited to INT_MAX there, since that is what
POSIX specifies is supplied with '*' [1]:
A field width, or precision, or both, may be indicated by an
<asterisk> ('*'). In this case an argument of type int supplies the
field width or precision.
E.g., in an invocation like this where the field width is supplied with
another argument:
$ printf '%0*d\n' 5 1
00001
I think my original reason for wanting a different printf variant was
escaping that limit.
For generating bytes where the values don't matter, in GNU coreutils we
frequently use 'yes' for that. As a result, the implementation is very
overengineered. E.g., it will use zero-copy since coreutils-9.11. Before
that, it would reuse ARGV if it was contiguous and the arguments used to
determine the printed string were long enough such that we weren't calling
read/write on a tiny string, or it would generate a large buffer before
using read and write. Here is the performance from before and after
zero-copy was introduced on a beefy Power10 machine, stolen from the
commit message [2]:
IBM,9043-MRX $ taskset 1 yes | taskset 2 pv > /dev/null
... [11.6GiB/s]
IBM,9043-MRX $ taskset 1 src/yes | taskset 2 pv > /dev/null
... [175GiB/s]
This was introduced around the time of the copy-fail vulnerability and
various zero-copy vulnerabilities, where there was serious consideration
given to nerfing splice and vmsplice. Luckily for us, that never
happened.
FreeBSD, and therefore, I assume, macOS use something similar to our
previous read/write method, which is fast enough. I think OpenBSD and
NetBSD use puts on ARGV or the "y" string, so it's relatively slow.
GNU 'tr' doesn't handle multi-byte characters, so you can adjust that
output if needed and remain relatively quick:
$ timeout 10 yes 0 | LC_ALL=C tr -d '\n' | pv -r > /dev/null
[ 616MiB/s]
I'm not sure if macOS 'tr' handles multibyte characters. But it might be
worth checking if they do, and if they keep a fast case for single-byte
locales. Doing that without duplicating code is quite tricky, which is
part of the reason why we haven't done it in GNU 'tr' yet (but recent
releases have improved multi-byte character support for other utilities,
so hopefully it is handled soon).
Probably more information than you need, but hopefully it helps you
generate slop sometimes needed for testing. :)
Collin
[1] https://pubs.opengroup.org/onlinepubs/9799919799/functions/printf.html
[2] https://github.com/coreutils/coreutils/commit/2b1c059e6a06eebbb721d010b1221ec54200cc33There was a problem hiding this comment.
Kristofer Karlsson wrote on the Git mailing list (how to reply to this email):
On Tue, 10 Nov 2026 at 07:08, Collin Funk <collin.funk1@gmail.com> wrote:
>
> I'm not sure if macOS 'tr' handles multibyte characters. But it might be
> worth checking if they do, and if they keep a fast case for single-byte
> locales. Doing that without duplicating code is quite tricky, which is
> part of the reason why we haven't done it in GNU 'tr' yet (but recent
> releases have improved multi-byte character support for other utilities,
> so hopefully it is handled soon).
Thanks, this was an interesting read (even if it may not solve
the problem at hand).
To answer your question about macOS tr: it does not seem to have a
single-byte fast path. Apple's tr (text_cmds tr/tr.c) goes
through getwchar()/putwchar() for every byte even with LC_ALL=C,
which is why "genzeros | tr" takes about 2 minutes for 2GB there,
compared to a few seconds with GNU tr.
"yes" is nicely optimized, but without an option to drop the
newlines we would still need to filter its output, and filtering
is exactly what is slow on macOS. I suppose the primitive I would
ideally want is something like "repeat <string> [<count>]".
In the end portability is the main problem: with tr and printf
behaving this differently across platforms, a tiny test-tool
helper is easier to reason about than any combination of shell
tools.
Thanks,
Kristofer
>
> Collin
>
> [1] https://pubs.opengroup.org/onlinepubs/9799919799/functions/printf.html
> [2] https://github.com/coreutils/coreutils/commit/2b1c059e6a06eebbb721d010b1221ec54200cc33
>|
User |
|
Kristofer Karlsson wrote on the Git mailing list (how to reply to this email): On Sun, 8 Nov 2026 at 00:04, Junio C Hamano <gitster@pobox.com> wrote:
>
> Just being curious. Would forcing C locale on it help? I.e.,
>
> genzeros | LANG=C LC_ALL=C tr ...
>
No, test-lib-functions.sh already sets those env variables,
but the macos implementation of tr always executes the slow path
unfortunately.
Thanks,
Kristofer |
|
Kristofer Karlsson wrote on the Git mailing list (how to reply to this email): On Sun, 8 Nov 2026 at 00:23, Junio C Hamano <gitster@pobox.com> wrote:
>
> "Kristofer Karlsson via GitGitGadget" <gitgitgadget@gmail.com>
> writes:
>
> > Open question: is patch 3 wanted? It folds "genzeros" into "genbytes",
> > spelling the NUL byte as "NUL", so that there is a single command.
>
> I would suggest genbytes to take the value of the byte in decimal or
> hex, instead of av[1][0], so that you do not have to treat NUL any
> specially. I would actually think it is even better to limit the
> input format only to [0-9a-f][0-9a-f], i.e., two hexadecimal digits
> that does not allow uppercase to represent values 0-255 (inclusive),
> without support for any other input format like decimal.
>
> After all, this is just a test helper, so ergonomics or flexibility
> do not matter as much as simplicity.
Yes, I actually had one local prototype that used this approach,
but I held off on it since it would mean a semantically larger
change. But I agree it would improve the ergonomics, so I can do it.
And it's not too many call sites to convert anyway.
Thanks,
Kristofer |
cbf1d9e to
f49af0e
Compare
|
/submit |
|
Submitted as pull.2301.v2.git.1794165304.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
|
User |
| TEST_BUILTINS_OBJS += test-dump-split-index.o | ||
| TEST_BUILTINS_OBJS += test-dump-untracked-cache.o | ||
| TEST_BUILTINS_OBJS += test-env-helper.o | ||
| TEST_BUILTINS_OBJS += test-example-tap.o |
There was a problem hiding this comment.
Junio C Hamano wrote on the Git mailing list (how to reply to this email):
"Kristofer Karlsson via GitGitGadget" <gitgitgadget@gmail.com>
writes:
> diff --git a/t/helper/test-genbytes.c b/t/helper/test-genbytes.c
> new file mode 100644
> index 0000000000..b760326b24
> --- /dev/null
> +++ b/t/helper/test-genbytes.c
> @@ -0,0 +1,61 @@
> +#define DISABLE_SIGN_COMPARE_WARNINGS
> +
> +#include "test-tool.h"
> +#include "git-compat-util.h"
> +#include "hex-ll.h"
> +
> +/*
> + * Write "count" copies of "byte" to stdout, or an endless stream of them if
> + * "count" is negative.
> + */
> +static int generate_bytes(char byte, intmax_t count)
> +{
> + char buf[256 * 1024];
Allocating 256kB on stack may not be a great practice, but this may
sufficiently work on most reasonable systems, as this is an internal
of a "run once and exit" test helper function.
Still, the stack memory for the primary thread of a process is
limited to 8MB or so on modern Linux systems, and I hear macOS also
uses 8MB. I am not familiar with Windows, but some documentation
found on the Web seems to say that their linker gives 1MB to their
product by default, and among 1MB, 256kB is not an insignificant
fraction.
Again, this is an internal of a "run once and exit" test helper, so
declaring this as "static char buf[]" should not pose an issue; it is
not like you would need to have multiple simultaneous invocations of
this function, each of which needs to fill this buffer with
different values.There was a problem hiding this comment.
Kristofer Karlsson wrote on the Git mailing list (how to reply to this email):
On Mon, 9 Nov 2026 at 02:39, Junio C Hamano <gitster@pobox.com> wrote:
>
> "Kristofer Karlsson via GitGitGadget" <gitgitgadget@gmail.com>
> writes:
>
> > diff --git a/t/helper/test-genbytes.c b/t/helper/test-genbytes.c
> > new file mode 100644
> > index 0000000000..b760326b24
> > --- /dev/null
> > +++ b/t/helper/test-genbytes.c
> > @@ -0,0 +1,61 @@
> > +#define DISABLE_SIGN_COMPARE_WARNINGS
> > +
> > +#include "test-tool.h"
> > +#include "git-compat-util.h"
> > +#include "hex-ll.h"
> > +
> > +/*
> > + * Write "count" copies of "byte" to stdout, or an endless stream of them if
> > + * "count" is negative.
> > + */
> > +static int generate_bytes(char byte, intmax_t count)
> > +{
> > + char buf[256 * 1024];
>
> Allocating 256kB on stack may not be a great practice, but this may
> sufficiently work on most reasonable systems, as this is an internal
> of a "run once and exit" test helper function.
>
> Still, the stack memory for the primary thread of a process is
> limited to 8MB or so on modern Linux systems, and I hear macOS also
> uses 8MB. I am not familiar with Windows, but some documentation
> found on the Web seems to say that their linker gives 1MB to their
> product by default, and among 1MB, 256kB is not an insignificant
> fraction.
>
> Again, this is an internal of a "run once and exit" test helper, so
> declaring this as "static char buf[]" should not pose an issue; it is
> not like you would need to have multiple simultaneous invocations of
> this function, each of which needs to fill this buffer with
> different values.
Yes, I think we have a few options to choose between:
1. keep the previous static. Technically this means that we
should also document it as not being thread-safe.
Currently it does not matter, since it's never called from
anywhere else, but there is a slight code smell to it
since the buffer no longer hosts a constant value.
2. remove static and keep the size. Since the stack is not
complex or deep here it should be safe. It would break
on platforms with a very small stack, but since it's only
use by CI and developers any breakage would be immediately
obvious and easy to fix.
3. remove static and shrink the buffer size. From my (limited)
local experiment, it seems the buffer is larger than needed.
We could shrink it without losing performance.
I don't have a strong opinion here; I went with (2) because it
seemed like the smallest change but I'd be happy to change it.
If we go with static, I'd prefer to have a small comment
above the function definition:
/* not thread-safe (due to the mutable static 'buf') */
Thanks,
KristoferThere was a problem hiding this comment.
Junio C Hamano wrote on the Git mailing list (how to reply to this email):
Kristofer Karlsson <krka@spotify.com> writes:
> 2. remove static and keep the size. Since the stack is not
> complex or deep here it should be safe. It would break
> on platforms with a very small stack, but since it's only
> use by CI and developers any breakage would be immediately
> obvious and easy to fix.
>
> 3. remove static and shrink the buffer size. From my (limited)
> local experiment, it seems the buffer is larger than needed.
> We could shrink it without losing performance.
>
> I don't have a strong opinion here; I went with (2) because it
> seemed like the smallest change but I'd be happy to change it.
>
> If we go with static, I'd prefer to have a small comment
> above the function definition:
>
> /* not thread-safe (due to the mutable static 'buf') */
I do not think such a comment is not warranted in this case, though.
After all this is more or less throw-away code used only by CI and
developers ;-)| TEST_BUILTINS_OBJS += test-dump-split-index.o | ||
| TEST_BUILTINS_OBJS += test-dump-untracked-cache.o | ||
| TEST_BUILTINS_OBJS += test-env-helper.o | ||
| TEST_BUILTINS_OBJS += test-example-tap.o |
There was a problem hiding this comment.
Patrick Steinhardt wrote on the Git mailing list (how to reply to this email):
On Sun, Nov 08, 2026 at 07:15:03PM +0000, Kristofer Karlsson via GitGitGadget wrote:
[snip]
> This makes t4205 much faster on macOS in CI. The Linux jobs, shown
> for comparison, only save the few seconds that GNU tr(1) took, which
> is within the usual noise:
>
> job before after change
> osx-gcc 821.1s 243.3s -70.4%
> osx-reftable 911.0s 194.0s -78.7%
> linux-asan-ubsan 226.7s 222.1s -2.0%
> linux-leaks 320.2s 298.8s -6.7%
Nice improvement.
> diff --git a/t/helper/test-genbytes.c b/t/helper/test-genbytes.c
> new file mode 100644
> index 0000000000..b760326b24
> --- /dev/null
> +++ b/t/helper/test-genbytes.c
> @@ -0,0 +1,61 @@
Okay. The diff is unfortunately quite useless as we do the file rename
plus all the changes in a single step. It might've been helpful to split
this up into:
1. A rename of the file.
2. A refactoring of `cmd_genzeros()` to use the write buffer.
3. A change to introduce `cmd_genbytes()`.
But maybe I'm also overdoing this.
> +#define DISABLE_SIGN_COMPARE_WARNINGS
> +
> +#include "test-tool.h"
> +#include "git-compat-util.h"
> +#include "hex-ll.h"
> +
> +/*
> + * Write "count" copies of "byte" to stdout, or an endless stream of them if
> + * "count" is negative.
> + */
> +static int generate_bytes(char byte, intmax_t count)
> +{
> + char buf[256 * 1024];
> + ssize_t n;
> +
> + memset(buf, byte, sizeof(buf));
> +
> + /* Writing out individual bytes is slow... */
> + while (count < 0)
> + if (xwrite(1, buf, ARRAY_SIZE(buf)) < 0)
> + die_errno("write error");
> +
> + while (count > 0) {
> + n = xwrite(1, buf,
> + count < ARRAY_SIZE(buf)
> + ? count : ARRAY_SIZE(buf));
> +
> + if (n < 0)
> + die_errno("write error");
> +
> + count -= n;
> + }
Okay. These two loops are a bit ugly if you ask me, but they are carried
over from preexisting code. So this is good enough.
> + return 0;
> +}
> +
> +static intmax_t parse_count(int argc, const char **argv, int pos)
> +{
> + return pos < argc ? strtoimax(argv[pos], NULL, 0) : -1;
> +}
We can use `git_parse_signed()` instead.
> diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh
> index 0b27d75fe3..b2032a799d 100755
> --- a/t/t0613-reftable-write-options.sh
> +++ b/t/t0613-reftable-write-options.sh
> @@ -146,7 +146,7 @@ test_expect_success 'small block size fails with large reflog message' '
> (
> cd repo &&
> test_commit A &&
> - test-tool genzeros 500 | tr "\000" "a" >logmsg &&
> + test-tool genbytes 61 500 >logmsg &&
> cat >expect <<-EOF &&
> fatal: update_ref failed for ref ${SQ}refs/heads/logme${SQ}: reftable: transaction failure: entry too large
> EOF
I wonder about the regression in readability though. Maybe that's just
me, but I don't happen to know all ASCII values by heart, so if I were
to read this I would now wonder what's so special about 61. And the next
thing to wonder about is whether this is supposed to encode "=" (decimal
61) or "a" (hexadecimal 61), and without reading the code of genbytes I
would not be able to tell.
So can we maybe make the underpinning logic a bit more intelligent, e.g.
by accepting "\NNN" or "\xNN" syntax instead of two hex characters? That
might also make it more extensible going forward if the next developer
wants to make genbytes take and repeat strings of arbitrary length
as-is.
PatrickThere was a problem hiding this comment.
Kristofer Karlsson wrote on the Git mailing list (how to reply to this email):
On Mon, 9 Nov 2026 at 07:59, Patrick Steinhardt <ps@pks.im> wrote:
>
> Okay. The diff is unfortunately quite useless as we do the file rename
> plus all the changes in a single step. It might've been helpful to split
> this up into:
>
> 1. A rename of the file.
>
> 2. A refactoring of `cmd_genzeros()` to use the write buffer.
>
> 3. A change to introduce `cmd_genbytes()`.
>
> But maybe I'm also overdoing this.
v1 had the rename as a separate commit, and Junio preferred to have
it squashed into the change. I do not have a strong opinion either
way, so I will wait for a consensus before sending v3.
>
> > + return 0;
> > +}
> > +
> > +static intmax_t parse_count(int argc, const char **argv, int pos)
> > +{
> > + return pos < argc ? strtoimax(argv[pos], NULL, 0) : -1;
> > +}
>
> We can use `git_parse_signed()` instead.
Good idea. It would also reject invalid counts, which are currently
silently treated as 0.
> I wonder about the regression in readability though. Maybe that's just
> me, but I don't happen to know all ASCII values by heart, so if I were
> to read this I would now wonder what's so special about 61. And the next
> thing to wonder about is whether this is supposed to encode "=" (decimal
> 61) or "a" (hexadecimal 61), and without reading the code of genbytes I
> would not be able to tell.
>
> So can we maybe make the underpinning logic a bit more intelligent, e.g.
> by accepting "\NNN" or "\xNN" syntax instead of two hex characters? That
> might also make it more extensible going forward if the next developer
> wants to make genbytes take and repeat strings of arbitrary length
> as-is.
That made me look at what the callers actually need:
callers need
t1007 (4x), t5318 NUL bytes
t1051 (2x), t1450 (2x), any byte, only the size matters
t4141
t0613, t4205, t7006, t9850 any byte except NUL and newline
t5300 three different non-NUL bytes
So none of them needs a particular byte value, and the only special
case is NUL, which "genzeros" already covers. For everything else, a
plain character is the most readable:
Given that, I am now leaning towards keeping "genzeros" as it is and
adding "genbytes <char> [<count>]" for the other callers, i.e. going
back to v1 without its last patch (and without a separate rename
patch). That avoids inventing a syntax for NUL, keeps the call
sites readable, and also leaves the existing "genzeros" callers
alone, which addresses Johannes' concern about the churn.
Junio, would that be acceptable, given your suggestion to use hex
digits so that NUL does not need special treatment?
The rename is perhaps less important to do if genzeros is kept
(though it also gains the genbytes function).
Thanks,
Kristofer|
This patch series was integrated into seen via git@b7caf80. |
f49af0e to
17e5baa
Compare
|
User |
|
This branch is now known as |
|
/submit |
|
Submitted as pull.2301.v3.git.1794335168170.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
|
There was a status update in the "New Topics" section about the branch The test helper used for generating large files with repeated bytes has been replaced with test-tool genbytes that generates them natively, bypassing slow tr performance on macOS. Expecting a reroll. cf. <CAL71e4Mrw2h-D+yKLVE30q-7dnkT31CVsyVmktmS9Wx6Xu6ocQ@mail.gmail.com> source: <pull.2301.v2.git.1794165304.gitgitgadget@gmail.com> |
|
Junio C Hamano wrote on the Git mailing list (how to reply to this email): > * test-genzeros.c is no longer renamed, and "genbytes" lives next to
> "genzeros" in the same file. Since "genzeros" stays, the current file
> name is still valid and not worth churning.
Yes, it is a great realization that the file will have both genbytes
and genzeros in it in the end, and there is no strong reason why it
needs to be renamed.
> diff --git a/t/helper/test-genzeros.c b/t/helper/test-genzeros.c
> index b895436a32..e8d83d5012 100644
> --- a/t/helper/test-genzeros.c
> +++ b/t/helper/test-genzeros.c
> @@ -2,30 +2,28 @@
>
> #include "test-tool.h"
> #include "git-compat-util.h"
> +#include "parse.h"
>
> +/*
> + * Write "count" copies of "byte" to stdout, or an endless stream of them if
> + * "count" is negative.
> + */
> +static int generate_bytes(unsigned char byte, intmax_t count)
> {
> + static char buf[256 * 1024];
> ssize_t n;
>
> + memset(buf, byte, sizeof(buf));
>
> + /* Writing out individual bytes is slow... */
> while (count < 0)
> + if (xwrite(1, buf, ARRAY_SIZE(buf)) < 0)
> die_errno("write error");
A curious infinite loop but that is OK.
> while (count > 0) {
> + n = xwrite(1, buf,
> + count < ARRAY_SIZE(buf)
> + ? count : ARRAY_SIZE(buf));
>
> if (n < 0)
> die_errno("write error");
> @@ -35,3 +33,29 @@ int cmd__genzeros(int argc, const char **argv)
>
> return 0;
> }
> +int cmd__genbytes(int argc, const char **argv)
> +{
> + intmax_t count = -1;
> +
> + if (argc < 2 || argc > 3 || strlen(argv[1]) != 1 ||
> + (argc == 3 && !git_parse_signed(argv[2], &count, INTMAX_MAX))) {
> + fprintf(stderr, "usage: %s <char> [<count>]\n", argv[0]);
> + return 1;
> + }
> +
> + return generate_bytes(argv[1][0], count);
> +}
> +
> +int cmd__genzeros(int argc, const char **argv)
> +{
> + intmax_t count = -1;
> +
> + if (argc > 2 ||
> + (argc == 2 && !git_parse_signed(argv[1], &count, INTMAX_MAX))) {
> + fprintf(stderr, "usage: %s [<count>]\n", argv[0]);
> + return 1;
> + }
> +
> + return generate_bytes('\0', count);
> +}
The callers are so straight-forward.
> diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh
> diff --git a/t/t4205-log-pretty-formats.sh b/t/t4205-log-pretty-formats.sh
> diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh
> diff --git a/t/t7006-pager.sh b/t/t7006-pager.sh
All conversions of these consumers are so trivially correct.
> diff --git a/t/t9850-shell.sh b/t/t9850-shell.sh
> index 27cc49bbd1..d56edebda7 100755
> --- a/t/t9850-shell.sh
> +++ b/t/t9850-shell.sh
> @@ -30,7 +30,7 @@ test_expect_success 'shell allows interactive command' '
> '
>
> test_expect_success 'shell complains of overlong commands' '
> - test-tool genzeros | tr "\000" "a" |
> + test-tool genbytes a |
> test_must_fail git shell 2>err &&
> test_grep "too long" err
> '
OK, so that's how an infinitely long sequence of characters is used.
Looking good. Let's mark the topic as ready for 'next'. |
Several tests create fixture data consisting of a single repeated character with "test-tool genzeros <count> | tr '\000' <char>". This works well on platforms where tr(1) is implemented efficiently. The implementation of tr(1) on macOS supports multi-byte characters, and processes its input one character at a time through getwchar() and putwchar(), even in the C locale. For small inputs that does not matter, but t4205 generates more than 2GB of data this way, which makes that test much slower on macOS than on Linux: on the macOS CI runners, tr(1) alone spends 95 to 123 seconds of CPU time on it, compared to about 6 seconds on Linux. Fix this by adding "test-tool genbytes <char> [<count>]" next to "genzeros", sharing the same code, which writes the requested character directly. Use it in all tests that piped "genzeros" through tr(1), not just t4205, so that it is the obvious choice for new tests. Callers that want NUL bytes keep using "genzeros". While at it, parse the count with git_parse_signed(), so that an invalid count is reported as a usage error instead of being silently treated as 0. This makes t4205 much faster on macOS in CI. The Linux jobs, shown for comparison, show smaller differences, which may partly be due to normal run-to-run variation: job before after change osx-gcc 821.1s 243.3s -70.4% osx-reftable 911.0s 194.0s -78.7% linux-asan-ubsan 226.7s 222.1s -2.0% linux-leaks 320.2s 298.8s -6.7% Signed-off-by: Kristofer Karlsson <krka@spotify.com>
17e5baa to
c94a8e0
Compare
|
/submit |
|
Submitted as pull.2301.v4.git.1794412751763.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
|
Patrick Steinhardt wrote on the Git mailing list (how to reply to this email): On Tue, Nov 10, 2026 at 06:26:08PM +0000, Kristofer Karlsson via GitGitGadget wrote:
> diff --git a/t/helper/test-genzeros.c b/t/helper/test-genzeros.c
> index b895436a32..e8d83d5012 100644
> --- a/t/helper/test-genzeros.c
> +++ b/t/helper/test-genzeros.c
> @@ -35,3 +33,29 @@ int cmd__genzeros(int argc, const char **argv)
>
> return 0;
> }
> +
> +int cmd__genbytes(int argc, const char **argv)
> +{
> + intmax_t count = -1;
> +
> + if (argc < 2 || argc > 3 || strlen(argv[1]) != 1 ||
> + (argc == 3 && !git_parse_signed(argv[2], &count, INTMAX_MAX))) {
> + fprintf(stderr, "usage: %s <char> [<count>]\n", argv[0]);
> + return 1;
> + }
> +
> + return generate_bytes(argv[1][0], count);
> +}
Okay, so instead of interpreting `argv[0]` we now just take it as-is and
verify that it's a single character, only. That makes users way easier
to reason about because you don't have to wonder about how the argument
is encoded. It's also extensible because we can easily introduce longer
strings or "\x00" and "\000" eventually without breaking any existing
callers.
> +int cmd__genzeros(int argc, const char **argv)
> +{
> + intmax_t count = -1;
> +
> + if (argc > 2 ||
> + (argc == 2 && !git_parse_signed(argv[1], &count, INTMAX_MAX))) {
> + fprintf(stderr, "usage: %s [<count>]\n", argv[0]);
> + return 1;
> + }
> +
> + return generate_bytes('\0', count);
> +}
And this one here is trivially adapted.
> diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh
> index 0b27d75fe3..0b74ff6157 100755
> --- a/t/t0613-reftable-write-options.sh
> +++ b/t/t0613-reftable-write-options.sh
> @@ -146,7 +146,7 @@ test_expect_success 'small block size fails with large reflog message' '
> (
> cd repo &&
> test_commit A &&
> - test-tool genzeros 500 | tr "\000" "a" >logmsg &&
> + test-tool genbytes a 500 >logmsg &&
> cat >expect <<-EOF &&
> fatal: update_ref failed for ref ${SQ}refs/heads/logme${SQ}: reftable: transaction failure: entry too large
> EOF
And all of the callers are easier to understand now.
Thanks, this looks great!
Patrick |
|
Junio C Hamano wrote on the Git mailing list (how to reply to this email): "Kristofer Karlsson via GitGitGadget" <gitgitgadget@gmail.com>
writes:
> From: Kristofer Karlsson <krka@spotify.com>
>
> Several tests create fixture data consisting of a single repeated
> character with "test-tool genzeros <count> | tr '\000' <char>". This
> works well on platforms where tr(1) is implemented efficiently.
This looks good as-is, but once we merge this down, we will find
that there is a new use of the pattern that pipes test-tool genzeros
into tr in t/t1600-index.sh recently merged as part of the Git
2.50.2 update.
Perhaps we should rebase to catch that callsite?
Thanks.
|
|
Kristofer Karlsson wrote on the Git mailing list (how to reply to this email): On Wed, 11 Nov 2026 at 16:38, Junio C Hamano <gitster@pobox.com> wrote:
>
> "Kristofer Karlsson via GitGitGadget" <gitgitgadget@gmail.com>
> writes:
>
> > From: Kristofer Karlsson <krka@spotify.com>
> >
> > Several tests create fixture data consisting of a single repeated
> > character with "test-tool genzeros <count> | tr '\000' <char>". This
> > works well on platforms where tr(1) is implemented efficiently.
>
> This looks good as-is, but once we merge this down, we will find
> that there is a new use of the pattern that pipes test-tool genzeros
> into tr in t/t1600-index.sh recently merged as part of the Git
> 2.50.2 update.
>
> Perhaps we should rebase to catch that callsite?
>
> Thanks.
>
Oh that was unlucky timing, but easy to fix. Not only is this
new test using tr, it's also using very large values.
I'll submit a v4 rebased on latest master, which should fix it.
Thanks,
Kristofer |
|
Junio C Hamano wrote on the Git mailing list (how to reply to this email): Kristofer Karlsson <krka@spotify.com> writes:
> On Wed, 11 Nov 2026 at 16:38, Junio C Hamano <gitster@pobox.com> wrote:
>>
>> "Kristofer Karlsson via GitGitGadget" <gitgitgadget@gmail.com>
>> writes:
>>
>> > From: Kristofer Karlsson <krka@spotify.com>
>> >
>> > Several tests create fixture data consisting of a single repeated
>> > character with "test-tool genzeros <count> | tr '\000' <char>". This
>> > works well on platforms where tr(1) is implemented efficiently.
>>
>> This looks good as-is, but once we merge this down, we will find
>> that there is a new use of the pattern that pipes test-tool genzeros
>> into tr in t/t1600-index.sh recently merged as part of the Git
>> 2.50.2 update.
>>
>> Perhaps we should rebase to catch that callsite?
>>
>> Thanks.
>>
>
> Oh that was unlucky timing, but easy to fix. Not only is this
> new test using tr, it's also using very large values.
> I'll submit a v4 rebased on latest master, which should fix it.
Thanks for a prompt response. |
|
There was a status update in the "Cooking" section about the branch The test helper used for generating large files with repeated bytes has been updated to avoid slow 'tr' performance on macOS. A new 'test-tool genbytes' now generates repeated bytes natively, replacing the 'genzeros | tr' pipeline in several tests. Will merge to 'next'? cf. <xmqqo6bwqy9n.fsf@gitster.g> cf. <avRJPDEJWTGWySKp@pks.im> source: <pull.2301.v4.git.1794412751763.gitgitgadget@gmail.com> |
|
This patch series was integrated into next via git@16177fe. |
|
There was a status update in the "Cooking" section about the branch The test helper used for generating large files with repeated bytes has been updated to avoid slow 'tr' performance on macOS. A new 'test-tool genbytes' now generates repeated bytes natively, replacing the 'genzeros | tr' pipeline in several tests. Will merge to 'master'. cf. <xmqqo6bwqy9n.fsf@gitster.g> cf. <avRJPDEJWTGWySKp@pks.im> source: <pull.2301.v4.git.1794412751763.gitgitgadget@gmail.com> |
While looking for slow tests in CI, t4205 stood out on macOS: its
EXPENSIVE tests take several times longer there than on Linux. A CI
experiment showed that much of the time was spent in tr(1), which is
very slow on macOS for the 2GB of data that t4205 generates.
This adds "test-tool genbytes []" and uses it instead
of "genzeros | tr", which is where the speedup comes from.
Changes since v3:
use in t1600.
Changes since v2:
"genbytes" takes a literal character again ("genbytes a")
instead of two hex digits ("genbytes 61"), which Patrick found
hard to read at the call sites.
"genzeros" and its callers that want NUL bytes are kept as they
are, and the patch that converted them to "genbytes 00" is
dropped. This avoids churn in tests that do not need to change,
as discussed with Johannes. Keeping "genzeros" also means that
"genbytes" does not need special handling for NUL bytes.
test-genzeros.c is no longer renamed, and "genbytes" lives next
to "genzeros" in the same file. Since "genzeros" stays, the
current file name is still valid and not worth churning.
The buffer is static again instead of 256 KiB on the stack, as
suggested by Junio, to avoid using a large part of the limited
stack space. There is no risk of concurrent use in a run-once
helper.
The count is parsed with git_parse_signed(), so that an invalid
count is a usage error instead of being silently treated as 0,
as suggested by Patrick. This applies to "genzeros" as well.
Changes in v2 (subsequently reverted in v3):
Squashed the rename of test-genzeros.c into the patch that changes
it, as suggested by Junio.
"genbytes" took the byte as two lowercase hexadecimal digits
instead of a literal character, as suggested by Junio. This also
worked for NUL bytes, so "genbytes 00" replaced special-casing
"NUL".
cc: Patrick Steinhardt ps@pks.im
cc: Johannes Sixt j6t@kdbg.org
cc: Kristofer Karlsson krka@spotify.com
cc: Collin Funk collin.funk1@gmail.com