Skip to content

Use symbolic links for libexec #1997

Closed
Closed

Description

@orgads
  • I was not able to find an open or closed issue matching what I'm seeing

Is it possible to deploy Git with symbolic links in libexec/git-core instead of many hard copies of the git executable?

Activity

  1. dscho commented on Dec 23, 2018

    @dscho
    Member

    Only if you build Git yourself.

    However:

    instead of many hard copies of the git executable?

    This is incorrect. Those are not copies, those are hard links. It is true that Windows Explorer double-counts them, but they do not occupy more disk space than one copy + 127 directory entries pointing to the same start disk sector. You can verify this by running the following command:

    $ ls -l /mingw64/bin/git.exe
    -rwxr-xr-x 127 johannes 197121 2755352 Dec 15 09:52 /mingw64/bin/git.exe*

    See that 127? That's the number of directory entries pointing at the same file contents.

  2. PhilipOakley commented on Dec 23, 2018

    @PhilipOakley

    @organds, Any suggestions as to where the best place(s) would be to highlight the information?

    It is a question that does come up moderately frequently, so as some one who has been hit by the issue, maybe have a think about where you looked (or even if didn't 'look', why that was) so that we can educate folks and reduce the misunderstandings. Every little helps.

    [Would it be worth adding a line to the 'Known Issues' of the Release notes file:///C:/Program%20Files/Git/ReleaseNotes.html, just the head the issue off at the pass]

  3. orgads commented on Dec 24, 2018

    @orgads
    Author

    See that 127? That's the number of directory entries pointing at the same file contents.

    I see. So you have one extra executable: libexec/git-core/git.exe:

    OrgadS@isr020134037 MINGW64 /c/Program Files/Git/mingw64/libexec/git-core
    $ ls -li ../../bin/git.exe git-add.exe git.exe
     34902897112204028 -rwxr-xr-x 127 OrgadS 1049089 2755352 Dec 24 12:14 ../../bin/git.exe*
    194499208907169829 -rwxr-xr-x   1 OrgadS 1049089 2755352 Dec 24 12:14 git.exe*
     34902897112204028 -rwxr-xr-x 127 OrgadS 1049089 2755352 Dec 24 12:14 git-add.exe*
    

    @PhilipOakley : Maybe add an empty file inside libexec/git-core named ___MOST-EXECUTABLES-ARE-HARDLINKS___. That would have attracted my attention :)

  4. dscho commented on Dec 26, 2018

    @dscho
    Member

    @orgads yeah, no. I don't think that would be a good idea. First of all, this file would have to be generated only if the files really are hardlinks, and then that file would have to be left out e.g. when copying those files. That's just too fragile.

    At some point, we have to bite the bullet and just accept that most users do not really understand their system and need the explanation. I just hope that the number of contributors who relieve the maintainer of explaining this (in favor of chasing much harder bugs, or in favor of trying to hammer down work hours to a normal level) will continue to increase.

  5. PhilipOakley commented on Jan 3, 2019

    @PhilipOakley

    @dscho wrote: just accept that most users do not really understand their system and need the explanation (that the "use links in libexec/git-core instead of many hard copies of the git executable" is already done via Windows hardlinks).

    Would I be reading this right, that my suggestion of a short(ish) extra line in the 'known issues' section could be that explanation? It may only find 40% of the users but at least its a reduction in complaints... (Should I add it to my list of things to do on retiral;-)

  6. dscho commented on Jan 7, 2019

    @dscho
    Member

    @PhilipOakley sounds good ;-)

  7. dscho commented on Feb 27, 2019

    @dscho
    Member
  8. orgads commented on Mar 1, 2019

    @orgads
    Author

    What about this issue?

    I see. So you have one extra executable: libexec/git-core/git.exe:

    OrgadS@isr020134037 MINGW64 /c/Program Files/Git/mingw64/libexec/git-core
    $ ls -li ../../bin/git.exe git-add.exe git.exe
     34902897112204028 -rwxr-xr-x 127 OrgadS 1049089 2755352 Dec 24 12:14 ../../bin/git.exe*
    194499208907169829 -rwxr-xr-x   1 OrgadS 1049089 2755352 Dec 24 12:14 git.exe*
     34902897112204028 -rwxr-xr-x 127 OrgadS 1049089 2755352 Dec 24 12:14 git-add.exe*
    
  9. PhilipOakley commented on Mar 1, 2019

    @PhilipOakley

    I'm updating the PR (#229 above) that adds this to the Known Issues list (assuming you are commenting about the "127" factor, rather than something I haven't understood)

  10. orgads commented on Mar 1, 2019

    @orgads
    Author

    I don't refer to the factor. I refer to the additional executable libexec/git-core/git.exe, which is not a hard-link.

  11. dscho commented on Mar 1, 2019

    @dscho
    Member

    I refer to the additional executable libexec/git-core/git.exe, which is not a hard-link.

    That's a bug, and if you can figure out a fix, I'll be happy to accept a PR.

  12. dscho commented on Mar 1, 2019

    @dscho
    Member

    (To clarify: this extra git.exe should also be a hardlink.)

  13. orgads commented on Mar 1, 2019

    @orgads
    Author

    First observation: make install creates all as hard links, so it's probably something in the installer...

  14. dscho commented on Mar 1, 2019

    @dscho
    Member

    It seems that the mingw-w64-git package has the hardlinks correct:

    $ tar tvf mingw-w64-x86_64-git-2.21.0.1.2481c4cbe9-1-any.pkg.tar.xz
    |grep 'git\.exe'
    -rwxr-xr-x ContainerAdministrator/380929  40728 2019-02-26 20:48 cmd/git.exe
    hrwxr-xr-x ContainerAdministrator/380929       0 2019-02-26 20:48 mingw64/bin/git.exe link to mingw64/bin/git-receive-pack.exe
    hrwxr-xr-x ContainerAdministrator/380929       0 2019-02-26 20:48 mingw64/libexec/git-core/git.exe link to mingw64/bin/git-receive-pack.exe

    Meaning: something must be going wrong when the installer does its magic. It does call HardlinkOrCopyGit(), or at least is supposed to, for all builtins here.

  15. dscho commented on Mar 1, 2019

    @dscho
    Member

    So the builtins are determined from the file /mingw64/share/git/builtins.txt, read here. And that file does not contain /mingw64/libexec/git-core/git.exe here.

  16. dscho commented on Mar 1, 2019

    @dscho
    Member

    That builtins.txt file is generated during installer/release.sh <version> time, and I think it probably determines which files to include by looking which files were replaced by the git-wrapper.exe in git-extra's post-install script.

    You see, we got so many concerns about that "massive" filesystem usage that we started to ship at least our portable Git with the builtins replaced not by hardlinked versions of git.exe, but by the Git wrapper (which basically just re-spawns the real git.exe with the builtin's command name inserted as first parameter). And that is why git-extra's post-install script edits the files installed by mingw-w64-git.

  17. dscho commented on Mar 1, 2019

    @dscho
    Member

    Oh, and I was wrong about installer/release.sh generating that builtins file. It is https://github.com/git-for-windows/MINGW-packages/blob/01d3f0e810b7e5a37488d97c860ba3df20685412/mingw-w64-git/PKGBUILD#L154

    Now, the reason why it does not include /mingw64/libexec/git-core/git.exe is obvious: that is not a builtin, if we replaced that by the Git wrapper, it would either cause an infinite loop, or it would not find the real git.exe to execute.

    I guess the best idea would be to teach git-wrapper.c to detect when it was called as .../libexec/git-core/git.exe and spawn .../bin/git.exe in that case, much like /cmd/git.exe does.

    Now, https://github.com/git-for-windows/MINGW-packages/blob/01d3f0e810b7e5a37488d97c860ba3df20685412/mingw-w64-git/git-wrapper.c#L696 tries to be very careful to strip only one directory level (the cmd part of C:\Program Files\Git\cmd\git.exe) in order to look whether we are in an MSYS2-based setup (essentially by looking whether <top-level-directory>\mingw64\bin exists).

    To extend this to /mingw64/libexec/git-core/git.exe, we would have to use 3 instead.

    And then we could teach the print-builtins target in the mingw-w64-git.mak file to also print that path.

    Of course, this all would need to be tested by

    1. patching MINGW-packages,
    2. building and installing that mingw-w64-x86_64-git package
    3. re-installing git-extra (to replace the builtins with the Git wrapper)
    4. verifying that /mingw64/libexec/git-core/git.exe works as intended.

    Seeing as I spent the last 38 minutes investigating it this far, I now think I should be doing more pressing things, and leave the rest to interested parties (I am looking at you, @orgads 😄 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions