Git 2.24 breaks existing repositories: filename in tree entry contains backslash
carlescufi opened this issue · 23 comments
What are users supposed to do here, rewrite the history of the repos so that the offending commit is removed?
The workaround is to disable core.protectNTFS:
git config --global core.protectNTFS false
Setup
- Which version of Git for Windows are you using? Is it 32-bit or 64-bit?
Git for Windows 2.24.1.windows.2 64-bit
$ git --version --build-options
git version 2.24.1.windows.2
cpu: x86_64
built from commit: 992f0773022527b1b0cb1e0c13aec97dd5248053
sizeof-long: 4
sizeof-size_t: 8
- Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?
$ cmd.exe /c ver
Microsoft Windows [Version 10.0.18363.535]
- What options did you set as part of the installation? Or did you choose the
defaults?
# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt
Editor Option: VIM
Custom Editor Path:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled
Enable Builtin Interactive Add: Disabled
- Any other interesting things about your environment that might be related
to the issue you're seeing?
Installed with Chocolatey
Details
- Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other
cmd.exe
- What commands did you run to trigger this issue? If you can provide a
Minimal, Complete, and Verifiable example
this will help us understand the issue.
git clone https://github.com/zephyrproject-rtos/civetweb.git
- What did you expect to occur after running these commands?
Cloned repo successfully.
- What actually happened instead?
Cloning into 'civetweb'...
remote: Enumerating objects: 38, done.
remote: Counting objects: 100% (38/38), done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 21161 (delta 17), reused 19 (delta 7), pack-reused 21123
Receiving objects: 100% (21161/21161), 24.53 MiB | 7.55 MiB/s, done.
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
error: filename in tree entry contains backslash: '\'
Resolving deltas: 100% (13566/13566), done.
Updating files: 100% (1251/1251), done.
- If the problem was occurring with a specific repository, can you provide the
URL to that repository to help us with testing?
@carlescufi did you find out which tree entries those might be? They really cannot be checked out on Windows as expected: if there is a backslash encoded in any file name in that repository, it will be mistaken for a directory separator on Windows, which is obviously not what we want here.
@carlescufi did you find out which tree entries those might be?
One commit with a bad file is zephyrproject-rtos/civetweb@9d3af98 which adds a test/\/a.txt. However, this file was later removed in zephyrproject-rtos/civetweb@83dc425, and no file in the latest master branch contains a backslash in its name.
They really cannot be checked out on Windows as expected:
No attempt is made to check out a file with a backslash when cloning the latest master.
And indeed, though git clone https://github.com/zephyrproject-rtos/civetweb prints errors, it exits with a 0 return code.
However, this git fetch fails with return code 1:
mkdir civetweb
cd civetweb
git init
git fetch https://github.com/zephyrproject-rtos/civetweb.git
That seems inconsistent and arguably wrong: since we can clone the repository without an error return code (just error messages), we should be able to fetch it, too, just not check out the bad revisions.
@dscho as @mbolivar pointed out there is no longer any file in the repository with a backslash in its name. This means that, short of changing the behavior of Git for Windows in an upcoming release, the only other alternative is to rewrite the history of the tree to completely remove any traces of commits that introduced filenames with backslashes. This might be very cumbersome on some repositories.
@carlescufi & @mbolivar I am sympathetic to your use case. However, I am still worried about lifting the protections that we carefully put in place. So I'd like to tread very, very carefully here.
To start us in a direction that hopefully will lead to an acceptable compromise that still keeps the Git for Windows users safe, but also allows users to clone civetweb as long as they do not try to check out commits with the inadmissible directory name, I opened #2437.
@dscho thank you for the understanding and the PR. Please let us know when we can fetch a binary from CI so we can test it ourselves and give a +1.
Please let us know when we can fetch a binary from CI so we can test it ourselves and give a +1.
Whoa. I would like to ask even more explicitly for help with thinking about the problem. Maybe you can give a hand with reviewing the code and trying to get it into a shape where we can be reasonably sure that it does not regress on the vulnerabilities? Before that, I won't even bother with thinking about binaries.
Please let us know when we can fetch a binary from CI so we can test it ourselves and give a +1.
Whoa. I would like to ask even more explicitly for help with thinking about the problem. Maybe you can give a hand with reviewing the code and trying to get it into a shape where we can be reasonably sure that it does not regress on the vulnerabilities? Before that, I won't even bother with thinking about binaries.
Sorry, since you had posted a Pull Request I assumed that you had considered those already. I will take a look at the current PR with @mbolivar and give feedback here.
I have this issue with Jenkins as well, but I have no idea how to implement the workaround, as I don't see a setting for the plugin or a way to use the GitSCM class to work around this. 'bat' script block with the workaround also do not work.
https://issues.jenkins-ci.org/browse/JENKINS-60632
Edit: Tracking the relevant PRs for resolution in addition to the above.
If we are ok with wiping the commit with the offending blackslash-named files from history, how can this be done if we want to fix this now? I've tried git filter-branch to no avail and bfg
$ git log --all --name-only --pretty=format:%H | grep -B 2 '\\'
hash1
"mypkg/\\"
--
hash2
"mypkg/\\"
# Various escape method to match
git filter-branch --index-filter 'git rm --cached --ignore-unmatch "\"mypkg/\\.*"'
Could y'all please test https://github.com/git-for-windows/git/releases/tag/v2.25.0-rc1.windows.1?
@mtdeguzis If you could test, too? And if successful, recommend to upgrade to v2.25.0 for the affected parties (Git for Windows v2.25.0 is expected to land on January 13th, i.e. 10 days from now)?
Could y'all please test https://github.com/git-for-windows/git/releases/tag/v2.25.0-rc1.windows.1?
This solves the problem on my end. The fetch completes without warning or error. Thanks!
@mbolivar could you also, just for shiggles, try to check out the faulty revision? (It should fail)
I'm not sure how I can test since mine is affected by Jenkins git plugin. I will test as much as possible though.
@mtdeguzis I thought you would have the option of bundling Git for Windows with the plugin (or if not, be able to install the -rc version on the machine running Jenkins)?
Could y'all please test https://github.com/git-for-windows/git/releases/tag/v2.25.0-rc1.windows.1?
2.25.0.rc1 works for me. Had error fatal: filename in tree entry contains backslash: with git 2.24.1.windows.2 and no error with git 2.25.0.rc1.windows.1
Thanks a lot! :)
@dscho I can confirm this fixes the issue using Git Bash under v2.25.0-rc1.windows.1 and the Jenkins git plugin. Those using Jenkins can also refer to https://issues.jenkins-ci.org/browse/JENKINS-60632.
Excellent. Thanks, @mtdeguzis!
@mbolivar could you also, just for shiggles, try to check out the faulty revision? (It should fail)
Confirmed; it fails.
>git --version
git version 2.25.0.rc1.windows.1
>git checkout 9d3af98
error: filename in tree entry contains backslash: 'test/\/a.txt'
HEAD remained the same after the checkout.
Fixed in v2.25.0-rc1 via git@224c7d7.
@dscho thank you for all the work on this issue.
When I am trying to clone this repo: https://github.com/vinhphunguyen/MPM-Julia
I can clone it, but cannot checkout.
$ git clone https://github.com/inkydragon/MPM-Julia.git
Cloning into 'MPM-Julia'...
remote: Enumerating objects: 237, done.
remote: Counting objects: 100% (237/237), done.
remote: Compressing objects: 100% (211/211), done.
remote: Total 237 (delta 26), reused 233 (delta 25), pack-reused 0
Receiving objects: 100% (237/237), 26.44 MiB | 35.00 KiB/s, done.
Resolving deltas: 100% (26/26), done.
error: invalid path 'CPDI_3D_T4_Cantilever/.\Figs\Cantelever_0.png'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'
$ git --version
git version 2.26.0.windows.1
$ git checkout master
error: invalid path 'CPDI_3D_T4_Cantilever/.\Figs\Cantelever_0.png'
error: invalid path 'Classic_2D_2Disk/..\..\Figs\plot_2Disk_Julia.pdf'
error: invalid path 'Classic_2D_SteelAluminum/.\Figs\DiskImpact_1.png'
error: invalid path 'Classic_2D_SteelAluminum/.\Figs\DiskImpact_1001.png'
...
Also, $ git config core.protectNTFS false doesnot work.
This repo seems created in macOS, and has strange filename such as: .\Figs\DiskImpact_1.png.
Yes it's a filename not a path.
see also #2438