Debug Windows build crashes inside __acrt_FlsGetValue2 due to some MSVCRT ABI mismatch
TrevorCash opened this issue · 25 comments
Hi mosra,
I am building magnum in a cmake project with fetchcontent_declare and fetchcontent_makeavailable.
I am getting the following error when building with visual studio 2022:
Severity Code Description Project File Line Suppression State Details
Error LNK1104 cannot open file 'C:\Users\casht\repos\netgame2\build_Debug_deps\magnum-build\src\Magnum\Shaders\MagnumShadersObjects.dir\Debug\resource_MagnumShaders_RESOURCES_GL.obj' MagnumShaders C:\Users\casht\repos\netgame2\build_Debug_deps\magnum-build\src\Magnum\Shaders\LINK 1
Even after building the MagnumShaders_RESOURCES_GL-dependencies project. building the MagnumShaders project gives the error above.
Microsoft Visual Studio Community 2022 (64-bit) - LTSC 17.10
Version 17.10.12
This might be an enviroment problem - but at the moment I am not sure.
Any help appreciated
-Trevor
Hi, congrats on issue number 666! 😄
This is most likely related to the corrade-rc utility, which is built as part of corrade, not working properly for some reason. Ideally you'd get a build error right when it attempts to execute the utility and the utility doesn't work, but that's somehow not happening here.
Some things to try:
- I assume you have corrade added through FetchContent as well, right? Does the build directory contain
corrade-rc.exesomewhere? If you run it through the console (say,corrade-rc --help), does it print something to the output, or does it crash violently? - The
resource_MagnumShaders_RESOURCES_GL.objfile should be generated fromresource_MagnumShaders_RESOURCES_GL.cppplaced somewhere inside the build directory as well. Does that file exist, and does it have meaningful content (in particular is not empty, looks like c++, and isn't truncated)? - If both of the above are fine, could there be some AV software throwing a wrench into the build process, where it temporarily quarantines a file or whatever while scanning it, resulting it in being locked for reading or not there at all?
The *-dependencies projects are just collecting files the resource is depending on -- e.g., if one of the shader files would change, it triggers a regeneration & rebuild of resource_MagnumShaders_RESOURCES_GL.cpp, so building that target alone doesn't do anything.
Hi mosra,
Its a sign lol
corrade-rc.exe exists. running corrade-rc.exe --help does not print anything. It just exits.
The resource_MagnumShaders_RESOURCES_GL.obj does not seem to be present (even after rebuilding the MagnumShaders_RESOURCES_GL-dependencies project in visual studio:
#Bring in corrade
fetchcontent_declare(
corrade
GIT_REPOSITORY
https://${AUTH_USER}:${AUTH_TOCKEN}@github.com/TrevorCash/corrade
GIT_TAG commitHash_efd817760bf69aebfa35ffe8e857c80e4671660e
GIT_PROGRESS
TRUE
GIT_SHALLOW
FALSE
FETCHCONTENT_FULLY_DISCONNECTED ${SKIP_FETCHES}
)
fetchcontent_makeavailable(corrade)
set(CMAKE_MODULE_PATH "${CMAKE_MODULE_PATH};${corrade_SOURCE_DIR}/modules")
#Bring in magnum
fetchcontent_declare(
magnum
GIT_REPOSITORY
https://${AUTH_USER}:${AUTH_TOCKEN}@github.com/TrevorCash/magnum
GIT_TAG commitHash_e478fab4f80def28b419da4305f93cb6f6fc327b
GIT_PROGRESS
TRUE
GIT_SHALLOW
FALSE
FETCHCONTENT_FULLY_DISCONNECTED ${SKIP_FETCHES}
)
I have some small patches to magnum. but nothing to do with resources or shaders.
corrade-rc.exe exists. running corrade-rc.exe --help does not print anything. It just exits.
That sounds like the root cause here. If this utility doesn't work and for some reason exits without even giving a failure return code, the buildsystem thinks the operation succeeded, and so it fails later because the resource_MagnumShaders_RESOURCES_GL.cpp file from which it should have made an resource_MagnumShaders_RESOURCES_GL.obj doesn't exist, so the *.obj doesn't exist either.
So -- just to be clear and avoid any misunderstanding -- if you open cmd.exe in the directory where corrade-rc.exe is and run
corrade-rc.exe --helpit doesn't print anything at all, and just shows the prompt again? What's the application return code, i.e., what does this print?
corrade-rc.exe --help
echo %errorlevel%Can you attach the exe file here?
Also - here is an attachement of the build tree the project - https://drive.google.com/open?id=12f-RUUq8k8AfnyT2wMSt9F0ceKYCKjXC&usp=drive_fs
Thanks for the return code -- that's 0xc0000141, "application was unable to start correctly". Unfortunately I have no idea what that could mean... It's deliberately built in a way to not rely on any DLL, except for the MSVC runtime, so I don't think it should be due to some DLL missing
Since it's built as part of the project, are you able to run it through a debugger?
That sounds like a dll issue. And most likely the MSVC runtime given the comment by @mosra above.
Can you check/verify if you have the msvc runtime installed?
https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170
@mounte visual studio runtime is installed.
debugging corrade-rc gives
Exception thrown at 0x00007FFE6F497E92 (KernelBase.dll) in corrade-rc.exe: 0xC0000005: Access violation writing location 0x00007FFD72142F98.
KernelBase.dll!00007ffe6f497e92() Unknown
ucrtbased.dll!00007ffd72016454() Unknown
ucrtbased.dll!00007ffd72013e33() Unknown
ucrtbased.dll!00007ffd72013db3() Unknown
ucrtbased.dll!00007ffd72014450() Unknown
ucrtbased.dll!00007ffd720146cd() Unknown
ucrtbased.dll!00007ffd72014753() Unknown
ucrtbased.dll!00007ffd72014e6b() Unknown
ucrtbased.dll!00007ffd72013695() Unknown
ucrtbased.dll!00007ffd71fe5579() Unknown
ucrtbased.dll!00007ffd71fe54c5() Unknown
ucrtbased.dll!00007ffd71fe5631() Unknown
ntdll.dll!00007ffe71eee8cf() Unknown
ntdll.dll!00007ffe71eec63c() Unknown
ntdll.dll!00007ffe71f05812() Unknown
ntdll.dll!00007ffe71f057fa() Unknown
ntdll.dll!00007ffe71f057fa() Unknown
ntdll.dll!00007ffe71f057fa() Unknown
ntdll.dll!00007ffe71f8d47b() Unknown
ntdll.dll!00007ffe71f5efec() Unknown
ntdll.dll!00007ffe71f5edfe() Unknown
ntdll.dll!00007ffe71f6187e() Unknown
I switched windows sdk version to 10.0.18362.0 which I thought might help but same result.
one thing i notice is resource_MagnumShaders_RESOURCES_GL.cpp.rule is empty - is this normal? The file just has the contents:
# generated from CMake
And that is the entire contents.
6>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for 'C:\Users\casht\repos\netgame2\build_Debug\CMakeFiles\2c268e9d73bd732ca5669d29df780565\resource_MagnumShaders_RESOURCES_GL.cpp.rule' exited with code -1073741502.
You can probably get a cleaner stack trace by enabling symbol loading from Microsoft's symbol servers - in Visual Studio, open the Tools menu from the menu bar, select Options, expand the Debugging section in the tree view on the left, select Symbols, and enable Microsoft Symbol Servers. Also select "Search for all module symbols unless excluded" and if you're in the debugger at the moment you can click the "Load All Symbols" button at the bottom. Afterward, all those "Unknown" in the stack trace should be replaced by actual function names.
It looks like it's trying to execute something weird and crashes due to that... The debugger stack trace doesn't give any info from where in Corrade's code this could happen, just the system DLLs, right?
Can you try enabling Corrade's own test and run some of those, if they exhibit the same behavior? In your CMake file:
fetchcontent_declare(
corrade
...
)
set(CORRADE_BUILD_TESTS ON CACHE BOOL "") # add this
fetchcontent_makeavailable(corrade)Then, build for example the TargetTest (which is the smallest and simplest one) and run it in a terminal. Does it also exit printing nothing?
Another random thing to try is to disable CORRADE_BUILD_CPU_RUNTIME_DISPATCH, which enables various code paths depending on what instruction sets the CPU advertises (SSE, AVX, etc.), and in some very unfortunate scenario where the CPU misadvertises its capabilities it could lead to invalid instructions being executed. So the following in your CMake file:
set(CORRADE_BUILD_CPU_RUNTIME_DISPATCH OFF CACHE BOOL "")
fetchcontent_makeavailable(corrade)Another idea -- if you clone & build Corrade directly (so, not via FetchContent from another project), does corrade-rc.exe --help` work properly, or does it also exit printing nothing?
It looks like it's trying to execute something weird and crashes due to that... The debugger stack trace doesn't give any info from where in Corrade's code this could happen, just the system DLLs, right?
Right
Here is the stacktrace with debug symbols on for corrade-rc.exe - thanks @LB-- :
KernelBase.dll!FlsGetValue2() Unknown
ucrtbased.dll!__acrt_FlsGetValue2(unsigned long fls_index) Line 553 C++
ucrtbased.dll!try_get_ptd_head<void * (__cdecl*)(unsigned long)>(void *(*)(unsigned long) flsgetvalue) Line 217 C++
ucrtbased.dll!internal_get_ptd_head<void * (__cdecl*)(unsigned long)>(void *(*)(unsigned long) flsgetvalue) Line 269 C++
ucrtbased.dll!internal_getptd_noexit2() Line 309 C++
ucrtbased.dll!__acrt_getptd_noexit() Line 336 C++
ucrtbased.dll!__acrt_initialize_ptd() Line 41 C++
> ucrtbased.dll!__acrt_execute_initializers(const __acrt_initializer * first, const __acrt_initializer * last) Line 25 C++
ucrtbased.dll!__acrt_initialize() Line 291 C++
ucrtbased.dll!DllMainProcessAttach() Line 24 C++
ucrtbased.dll!DllMainDispatch(HINSTANCE__ * __formal, const unsigned long fdwReason, void * const lpReserved) Line 63 C++
ucrtbased.dll!__acrt_DllMain(HINSTANCE__ * const hInstance, const unsigned long fdwReason, void * const lpReserved) Line 102 C++
ntdll.dll!LdrpCallInitRoutine() Unknown
ntdll.dll!LdrpInitializeNode() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeProcess() Unknown
ntdll.dll!_LdrpInitialize() Unknown
ntdll.dll!LdrpInitializeInternal() Unknown
ntdll.dll!LdrInitializeThunk() Unknown
Testing TargetTest also exits - similar stack trace:
KernelBase.dll!FlsGetValue2() Unknown
ucrtbased.dll!__acrt_FlsGetValue2(unsigned long fls_index) Line 553 C++
ucrtbased.dll!try_get_ptd_head<void * (__cdecl*)(unsigned long)>(void *(*)(unsigned long) flsgetvalue) Line 217 C++
ucrtbased.dll!internal_get_ptd_head<void * (__cdecl*)(unsigned long)>(void *(*)(unsigned long) flsgetvalue) Line 269 C++
ucrtbased.dll!internal_getptd_noexit2() Line 309 C++
ucrtbased.dll!__acrt_getptd_noexit() Line 336 C++
ucrtbased.dll!__acrt_initialize_ptd() Line 41 C++
> ucrtbased.dll!__acrt_execute_initializers(const __acrt_initializer * first, const __acrt_initializer * last) Line 25 C++
ucrtbased.dll!__acrt_initialize() Line 291 C++
ucrtbased.dll!DllMainProcessAttach() Line 24 C++
ucrtbased.dll!DllMainDispatch(HINSTANCE__ * __formal, const unsigned long fdwReason, void * const lpReserved) Line 63 C++
ucrtbased.dll!__acrt_DllMain(HINSTANCE__ * const hInstance, const unsigned long fdwReason, void * const lpReserved) Line 102 C++
ntdll.dll!LdrpCallInitRoutine() Unknown
ntdll.dll!LdrpInitializeNode() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeProcess() Unknown
ntdll.dll!_LdrpInitialize() Unknown
ntdll.dll!LdrpInitializeInternal() Unknown
ntdll.dll!LdrInitializeThunk() Unknown
(I had to copy some of the dlls around manually in the build tree) :
I also ran cmake on a fresh clone of corrade (commit: efd817760bf69aebfa35ffe8e857c80e4671660e)
running corrade-rc from in visual studio resulted in the same stacktrace as above. (FlsGetValue2()) ( I tried with and without CORRADE_BUILD_CPU_RUNTIME_DISPATCH ) seemingly same stacktrace
with pointer dispatch:
KernelBase.dll!FlsGetValue2() Unknown
ucrtbased.dll!__acrt_FlsGetValue2(unsigned long fls_index) Line 553 C++
ucrtbased.dll!try_get_ptd_head<void * (__cdecl*)(unsigned long)>(void *(*)(unsigned long) flsgetvalue) Line 217 C++
ucrtbased.dll!internal_get_ptd_head<void * (__cdecl*)(unsigned long)>(void *(*)(unsigned long) flsgetvalue) Line 269 C++
ucrtbased.dll!internal_getptd_noexit2() Line 309 C++
ucrtbased.dll!__acrt_getptd_noexit() Line 336 C++
ucrtbased.dll!__acrt_initialize_ptd() Line 41 C++
> ucrtbased.dll!__acrt_execute_initializers(const __acrt_initializer * first, const __acrt_initializer * last) Line 25 C++
ucrtbased.dll!__acrt_initialize() Line 291 C++
ucrtbased.dll!DllMainProcessAttach() Line 24 C++
ucrtbased.dll!DllMainDispatch(HINSTANCE__ * __formal, const unsigned long fdwReason, void * const lpReserved) Line 63 C++
ucrtbased.dll!__acrt_DllMain(HINSTANCE__ * const hInstance, const unsigned long fdwReason, void * const lpReserved) Line 102 C++
ntdll.dll!LdrpCallInitRoutine() Unknown
ntdll.dll!LdrpInitializeNode() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeProcess() Unknown
ntdll.dll!_LdrpInitialize() Unknown
ntdll.dll!LdrpInitializeInternal() Unknown
ntdll.dll!LdrInitializeThunk() Unknown
without pointer dispatch:
KernelBase.dll!FlsGetValue2() Unknown
ucrtbased.dll!__acrt_FlsGetValue2(unsigned long fls_index) Line 553 C++
ucrtbased.dll!try_get_ptd_head<void * (__cdecl*)(unsigned long)>(void *(*)(unsigned long) flsgetvalue) Line 217 C++
ucrtbased.dll!internal_get_ptd_head<void * (__cdecl*)(unsigned long)>(void *(*)(unsigned long) flsgetvalue) Line 269 C++
ucrtbased.dll!internal_getptd_noexit2() Line 309 C++
ucrtbased.dll!__acrt_getptd_noexit() Line 336 C++
ucrtbased.dll!__acrt_initialize_ptd() Line 41 C++
> ucrtbased.dll!__acrt_execute_initializers(const __acrt_initializer * first, const __acrt_initializer * last) Line 25 C++
ucrtbased.dll!__acrt_initialize() Line 291 C++
ucrtbased.dll!DllMainProcessAttach() Line 24 C++
ucrtbased.dll!DllMainDispatch(HINSTANCE__ * __formal, const unsigned long fdwReason, void * const lpReserved) Line 63 C++
ucrtbased.dll!__acrt_DllMain(HINSTANCE__ * const hInstance, const unsigned long fdwReason, void * const lpReserved) Line 102 C++
ntdll.dll!LdrpCallInitRoutine() Unknown
ntdll.dll!LdrpInitializeNode() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeProcess() Unknown
ntdll.dll!_LdrpInitialize() Unknown
ntdll.dll!LdrpInitializeInternal() Unknown
ntdll.dll!LdrInitializeThunk() Unknown
Here is the build tree of just building corrade: https://drive.google.com/open?id=12hyrPDGdx4jCKxN5ztwJ1V6ohlfK7UQe&usp=drive_fs
This could be because I am compiling on a preview addition of windows (even though I am on a LTS version of visual studio 2022)
Edition Windows 11 Pro Insider Preview
Version 23H2
Installed on 1/28/2024
OS build 26040.1000
Experience Windows Feature Experience Pack 1000.26040.1000.0
Thanks, the stack traces are a lot more useful now, even though the problem seems to be increasingly cursed :D FlsGetValue2() seems to be related to "fiber-local storage", so I wonder if that maybe has any connection to thread-local storage that I'm using in a few places.
Can you try building with CORRADE_BUILD_MULTITHREADED disabled in CMake? That should disable use of all thread-local storage. With the downside of debug output, plugin manager etc. not being usable in a multi-threaded environment, but that's now less important than things not working at all.
with multithreaded off same stacktrace for corrade-rc.exe (corrade only cmake build). Are there other windows, visual studio 2022 user/machines testing with corrade currently?
KernelBase.dll!FlsGetValue2() Unknown
ucrtbased.dll!__acrt_FlsGetValue2(unsigned long fls_index) Line 553 C++
ucrtbased.dll!try_get_ptd_head<void * (__cdecl*)(unsigned long)>(void *(*)(unsigned long) flsgetvalue) Line 217 C++
ucrtbased.dll!internal_get_ptd_head<void * (__cdecl*)(unsigned long)>(void *(*)(unsigned long) flsgetvalue) Line 269 C++
ucrtbased.dll!internal_getptd_noexit2() Line 309 C++
ucrtbased.dll!__acrt_getptd_noexit() Line 336 C++
ucrtbased.dll!__acrt_initialize_ptd() Line 41 C++
ucrtbased.dll!__acrt_execute_initializers(const __acrt_initializer * first, const __acrt_initializer * last) Line 25 C++
ucrtbased.dll!__acrt_initialize() Line 291 C++
ucrtbased.dll!DllMainProcessAttach() Line 24 C++
ucrtbased.dll!DllMainDispatch(HINSTANCE__ * __formal, const unsigned long fdwReason, void * const lpReserved) Line 63 C++
ucrtbased.dll!__acrt_DllMain(HINSTANCE__ * const hInstance, const unsigned long fdwReason, void * const lpReserved) Line 102 C++
ntdll.dll!LdrpCallInitRoutine() Unknown
ntdll.dll!LdrpInitializeNode() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeGraphRecurse() Unknown
ntdll.dll!LdrpInitializeProcess() Unknown
ntdll.dll!_LdrpInitialize() Unknown
ntdll.dll!LdrpInitializeInternal() Unknown
ntdll.dll!LdrInitializeThunk() Unknown
I just did a build of Corrade on Windows 10 x86-64 (19045.5737) with VS 2022 (17.13.6) and tests enabled and ran the tests, everything passed except UtilityPathTest and UtilityDirectoryTest. Then I tried the same on Windows 11 ARM64 (26120.3585) with VS 2022 ARM64 (17.14.0 Preview 2.0), but there are a lot of build errors. I don't have any x86-64 Windows 11 systems to test on.
EDIT: I fixed/worked around the errors I was getting and was able to run tests on Windows 11 ARM64, the only tests that failed were InterconnectTest, UtilityPathTest, UtlityDirectoryTest, and it says CpuTest was not run. corrade-rc.exe works fine on both systems for me.
Are there other windows, visual studio 2022 user/machines testing with corrade currently?
Yes, several, plus it's tested on various MSVC versions in multiple configurations on each push.
I'm out of ideas... :( If you build and run a minimal Hello World executable, does that work? If it does, does it work if you link to Corrade with for example just Utility::Debug{} << "hello world";? Do you have any other machine you could try to run the crashy corrade-rc.exe on?
Can anybody else who's on Windows 11 with MSVC 2022 installed try to download and run the corrade-rc.exe from the zip in the comment above? Does it work or does it crash the same way? @mounte when you checked the file, did you try to run it as well?
Thanks everyone for the help.
Can anybody else who's on Windows 11 with MSVC 2022 installed try to download and run the corrade-rc.exe from #666 (comment)?
I ran it in Windows Sandbox on my Windows 11 ARM64 machine, had to install the x86-64 VCRedist at first and then it exhibited the same behavior as described above - no output, sets errorlevel to -1073741515.
Thanks, so that means the toolchain used to build this is broken somehow, not that it would be something wrong with the system DLLs or the way the executables are run on that particular machine.
That's all I have, no clue what else to do to resolve this.
I ran the corrade-rc.exe from that zip on my windows 11 x64 (not ARM) without a problem.
ok - I suspect it is windows pre-release related.
Still very strange combination of symptoms though - my own build worked and the provided build failed on the same Windows 11 system, and the provided build worked fine for someone else on a different Windows 11 system. That smells like two separate recessive bugs that only express themselves when combined. An issue in the build system and an issue in the operating system, independently okay but together problematic?
Interesting. This sounds like some rather unfortunate ABI breakage, where the MSVC (debug?) runtime works correctly only on a specific OS version, and breaks everywhere else, and you were unlucky enough that your OS version didn't match the version the MSVC debug runtime was meant to be run on.
One last thing if you still want to dig further into this, if you build a release build, instead of a debug one (so it doesn't rely on ucrtbased.dll but rather ucrtbase.dll), does it crash too? I'd hope that one works because the DLLs are part of the system (or the redistributable), not of the concrete MSVC version, and could be less prone to such ABI differences.
I tried release mode and it built succesfully. so yeah - now if only I could actually get windows to update to the latest general public release...



