libcoreclr.dylib leads to libsystem_c.dylib!abort and crashing repeatedly
sfoslund opened this issue · 33 comments
Issue moved from dotnet/vscode-dotnet-runtime#252
- Please respond to @winzig.
From @winzig on Wednesday, June 30, 2021 4:53:47 PM
Initially reported to vscode an issue where chrome_crashpad_handler was constantly running with high CPU/disk usage here: microsoft/vscode#127392 (and someone else reported it here: microsoft/vscode#124713)
They looked at the attached dmp's and stated:
Thanks for the logs, all of them point to crash in the dotnet runtime specifically a call from libcoreclr.dylib leads to libsystem_c.dylib!abort case. Since I don't have symbols for this extension binary, I can't specifically say why.
They suggested I open an issue here and attach my dmp files, so here it is.
Issue moved from dotnet/vscode-dotnet-runtime#252
- Please respond to @winzig.
From @winzig on Wednesday, June 30, 2021 4:58:14 PM
Here are a few more .dmp's from this morning. Unlike the others I uploaded, these are from the completed folder, so perhaps they'll be more useful as I'm guessing they're uploaded to Microsoft's system.
Issue moved from dotnet/vscode-dotnet-runtime#252
- Please respond to @pascallapradebrite4.
From @pascallapradebrite4 on Wednesday, June 30, 2021 6:10:57 PM
I was also advised that my problem with VS Code's Crashpad (105,780 dmps in total, so more 10,000 dmps per hour during one workday last week, fan spinning the whole day, ended up taking 20.43 gb of disk space) was seemingly caused by the vscode-dotnet-runtime extension.
Here are my specs:
MacBook Pro (15-inch, 2019)
CPU: 2,3 GHz 8-Core Intel Core i9
Memory: 16 GB 2400 MHz DDR4
OS: Big Sur 11.4 (20F71)
VS Code version: Code 1.57.1
OS version: Darwin x64 20.5.0
And in case it can help, here are some of my dmps as well:
Issue moved from dotnet/vscode-dotnet-runtime#252
- Please respond to @sfoslund.
From @sfoslund on Wednesday, June 30, 2021 6:20:46 PM
Can you please provide the vscode-dotnet-runtime logs? You can get a file path by running the report issue command: https://github.com/dotnet/vscode-dotnet-runtime/blob/main/Documentation/troubleshooting-runtime.md#other-issues
Issue moved from dotnet/vscode-dotnet-runtime#252
- Please respond to @winzig.
From @winzig on Wednesday, June 30, 2021 8:17:13 PM
@sfoslund The file located in the specified folder (~/Library/Application Support/Code/logs/20210630T131453/exthost1/ms-dotnettools.vscode-dotnet-runtime/DotNetAcquisition-ms-dotnettools.vscode-dotnet-runtime-1625084122504.txt) is currently zero bytes in size.
Issue moved from dotnet/vscode-dotnet-runtime#252
- Please respond to @pascallapradebrite4.
From @pascallapradebrite4 on Wednesday, June 30, 2021 8:19:08 PM
Hi @sfoslund ! Thanks for your help!
Sadly, the logs are completely empty on my machine - all the while the Crashpad handler is running constantly, indicating the issue(s) is presumably happening right now. I ran the command on all my windows, and all .txt reports were empty.
Given that info wasn't available, I went through a more thorough investigation on my side, since it's quite a hassle in my workday to have my disk fill up like that. I had many .NET projects and non-.NET projects opened at the same time. I closed and opened each of them one by one until I could identify a single window is causing the problem:
- I found one window where closing it would immediately stop the Crashpad handler. Opening it would immediately start it again. My other windows didn't have that effect. (Note: I disabled all extensions and re-enable them on a workspace basis to help VS Code start faster, so it helped identifying which specific window was responsible)
- I ran VS Code's Extensions Bisect tool, and sure enough, the extension found was
ms-dotnettools.vscode-dotnet-runtime, which matches what the person that helped me in the original issue in VS Code said. - I then tried to disable the extension for that window, but couldn't, as another extension depends on it:
stripe.vscode-stripe. - I then disabled
stripe.vscode-stripebut letms-dotnettools.vscode-dotnet-runtimeenabled, and... the Crashpad handler wasn't running anymore.
Which is very interesting, since I never installed ms-dotnettools.vscode-dotnet-runtime myself, so it seems that Stripe's extension eventually added it as a dependency - likely around the time when I started noticing the problem. I went and looked in their repo, and it seems indeed they added it only 12 days ago: 0511ab4#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519
Now, the way I see it, the issue could either be in ms-dotnettools.vscode-dotnet-runtime and something called by stripe.vscode-stripe is crashing; or it's reversed, and it's stripe.vscode-stripe that is improperly using this extension, and causing it to crash.
If @winzig also has the stripe.vscode-stripe extension installed, and especially if the Crashpad handler stops running when deactivating the extension, then the issue is maybe on the side of the extension and we can open an issue in that repo instead. But if the OP doesn't have the Stripe extension, then it's more likely that the issue is in ms-dotnettools.vscode-dotnet-runtime.
At the very least, based on the issue in VS Code's repo, it seems that the stacktrace found in the dump files point to ms-dotnettools.vscode-dotnet-runtime specifically.
As for me, for now, I'll disable Stripe's extension, now that I know how to avoid the issue completely!
So, as I said, I ran the .NET Install Tool: Report an issue with the .NET Install Tool for Extension Authors on the window causing the problem (before disabling Stripe's extension).
Given that it is empty, GitHub doesn't let me upload it, though I doubt it would have been helpful in that state.
The name of the empty file is DotNetAcquisition-ms-dotnettools.vscode-dotnet-runtime-1625083059683.txt.
Issue moved from dotnet/vscode-dotnet-runtime#252
- Please respond to @winzig.
From @winzig on Wednesday, June 30, 2021 8:19:09 PM
I also checked for any other files named DotNetAcquisition-ms-dotnettools.vscode-dotnet-runtime-* and there are 11 of them, but they are also zero bytes.
Issue moved from dotnet/vscode-dotnet-runtime#252
- Please respond to @winzig.
From @winzig on Wednesday, June 30, 2021 8:19:58 PM
Interesting, I also have the stripe extension installed...
Issue moved from dotnet/vscode-dotnet-runtime#252
- Please respond to @winzig.
From @winzig on Wednesday, June 30, 2021 8:23:07 PM
cc @gracegoo-stripe @vcheung-stripe are you folks aware of any issues with vscode-stripe that might explain the above?
Issue moved from dotnet/vscode-dotnet-runtime#252
- Please respond to @sfoslund.
From @sfoslund on Wednesday, June 30, 2021 8:29:45 PM
I suspect that this is an issue with the extension that is leveraging the vscode-dotnet-runtime extension (in this case, it sounds like that's vscode-stripe). The vscode-dotnet-runtime extension simply installs the .NET runtime onto the machine but never runs it, the leveraging extension can do whatever they want with the runtime from there. Based on the error message it looks like this is an issue with how the runtime is being used, which is unrelated to the installation this extension manages.
As a result, I'm going to transfer this issue to the vscode-stripe repo, feel free to move it back if you find a installation specific issue, thanks!
Thanks for reporting, taking a look.
I re-enabled crash reporting in vscode, restarted it, saw chrome_crashpad_handler CPU tick way up and the pending report folder fill up. I then disabled vscode-stripe extension, restarted vscode, and the CPU dropped and the pending folder is no longer filling up.
Thinking back, about a week ago I remember seeing an error popup in vscode related to the Stripe extension, but at the time I dismissed it as I wasn't using the extension and didn't have time to dig into it.
@winzig, do you see any logs of interest any of the output channels? Main ones of interest would be Stripe Language Client or Stripe C# Server.
@gracegoo-stripe Was just checking that but not seeing anything at all really, maybe because I disabled the extension. Let me re-enable it and get back to you.
Here we go, this repeats infinitely when stripe is enabled:
[Info - 1:50:58 PM] Connection to server got closed. Server will restart.
Unhandled exception. System.InvalidOperationException: No instances of MSBuild could be detected.
Try calling RegisterInstance or RegisterMSBuildPath to manually register one.
at Microsoft.Build.Locator.MSBuildLocator.RegisterDefaults()
at stripe.LanguageServer.Program.Main(String[] args) in /Users/gracegoo/stripe/vscode-stripe/src/stripeDotnetLanguageServer/stripe.LanguageServer/Program.cs:line 34
at stripe.LanguageServer.Program.<Main>(String[] args)
Thanks! I think what's happening is that our extension is starting the dotnet language server because we think you have a .NET project, but the server is failing to start up because it can't find an MSBuild instance for that said project.
What kind of project do you currently have open? And what do the logs from Stripe Language Client look like?
All that aside, it's bad that the client keeps trying to restart the server when it fails with an unrecoverable exception. I'll look into this
Other interesting thing is I was seeing references in the Stripe Language Client to dotnet 5.0.7:
Detected C# Project file: /Users/tom/Documents/diag.net/Repositories/code/DiagnosticNetwork.sln
dotnet runtime acquired: /Users/tom/Library/Application Support/Code/User/globalStorage/ms-dotnettools.vscode-dotnet-runtime/.dotnet/5.0.7/dotnet
Starting C# language service for /Users/tom/Documents/diag.net/Repositories/code/DiagnosticNetwork.sln
However, I don't recall ever installing dotnet 5.x (my projects are still on 3.x). And when I run dotnet --info, I do see 5.x, but 5.0.6, not 5.0.7:
dotnet --info
.NET SDK (reflecting any global.json):
Version: 5.0.203
Commit: 383637d63f
Runtime Environment:
OS Name: Mac OS X
OS Version: 11.0
OS Platform: Darwin
RID: osx.11.0-x64
Base Path: /usr/local/share/dotnet/sdk/5.0.203/
Host (useful for support):
Version: 5.0.6
Commit: 478b2f8c0e
.NET SDKs installed:
3.1.404 [/usr/local/share/dotnet/sdk]
3.1.409 [/usr/local/share/dotnet/sdk]
5.0.203 [/usr/local/share/dotnet/sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 3.1.10 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.15 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.6 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.10 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.15 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.6 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
I'll leave dotnet/stripe alone on this machine so I can test any potential fixes you may release.
5.0 is the runtime that we use to run Stripe's C# language server app and 5.07 is installed by the vscode-dotnet-runtime extension we depend on because that's the latest version -- nothing to do with the runtime your project depends on
Thanks so much for all those logs! I have a way to reproduce this now, I'd recommend disabling the extension for .NET workspaces for now so it doesn't impeded your productivity! I'll let you know when we have a version out for you to test.
@gracegoo-stripe As for me, I am using the Stripe extension in 3 workspaces only: a .NET project, a PHP project and a JS project.
The extension is not causing issues in the PHP and JS projects. The extension is enabled for those workspaces, and the Crashpad handler is not running.
Only for the .NET workspace did the extension caused trouble for me, so I only disabled it for that project and not the other two.
As for the MSBuild locator, on macOS, I don't believe that .NET ships with msbuild.exe, only with dotnet, on which one can then call dotnet msbuild if needed. I don't know if this can be related to the issue since we're both on macOS, or maybe the dotnet-runtime extension is supposed the install msbuild, I don't know.
Thanks so much for the additional info, Pascal! Glad to hear that it's not impacting your other projects, we don't start up the C# language server unless we detect that you're working on a .NET project.
Just released 2.0.3 which addresses the constant restarting of the server. This version doesn't address the missing MSBuild instance, but it will stop trying for you after the initial attempt. You will still be able to leverage most of the Stripe extension's features that do not depend on the language server.
I can confirm 2.0.3 works as you described, I see the error in the log, but it's just a single error and no longer seeing the crashpad/dmp issues.
Unhandled exception. System.InvalidOperationException: No instances of MSBuild could be detected.
Try calling RegisterInstance or RegisterMSBuildPath to manually register one.
at Microsoft.Build.Locator.MSBuildLocator.RegisterDefaults()
at stripe.LanguageServer.Program.Main(String[] args) in /Users/gracegoo/stripe/vscode-stripe/src/stripeDotnetLanguageServer/stripe.LanguageServer/Program.cs:line 34
at stripe.LanguageServer.Program.<Main>(String[] args)
[Error - 3:58:48 PM] Connection to server got closed. Server will not be restarted.
Thank you @winzig and @pascallapradebrite4 for reporting and helping debug this issue! I will keep this issue open to look into the root cause of the startup failure.
Closing this issue and tracking the MSBuild missing issue here: #363
Closing this issue and tracking the MSBuild missing issue here: #363
That issue is closed and has not seen any activity for 66 days except myself and others saying that it still isn't fixed. I'm not sure that closing this issue and tracking it there is any better, to be honest, as this thread has a much deeper analysis of the core issue.
To be clear, on a brand new dev machine (2020 MacBook Pro) fully configured with latest and greatest, installed clean fresh install of VS Code ... and the Stripe for VS Code extension does not work, at all. I just get this error any time I try to run any Stripe command (e.g. CMD+Shift+P -> Stripe: any command).
Unhandled exception. System.InvalidOperationException: No instances of MSBuild could be detected.
Try calling RegisterInstance or RegisterMSBuildPath to manually register one.
at Microsoft.Build.Locator.MSBuildLocator.RegisterDefaults()
at stripe.LanguageServer.Program.Main(String[] args) in /Users/etsai/stripe/vscode-stripe/src/stripeDotnetLanguageServer/stripe.LanguageServer/Program.cs:line 33
at stripe.LanguageServer.Program.<Main>(String[] args)
[Error - 10:27:55 AM] Connection to server got closed. Server will not be restarted.
As I wrote in the other thread I do have these .NET SDKs installed:
dotnet --list-sdks Sat Jan 22 09:08:03 2022
3.1.407 [/usr/local/share/dotnet/sdk]
3.1.408 [/usr/local/share/dotnet/sdk]
5.0.103 [/usr/local/share/dotnet/sdk]
5.0.200 [/usr/local/share/dotnet/sdk]
5.0.201 [/usr/local/share/dotnet/sdk]
5.0.202 [/usr/local/share/dotnet/sdk]
Hey Jonah, I'm sorry about that, I'll take another stab at tracking this down. Meanwhile, you mentioned that none of the commands for the extension are working which is surprising. This error should only effect the hover-over feature. Which commands did you try?
No worries, glad it's working for you! The error is still not ideal so we'll work on tracking it down. Thanks for your feedback!