PowerShell/vscode-powershell

VSCode UI locks up occassionally when using "Powershell: Restart Current Session"

JustinGrote opened this issue ยท 79 comments

Issue Description

VSCode locks up on occasion when using powershell: restart session. It looks like it happens before the PSES loads. No debug was used in this attempt.

image

Attached Logs

Follow the instructions in the README about
capturing and sending logs.

Environment Information

Visual Studio Code

Name Version
Operating System Windows_NT x64 10.0.19041
VSCode 1.42.0-insider
PowerShell Extension Version 2019.12.0

PowerShell Information

Name Value
PSVersion 5.1.19041.1
PSEdition Desktop
PSCompatibleVersions 1.0 2.0 3.0 4.0 5.0 5.1.19041.1
BuildVersion 10.0.19041.1
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1

##Issue Type: Performance Issue

Latest Powershell Preview: Use the command "Powershell: Restart Current Session". The freezing is intermittent but I have been able to reproduce on two separate computers now.

Extension version: 2019.12.0
VS Code version: Code - Insiders 1.42.0-insider (241d4048aabc31db0baed66bb3d58cf3210d981d, 2019-12-17T10:43:45.626Z)
OS version: Windows_NT x64 10.0.19041

System Info
Item Value
CPUs Intel(R) Core(TM) i5-1035G7 CPU @ 1.20GHz (8 x 1498)
GPU Status 2d_canvas: enabled
flash_3d: enabled
flash_stage3d: enabled
flash_stage3d_baseline: enabled
gpu_compositing: enabled
metal: disabled_off
multiple_raster_threads: enabled_on
oop_rasterization: disabled_off
protected_video_decode: enabled
rasterization: enabled
skia_renderer: disabled_off
surface_control: disabled_off
surface_synchronization: enabled_on
video_decode: enabled
viz_display_compositor: enabled_on
viz_hit_test_surface_layer: disabled_off
webgl: enabled
webgl2: enabled
Load (avg) undefined
Memory (System) 7.60GB (1.49GB free)
Process Argv --folder-uri file:///c%3A/Users/jgrote/Projects/CredDriveGist/97fe1269f17d471191a08610fe71b956
Screen Reader no
VM 0%
Process Info
CPU %	Mem MB	   PID	Process
    1	   116	 13408	code-insiders main
    0	   222	  9032	   window (Untitled-1 - 97fe1269f17d471191a08610fe71b956 - Visual Studio Code - Insiders)
    0	    15	  1520	     watcherService 
    0	    11	  6092	       console-window-host (Windows internal process)
    0	     6	 10580	     console-window-host (Windows internal process)
    0	   138	 10960	     C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass -Command "Import-Module 'c:\Users\jgrote\.vscode-insiders\extensions\ms-vscode.powershell-preview-2019.12.0\modules\PowerShellEditorServices\PowerShellEditorServices.psd1'; Start-EditorServices -HostName 'Visual Studio Code Host' -HostProfileId 'Microsoft.VSCode' -HostVersion '2019.12.0' -AdditionalModules @('PowerShellEditorServices.VSCode') -BundledModulesPath 'c:\Users\jgrote\.vscode-insiders\extensions\ms-vscode.powershell-preview-2019.12.0\modules' -EnableConsoleRepl -LogLevel 'Normal' -LogPath 'c:\Users\jgrote\.vscode-insiders\extensions\ms-vscode.powershell-preview-2019.12.0\logs\1576688022-5d54cd62-c730-4487-bd08-747a9bd419e61576688019997\EditorServices.log' -SessionDetailsPath 'c:\Users\jgrote\.vscode-insiders\extensions\ms-vscode.powershell-preview-2019.12.0\sessions\PSES-VSCode-13408-997449' -FeatureFlags @() "
    0	    86	 10976	     pwsh.exe -nologo
    0	   135	 14120	     extensionHost
    0	    72	 14296	     searchService
    0	     6	 15512	     console-window-host (Windows internal process)
    0	    86	 10884	   window (Issue Reporter)
    0	     7	 12116	   C:\Users\jgrote\AppData\Local\Temp\vscode-update-user-x64\CodeSetup-insider-26f5dfcd76c9a02f116a706844611aeb09ae8c0b.exe /verysilent /update="C:\Users\jgrote\AppData\Local\Temp\vscode-update-user-x64\CodeSetup-insider-26f5dfcd76c9a02f116a706844611aeb09ae8c0b.flag" /nocloseapplications /mergetasks=runcode,!desktopicon,!quicklaunchicon
    0	    22	 12104	     "C:\Users\jgrote\AppData\Local\Temp\is-VT9TQ.tmp\CodeSetup-insider-26f5dfcd76c9a02f116a706844611aeb09ae8c0b.tmp" /SL5="$30662,58711380,168448,C:\Users\jgrote\AppData\Local\Temp\vscode-update-user-x64\CodeSetup-insider-26f5dfcd76c9a02f116a706844611aeb09ae8c0b.exe" /verysilent /update="C:\Users\jgrote\AppData\Local\Temp\vscode-update-user-x64\CodeSetup-insider-26f5dfcd76c9a02f116a706844611aeb09ae8c0b.flag" /nocloseapplications /mergetasks=runcode,!desktopicon,!quicklaunchicon
    2	   261	 14192	   gpu-process
    1	   178	 14760	   window (EditorServices.log - 1576688022-5d54cd62-c730-4487-bd08-747a9bd419e61576688019997 - Visual Studio Code - Insiders)
    0	    73	  9372	     searchService
    0	   126	 12912	     extensionHost
    0	    15	 14976	     watcherService 
    0	    11	 14004	       console-window-host (Windows internal process)
    0	    54	 15252	   electron-crash-reporter
    0	    88	 15368	   shared-process
Workspace Info
|  Window (Untitled-1 - 97fe1269f17d471191a08610fe71b956 - Visual Studio Code - Insiders)
|  Window (EditorServices.log - 1576688022-5d54cd62-c730-4487-bd08-747a9bd419e61576688019997 - Visual Studio Code - Insiders)
|    Folder (1576688022-5d54cd62-c730-4487-bd08-747a9bd419e61576688019997): 3 files
|      File types: log(3)
|      Conf files:
|    Folder (97fe1269f17d471191a08610fe71b956): 5 files
|      File types: psm1(1) ps1xml(1) dll(1) ps1(1)
|      Conf files:;

Visual Studio Code Extensions

Visual Studio Code Extensions(Click to Expand)
Extension Author Version
better-align wwm 1.1.6
better-comments aaron-bond 2.0.5
better-powershell-syntax-highlighting justin-grote 0.0.2
bracket-pair-colorizer-2 CoenraadS 0.0.29
code-settings-sync Shan 3.4.3
gc-excelviewer GrapeCity 2.1.32
gistfs vsls-contrib 0.0.21
git-graph mhutchie 1.19.1
gitlens eamodio 10.2.0
indent-rainbow oderwat 7.4.0
markdown-all-in-one yzhang 2.6.1
powershell-preview ms-vscode 2019.12.0
remote-containers ms-vscode-remote 0.96.0
remote-ssh-edit-nightly ms-vscode-remote 2019.12.24000
remote-ssh-nightly ms-vscode-remote 2019.12.24000
remote-wsl ms-vscode-remote 0.41.6
todo-tree Gruntfuggly 0.0.162
vscode-icons vscode-icons-team 9.6.0
vscode-peacock johnpapa 3.2.0
vscode-sort-json richie5um2 1.18.0
vscode-zipexplorer slevesque 0.3.1

It also appears to happen when just closing the terminal with the X button, so I imagine this is tied to the shutdown process in some way.

Happened to me just now when closing it with the "trash" button on the terminal bar, toast didn't even completely appear before it froze:

image

This is odd... I don't see how the extension could be causing VS Code to hang like this. I notice you are using insiders... I'm curious to know if you only experience this in insiders or it also happens in stable.

Yeah I don't use insiders, it's on stable too.

@TylerLeonhardt Are you not able to reproduce it? Maybe it's a Windows thing? It happens at least 50% of the time for me.

Happens for me too on stable release.

Are we going to get freezing consoles/VSCode until windows 21H1? can't it be fixed sooner?
microsoft/vscode#71966
microsoft/terminal#1810

Originally posted by @ili101 in PowerShell/PowerShell-RFC#238 (comment)

To reproduce write a in the console (no enter) then trash the console. VSCode will freeze.
Reproducible on Windows 10 1909 (19H2) and 1903 (19H1) I think.
Can vscode-powershell team code some workaround for this?

@ili101 Oh wow nice find, I must have updated windows at the same time as the preview extension and didn't think about it. Unfortunately it's unlikely that this extension can do anything. Any code in this project is several layers of abstraction away from whatever is causing the issue.

Hmm. Probably no way to proactively detect this and throw a toast or something that says "it's not Powershell's fault!"? :)

microsoft/terminal#1810 references that you have to be using ConPTY for it to happen. I'm going to try unchecking this box:
image

And seeing if it still happens. If not, then that's something that can definitely be warned about, the extension can pop up a toast with "there is a known issue with using Conpty where you may experience lockups, would you like to turn it off?"

EDIT: Cure may be worse than the disease: microsoft/vscode#45693

Added this function to my profile so that I don't have to wait forever for code to realize it is hung.

function bounceCode {get-process code* | stop-process;code}

For update, this is stuck on microsoft/terminal#1810 until its fixed, that's the root cause it seems.

yikes... ok. Well, I'm tempted to close this but for now, I'll mark it as a vscode-bug until I'm in a triage meeting with Sydney and Rob again.

According to the terminal team it can be fixed in VSCode, looking for a related issue if it already exists.
microsoft/terminal#1810 (comment)

EDIT: New issue created as microsoft/vscode#88003

The new root issue is microsoft/node-pty#375 which vscode depends on.

I just hit this issue myself. The way to differentiate this from other hangs is that the entire VSCode window will stop responding, so no responses to mouse clicks like highlighting or opening files. If I hit it again I'll get a proc dump.

As a workaround, I wonder if using powershell.integratedConsole.useLegacyReadLine will have an effect here since it doesn't have PSReadLine... I doubt it though.

@Tyriar just so you're aware of this issue - there are a few folks on Twitter too who've complained about this issue.

@TylerLeonhardt you can also move back to winpty to avoid it as the crash is due to conpty:

"terminal.integrated.windowsEnableConpty": false

You might notice other quirks with that though.

@TylerLeonhardt you can also move back to winpty to avoid it as the crash is due to conpty:

"terminal.integrated.windowsEnableConpty": false

You might notice other quirks with that though.

@Tyriar
I continue to have this issue across both of my PCs. Windows build 18309 or higher is required to disable conpty with that setting. My company's PCs are not to this build yet. Adding LegacyReadLine does not help the issue. Unfortunately these workarounds do not help for me.

I only have this issue on the preview extension. I previously submitted #2387 with logs and that was closed as a duplicate of this issue. If this issue makes it to stable it will kill VSCode PowerShell for my team.

I'm happy to provide more logs or perform additional steps to help with this issue.

@joshfria are you experiencing a lock up in the UI of VSCode as well? Or just in the terminal? Are you able to provide the following:

  • Your extension logs
  • A gif of your screen when the lock up happens
  • The versions you're using of VSCode (in the About menu), PowerShell ($PSVersionTable) and Windows (systeminfo.exe | select -first 2 -skip 2)

@rjmholt this is certainly microsoft/node-pty#375 (microsoft/terminal#1810, microsoft/vscode#71966) which happens when conpty terminals get closed. I don't think further info is needed.

In @joshfria's case, he reported that it only happens with the preview extension. I've reopened #2387 to track that separately and see if I can reproduce it. Given that it's on an older Windows version and appears only in the preview extension, I'm thinking the issue he's hitting isn't what we're tracking in this issue.

Still a problem on 2020.2.0 Preview
(I know it's a different Microsoft department bug but you are going to probably get a lot of complaints when preview becomes stable if it's not fixed)

Still a problem on 2020.2.0 Preview

This issue is tracking a VSCode bug which occurs in stable too, so the particular extension installed won't affect its occurrence, only the fact of the terminal being used by the extension

Still a problem on 2020.2.0 Preview

This issue is tracking a VSCode bug which occurs in stable too, so the particular extension installed won't affect its occurrence, only the fact of the terminal being used by the extension

To reproduce write a in the console (no enter) then trash the console. VSCode will freeze.

Crushes 100% of the times on preview, never crashes on stable.

@ili101 can you try either setting:

powershell.integratedConsole.useLegacyReadLine: true

or

"terminal.integrated.windowsEnableConpty": false

and see if that changes anything for you? We need more data.

@TylerLeonhardt I tested using the "a" method (type a and then trash the window) and did it 3 times in each style. vscode vs. vscode-insiders didn't matter. I did it in a "clean" install by downloading the portable zip and adding the data folder.

image

@ili101 @JustinGrote just to be clear, are you seeing a UI freeze (where clicking on things in VSCode or on the X does nothing) or just a terminal freeze?

@rjmholt we get a full UI freeze like posted before:

Happened to me just now when closing it with the "trash" button on the terminal bar, toast didn't even completely appear before it froze:

image

Then you wait for ~30 seconds for VSCode to notice it's frozen and you get the massage as posted in OP first message:

Issue Description

VSCode locks up on occasion when using powershell: restart session. It looks like it happens before the PSES loads. No debug was used in this attempt.

image

we get a full UI freeze like posted before

Ok understood. It's not clear to me why the legacy extension wouldn't cause this if it's using the same terminal API to restart the PowerShell process. That's the source of my confusion.

I'll see if I can look into the source code and see if there are any differences.

@rjmholt @ili101 covered the issue specifically.

As mentioned microsoft/node-pty#375 is the root cause most likely. I would further assume it's something in PSReadline or in code added to support psreadline, since that's the main difference between legacy and stable (legacy doesn't run psreadline) that interacts with this ConPTY thread issue. Disabling ConPTY or Disabling PSReadline (while leaving conPTY enabled) will "fix" the issue, but neither of those are desirable at all.

I would further assume it's something in PSReadline

This is probably the most likely explanation. Unfortunately that suggests it's two libraries we don't control colliding with each other...

Disabling ConPTY or Disabling PSReadline (while leaving conPTY enabled) will "fix" the issue, but neither of those are desirable at all.

I absolutely agree

@rjmholt FYI let us know if you can't reproduce, it's pretty simple:

  1. Start an Integrated Terminal in a code instance with powershell preview extension installed.
  2. Type 'a'
  3. Click the trash icon in the terminal bar.

It's the typing 'a' part that makes me think something in PSReadline's evaluation of what you type calls some ConPTY function that triggers the microsoft/node-pty#375 behavior

Yeah, that's a nice repro, happens for me too. And it does seem likely that something in the depths of PSReadLine is exacerbating the issue. But currently not clear at all what we can do to fix it. @TylerLeonhardt is currently looking into it.

Does anyone feel like this happens when they don't interact with the console before hand? Cause we could workaround this issue by hiding the PSIC until it's ready to read input, but that's a workaround to this problem and not really a fix.

So a bit more investigating here reveals that it occurs in ordinary PowerShell within VSCode as well (I cut some frames out while we wait for the crash dialogue, since otherwise it was quite a long GIF):

lock

@SteveL-MSFT and @daxian-dbw might be interested in this

@JustinGrote and @ili101 thanks for following up with us on this btw. @JustinGrote I see you've tracked this issue across several (Microsoft-owned) GitHub repos and just want to thank you for that too.

We're going to do our best to follow up on this issue on our end since PowerShell users are heavily affected.

I just ran into this with the 2020.3.0 update. :-(

I'd encourage anyone hitting this issue to add their ๐Ÿ‘ to this issue and:

Are we looking at years on this issue?

@joshfria as previously noted, it's an upstream issue the vscode-powershell team has zero control over. I encourage you add a thumbs up and to lodge your complaint at :
microsoft/node-pty#375

Why is this bug only present in 2020.3 and ahead?

Why is this bug only present in 2020.3 and ahead?

Because 2020.3 adds the PSReadLine module, which is involved in triggering the underlying VSCode bug. You can try the setting in #2377 (comment) to disable it

I did try disabling it last month and it doesn't work unfortunately. Could it be loading PSReadline when the terminal starts and then disabling it?

@joshfria the bug usually shows up when you type input into the terminal before it is ready. Can you confirm that you see this issue once you start typing in the console?

@TylerLeonhardt actually the easiest way to reproduce is type the letter 'a' and then trash the console with the trashcan icon. It freezes every time.

Interestingly, typing a and then backspacing it and it will be fine when you trash the console. Maybe something PSReadline is emitting for a "current prompt" that is causing the hang that may be able to be extracted or filtered?

@JustinGrote yeah I'm trying to toy with the idea of having the PSIC be readonly until the prompt shows... as a workaround until the issue is fixed. If that's not possible (vscode API doesn't exist), then hiding the terminal until it's ready and then show it.

There is another idea to workaround this in PSRL... but I don't think we have enough to go off of there just yet.

@TylerLeonhardt It happens even if the prompt is shown. You can leave it open for 5 minutes, then just type 'a' and click trash and it will happen every time (at least for me). Maybe these are different crashes?

Well that's a repo but I guess the question is "why are people doing that?" if that's what they're doing that shows this issue.

My initial thought was that the majority of users saw this was happening because users tried typing in the Integrated Console before it was ready... but if it's because they're doing what you're saying, I think we need to understand why they're doing that because I can't prevent them from doing that in any way.

For me this happens daily on VSCode restart (plugin update) or restart of PS session to fix IntelliSense issues). I don't have to type anything for it to crash. It's not related to any of my actions in console.

@TylerLeonhardt let me rephrase.

  1. It is consistently reproducible if there is "pending" text in the terminal console. I just used 'a' as an example. You can type "thiswillcrash" and then click the trash button and same will happen
  2. It only occurs with the trash button easily vs. doing restart current session or reload window.
  3. It doesn't happen if Use Legacy Read Line is checked (disable psreadline)

Capture

In this demo I forgot to show that clicking trash with nothing at the terminal prompt will work fine.

I've generally been able to avoid hard locks by just making sure my terminal prompt is clear before I reload. So far haven't had a hang following this practice but that's anecdotal at the moment.

So I wonder if it is something in PSReadline or the interaction between PSReadline/PSES that is emitting some kind of special character or handler that gets manifested by microsoft/node-pty#375 and perhaps whatever that is could be suppressed temporarily if it doesn't cause a massive breaking change?

That's a great summary, @JustinGrote.

With that said, @joshfria seems to be experiencing something else I guess if it repros for them when using the legacy readline...

@TylerLeonhardt I think the issue I'm experiencing may be Avecto related based on other issues I'm reading. Why it only happens on Windows PowerShell 64bit I don't know.

Ahh so you're using Avecto as well... Yeah I'm not sure why, unfortunately.

Here's where we stand with that:
#2526 (comment)

The upstream PR fix is in the works for this!
microsoft/node-pty#415

@TylerLeonhardt can someone proactively test a vscode build with this PR once it is ready to make sure that it addresses this issue, so feedback can be made to the PR prior to it being merged if it doesn't?

@JustinGrote I just tested in VS Code and it works, I need to figure out how to bring it in though as there are complications around using worker_threads in Electron microsoft/node-pty#415 (comment)

@Tyriar, thanks for your hard work on this, you need a beer donate button :). I was mostly concerned with testing it against the Powershell Integrated Console. I'm 90% sure your fix will carry downstream but wanted to make sure there wasn't an unrelated issue.

@Tyriar lmk if you need me to do PowerShell extension testing

@Tyriar checking in, I saw you created a "hang_fix" branch that someone else said worked, is this still valid? Can you provide me a link so I can try to build a non-crashing vscode? :)

I believe @Tyriar is on vacation and maybe @meganrogge could chime in?

The fix is still good, but it relies on microsoft/vscode#74620 which is a bigger piece of debt work. Luckily part of that was done for the upcoming release (flow control) and that issue will also block the upcoming version of Electron, so we will be forced to handle it very soon.

@Tyriar I'm happy to see forward progress on it, this still annoys me literally every day and it's been over a year, but I understand the hurdle to clear. Just happy there are people like you smarter and/or more experienced on the C/C#/Typescript side than me to figure it out :)

This is likely fixed by
microsoft/vscode#116185

Which should land in insiders within the next week. Fingers crossed.

@rjmholt @andschwa The latest insiders has fixed the hard lock issue finally, but the PTY still crashes and has to be restarted, and its still a prolongued recovery route without reloading the entire window.

May need to tie in with @Tyriar's pty restart logic in microsoft/vscode#116948 and restart the powershell integrated console when this event is fired.

To reproduce:

  1. Download latest insiders
  2. type 'a' into the integrated console and then click the trash icon.
  3. No longer causes a hard lock but the PTY becomes unresponsive and the language service freaks out even if "Powershell: Restart Current Session" is run.

@andschwa to follow-up, if you can detect the event that causes the integrated console to go into "disconnected state" due to PTY failure, you could pop up the same restart session dialog that usually shows up, and at least make this issue recovery flow more friendly.

Capture.mp4

That sounds nice! Excited to be able to address this.

This thread is really long,.. was there a workaround for this at the moment ?

image

@brwilkinson This is the first part of the solution that is now in @code. @Tyriar made it so that it no longer hard locks when the original issue occurs.

At this point to recover, you just need to do the following steps:

  1. Click the Restart PTY Host
  2. Wait a moment
  3. F1 -> Powershell: Restart Current Session

And you should be back in business rather than having to wait for vscode to allow you to kill it.

The second part of the fix will hopefully be twofold:

  1. @andschwa can maybe hook into the PTY host failure event and automate the recovery steps above.
  2. @Tyriar will be able to fix what causes the "hard crash" of the powershell console in the first place (this is not a powershell specific issue, it affects all terminals, the integrated terminal just has some behavior that makes it much more likely to happen)

Thanks @JustinGrote any preferred environment to work in for least issues?

i.e. code vs code-insiders ?
powershell extension vs powershell preview extension ?

Happens everywhere still currently, but I'd say to avoid the crash don't ever click the "trash" button on the powershell integrated terminal, especially if you have some already-entered text at the command prompt.

The way to consistently reproduce it is to type 'a' at the integrated console and then click the trash icon, it will hard crash every time.

haa I haven't had any problem trying to reproduce it ๐Ÿ˜€ ...

some days getting any work done feels like wrestling an octopus ๐Ÿ™

seems like there is a plan in place. .. ๐Ÿ‘๐Ÿป ...

#3216 I'm in an endless loop between this other issue and this one.

@JustinGrote @brwilkinson Can either of you confirm that this is still reproducing or not on the latest version of VS Code and the preview extension? We've not automated the recovery steps, but I think the UI lockups should be gone (I can't repro on my machine).

hmmm yes and no :) let me think about it for a day... Definitely far less crashes/freezes...

I mainly want to think about the following, which I am still seeing a little bit.

i.e. The connection to the terminal's pty host process is unresponsive, the terminals may stop working.

which then goes to

image

One thing for sure I am not currently losing any productivity over this... using powershell preview extension in both Code and Insiders.

@andschwa UI full lockups are gone due to PTY updates but they've basically just been masked. It's easy to reproduce, just type 'a' into integrated console and kill the console

The PTY handler will crash, you have to restart it, and then you also have to coordinate the restart of the integrated console after. I think @Tyriar is still working on the PTY crash part. Maybe @TylerLeonhardt can help with a press for resources on the vscode side for this tricky issue :)

Video.mp4

just type 'a' into integrated console and kill the console

Is this in a particular environment? I've tested on my Mac and it doesn't repro.

Screen.Recording.2021-04-28.at.3.33.13.PM.mov

I'm on Windows Insider, able to do it in both stable and insiders for years (that demo was just a few minutes ago). I don't think it's explicity OS specific, it just may require a different character to reproduce.

I think this is the most current upstream issue:
microsoft/vscode#71966

Oh...I see. IMHO terminal support on Windows is still...beta. Heck, even the Terminal app on Windows is in preview (still? I think). Thanks for the update!

@JustinGrote The bug you linked appears now resolved. Is this still reproducing?

I can no longer reproduce with the "a and then trash icon" method, so hopefully we are good! @Tyriar thanks so much for finally helping put this long standing aggravating issue to bed.