Remote attach debug server not dying upon killing the attached process

Question

Remote attach debug server not dying upon killing the attached process

zbs opened this issue 24 days ago · 19 comments

Type: Bug

Behaviour

Extremely frequently the VS Code remote attach debugger will not die when I kill the python -m debugpy ... process that is attaching to the port 5678. Until I restart the IDE I cannot start debugging again, because otherwise it will complain that "Python debugger is already debugging another instance". My colleaques haven experiened this issue as well, and it seems to be a recent regression (last couple of weeks).

As I wrote this, I cannot click the "Create on GitHub" button because VSCode is seeming unresponsive. I resolved by restarting the IDE.

Steps to reproduce:

Use the below config for debugger
Run python -m debugpy --wait-for-client --listen 5678 /path/to/module.py
Run debugger
Either detach debugger or CTRL-C the process on command line

Diagnostic data

launch.json configuration

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [

        {
            "name": "Python: Remote Attach",
            "type": "python",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "."
                }
            ],
            "justMyCode": false
        }
    ]
}

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

Output for Python Debugger in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python Debugger)

XXX

Extension version: 2024.15.2024121301
VS Code version: Code 1.95.3 (f1a4fb101478ce6ec82fe9627c43efbf9e98c813, 2024-11-13T14:50:04.152Z)
OS version: Windows_NT x64 10.0.22631
Modes:
Remote OS version: Linux x64 4.18.0-372.32.1.el8_6.x86_64
Remote OS version: Linux x64 4.18.0-372.32.1.el8_6.x86_64
Remote OS version: Linux x64 4.18.0-372.32.1.el8_6.x86_64
Connection to 'SSH: zbs-carp2' could not be established Canceled
Remote OS version: Linux x64 4.18.0-372.32.1.el8_6.x86_64

Python version (& distribution if applicable, e.g. Anaconda): 3.8.19
Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Conda

System Info

Item	Value
CPUs	11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz (16 x 2611)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	undefined
Memory (System)	63.73GB (37.47GB free)
Process Argv	--folder-uri vscode-remote://ssh-remote%2Bzbs-carp2/physical/gpfs/carp2-home/car_home02/data_files/jcs/zsilversmith/dev/git/eis3 --crash-reporter-id 2a67a766-b9ca-442f-87e6-12fdf4c1e37f
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: zbs-carp2
OS	Linux x64 4.18.0-372.32.1.el8_6.x86_64
CPUs	AMD EPYC 7763 64-Core Processor (128 x 0)
Memory (System)	2014.91GB (1217.89GB free)
VM	0%

Item	Value
Remote	SSH: zbs-carp2
OS	Linux x64 4.18.0-372.32.1.el8_6.x86_64
CPUs	AMD EPYC 7763 64-Core Processor (128 x 0)
Memory (System)	2014.91GB (1217.89GB free)
VM	0%

Item	Value
Remote	SSH: zbs-carp2
OS	Linux x64 4.18.0-372.32.1.el8_6.x86_64
CPUs	AMD EPYC 7763 64-Core Processor (128 x 0)
Memory (System)	2014.91GB (1217.89GB free)
VM	0%

Connection to 'SSH: zbs-carp2' could not be established Canceled

Item	Value
Remote	SSH: zbs-carp2
OS	Linux x64 4.18.0-372.32.1.el8_6.x86_64
CPUs	AMD EPYC 7763 64-Core Processor (128 x 0)
Memory (System)	2014.91GB (1217.89GB free)
VM	0%

A/B Experiments

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805cf:30301675
binariesv615:30325510
vsaa593cf:30376535
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
2i9eh265:30646982
962ge761:30959799
pythonnoceb:30805159
pythonmypyd1:30879173
2e7ec940:31000449
pythontbext0:30879054
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
dvdeprecation:31068756
dwnewjupyter:31046869
newcmakeconfigv2:31071590
nativerepl2:31139839
pythonrstrctxt:31112756
nativeloc1:31192215
cf971741:31144450
iacca1:31171482
notype1:31157159
5fd0e150:31155592
dwcopilot:31170013
stablechunks:31184530
6074i472:31201624

Answer 1 · 2024-12-16T17:12:17.000Z

@zbs In this case is the debugger process (i.e., python -m debugpy) terminated? or is that also blocked.

Moving this to debugpy to investigate. Basically, if the process is terminated, we do an atexit registration that should let IDE know that the debugger is terminated. We need to make sure that the termination message is being sent to the IDE.

If debugpy is terminating correctly after sending the message, then this issue is in the DAP Client, and we should move this issue to VS Code core.

Answer 2 · 2024-12-16T18:10:06.000Z

I can't seem to reproduce this myself. Can you turn on logging and send back the logs? They'll end up on the remote machine. It should say if the debugger sent the terminate or not.

Answer 3 · 2024-12-16T19:51:02.000Z

Thanks all. Running with logging and will report back

@zbs In this case is the debugger process (i.e., python -m debugpy) terminated? or is that also blocked.

Yes AFAICT it's properly killed.

Answer 4 · 2024-12-16T20:07:10.000Z

OK repro'd again. This time, steps were

python -m debugpy --wait-for-client --listen 5678 --log-to ~/vscode_debug_logs ....
Attach via vs code gui
Let the debugger break on an exception/breakpoint
CTRL-C out of the command line process
Re-run step 1
Re-run step 2. At this point, the blue "processing/in progress" animation starts and never stops
Kill the command line process that is waiting
Re-run step 1
Attempt to use VS code debug gui once more. This is when I hit a "another instance running" message.

I have the logs, how should I send them, or what should I extract from them?

Answer 5 · 2024-12-16T20:16:57.000Z

You can just zip them and drop the zip file in the issue.

Answer 6 · 2024-12-16T20:17:45.000Z

We might need the VS code side logs too, that would be using the logToFile option in the launch.json. It will write out the logs to where the debugger extension is installed.

Answer 7 · 2024-12-16T20:34:02.000Z

logs.tar.gz

Answer 8 · 2024-12-16T21:23:29.000Z

There should have been some pydevd logs too, not sure why those didn't show up. There were four instances of the debugger starting.

Two looked like it was killed by the keyboard interrupt and the client responded with the disconnect. The other two look like it failed to connect.

I'd fix this part first though. You're using a super old debugpy on the server. Not sure how it even works:

I+00000.035: Linux-4.18.0-372.32.1.el8_6.x86_64-x86_64-with-glibc2.10 x86_64
             CPython 3.8.15 (64-bit)
             debugpy 1.0.0b12

Debugpy's latest is 1.8.11

Answer 9 · 2024-12-16T21:23:54.000Z

You might also use a newer CPython, as that version isn't technically supported anymore.

Answer 10 · 2024-12-16T21:36:10.000Z

Ah, that's probably why. I'll ask our internal maintainer of that conda environment to bump the debugpy version. After you pointed that out, I've switched back to our main environment, which has version 1.8, and I have yet to repro this issue.

Since that seems like the strongest lead, we can probably close this out unless I encounter this problem again. Thanks for all the help and sorry for the time waste.

Answer 11 · 2024-12-16T21:57:18.000Z

OK looks like this still repros with 1.8. One thing I observed is that when I get in this state, the debug extension on the left hand side has a blue 1, and upon hover, it says "1 active session". However there's no debug control modals (or blue bar on the bottom) indicating a debug session is actually happening

Answer 12 · 2024-12-16T22:17:21.000Z

And this is after you've killed all the processes on the server? Does the process list include anything with debugpy in it?

Answer 13 · 2024-12-16T22:27:42.000Z

And this is after you've killed all the processes on the server?

yes

Does the process list include anything with debugpy in it?

there were other debugpy servers attached to different ports. I killed them but I'm still blocked from starting another debug session

Answer 14 · 2024-12-16T22:29:25.000Z

This might be a red herring, but I noticed that if I click the "Play" button when it's in this state, nothing happens; however, when I do CTRL + ALT + F10 (which I have bound to the command Debug: start debugging) the "Python: Remote Attach is already running" warning pops up

Answer 15 · 2024-12-16T22:30:12.000Z

Is this server shared with other people? How were there other debugpy instances running?

I think if you just killed them it didn't shutdown correctly. Hitting CTRL+C on the app when it's running should send the correct shutdown, but killing all of the debugpy processes isn't going to send the terminate across. Launching the debugger starts a number of different processes.

Answer 16 · 2024-12-16T22:31:50.000Z

Is this server shared with other people? How were there other debugpy instances running?

Yes but that's moot -- I run several workspaces off this remote host, and have different workspaces associated with different ports for remote debug attaching.

Answer 17 · 2024-12-16T22:33:58.000Z

We likely need the log on both sides to see what the last thing that VS code saw was.

Answer 18 · 2024-12-16T22:44:28.000Z

OK, which of these files?

drwxr-sr-x  3 zsilversmith kb   4096 Dec 13 08:18 bundled
-rw-r--r--  1 zsilversmith kb   4020 Dec 13 08:18 component-detection-pip-report.json
-rw-r--r--  1 zsilversmith kb   2706 Dec 13 08:18 debugpy_info.json
drwxr-sr-x  2 zsilversmith kb   4096 Dec 13 08:18 dist
-rw-r--r--  1 zsilversmith kb    278 Dec 13 08:18 .flake8
-rw-r--r--  1 zsilversmith kb  48597 Dec 13 08:18 icon.png
drwxr-sr-x  2 zsilversmith kb   4096 Dec 13 08:18 l10n
-rw-r--r--  1 zsilversmith kb   1141 Dec 13 08:18 LICENSE.txt
-rw-r--r--  1 zsilversmith kb  17932 Dec 16 13:34 package.json
-rw-r--r--  1 zsilversmith kb    743 Dec 13 08:18 package.nls.cs.json
-rw-r--r--  1 zsilversmith kb    685 Dec 13 08:18 package.nls.de.json
-rw-r--r--  1 zsilversmith kb    696 Dec 13 08:18 package.nls.es.json
-rw-r--r--  1 zsilversmith kb    734 Dec 13 08:18 package.nls.fr.json
-rw-r--r--  1 zsilversmith kb    705 Dec 13 08:18 package.nls.it.json
-rw-r--r--  1 zsilversmith kb    815 Dec 13 08:18 package.nls.ja.json
-rw-r--r--  1 zsilversmith kb    601 Dec 13 08:18 package.nls.json
-rw-r--r--  1 zsilversmith kb    753 Dec 13 08:18 package.nls.ko.json
-rw-r--r--  1 zsilversmith kb    747 Dec 13 08:18 package.nls.pl.json
-rw-r--r--  1 zsilversmith kb    659 Dec 13 08:18 package.nls.pt-br.json
-rw-r--r--  1 zsilversmith kb    743 Dec 13 08:18 package.nls.qps-ploc.json
-rw-r--r--  1 zsilversmith kb    946 Dec 13 08:18 package.nls.ru.json
-rw-r--r--  1 zsilversmith kb    807 Dec 13 08:18 package.nls.tr.json
-rw-r--r--  1 zsilversmith kb    608 Dec 13 08:18 package.nls.zh-cn.json
-rw-r--r--  1 zsilversmith kb    629 Dec 13 08:18 package.nls.zh-tw.json
-rw-r--r--  1 zsilversmith kb   4996 Dec 13 08:18 readme.md
drwxr-sr-x  2 zsilversmith kb   4096 Dec 13 08:18 resources
-rw-r--r--  1 zsilversmith kb   2757 Dec 13 08:18 SECURITY.md
-rw-r--r--  1 zsilversmith kb    730 Dec 13 08:18 SUPPORT.md
-rw-r--r--  1 zsilversmith kb   5483 Dec 13 08:18 telemetry.json
-rw-r--r--  1 zsilversmith kb 180326 Dec 13 08:18 ThirdPartyNotices.txt
-rw-r--r--  1 zsilversmith kb   2876 Dec 13 08:18 .vsixmanifest

Answer 19 · 2024-12-16T22:47:31.000Z

There should be a bunch of log files. Hmm, maybe it wouldn't work in remote, but I thought the extension (vscode-python-debug) wrote out the VS code side.

It should happen if you add logToFile to the launch.json.