OlegKunitsyn/gnucobol-debug

Add option to attach to a running process

GitMensch opened this issue · 50 comments

GitMensch commented

We currently have the dependencies listed, it would be nice to have a sample launch configuration to debug and to attach (ideally with input variable);

    "inputs": [
        {
            "id": "pidattach",
            "type": "promptString",
            "description": "Enter the PID to attach to",
            "default": ""
        }
    ]

Bonus points (should maybe tracked in a different issue) for documentation how to setup remote debugging :-)

OlegKunitsyn commented

'Attach' type is not supported by the extension yet. Thank you for the feature request.

GitMensch commented

@OlegKunitsyn as @brunopacheco1 works on the field display and accept, may I suggest that you have a look at

  • debug existing programs (this hopefully means to primarily just skip the compilation)
  • attach to a local pid (remote debugging would be marvelous and actually one of the things I'm looking for)

As soon as these things are done I really would say the "beta" part can be removed and a version bump to 1.0 (or something near) be done.
Other issues like lmdb implementation are nice (you may see a higher priority in that, I currently don't know, but would like to) but from a "practical" use on a "default" GNU/Linux environment not as important as this and the field data issue.

OlegKunitsyn commented

Yes, this feature request is important (as well as Mac support and strange warnings in Docker), despite the fact that I focus on standard continuous integration.

  • 👍1
brunopacheco1 commented

Hallo. So I've been playing around on this issue, and by the GDB documentation and some samples online, attaching to a running process should be simple, it should. 😄

MI has a command target-attach and by that GDB should attach properly, but when the debugger is started, the initial process is killed and a new one is started, even after receiving apparently a successful message. Please, refer to the log below.

The initial PID was 3009, but then it changes to 3026.

I believe I'm doing something wrong or missing a couple of configurations. If some of you have a clue about that, it will be quite helpful.

attach_log.txt

brunopacheco1 commented

As I suspected, I was doing something wrong. So basically the debugger was attaching to a running process and executing the command exec-run which runs again the program, killing the first.

Fixing this the debugger is capable to attach correctly! 🎉

But unfortunately, life is not a colorful paradise full of unicorns. The solution for getting display value executing cob_display is printing the MI event in the running program terminal, not visible in the debugger console.

I kinda liked separated terminals, but I'll have to figure out how to print to the debugger terminal instead. Or come up with another solution for getting cob field display value.

GitMensch commented

Sounds like both "yay" and "nooo" :-)
Hm, is the output different again when attaching via gdbserver to a different machine? If yes then it may be reasonable to only support gdbserver (would be possible locally, too).
Question: you did stay in the same environment (nothing like process running in WSL), correct?

brunopacheco1 commented

Everything is running in the same local environment, but yes I'm running gdb and the running process on a WSL2. I may try directly on the Windows. Let's see.

My guess is it should behave in the same way.

GitMensch commented

I may try directly on the Windows. Let's see.

I especially meant to try on GNU/Linux native ;-)
If the "run+debug" part still works as expected and the "attach+debug" has "just" the issue that the fields cannot be parsed I suggest to pull that changes to master, allowing more people to test/investigate (and I'd try it on my Trisquel box).

brunopacheco1 commented

Well, you are right as the attaching to a running process feature is not related to displaying the variables, it just brought up a problem that soon or later would be raised.

brunopacheco1 commented

I was testing on Docker and it doesn't work out of the box. There's an option to allow the VM to see the host process tree, but it didn't help as supposed. I'm not sure if we should tackle this right now.

EDIT: I meant if we should tackle attaching to a running process from Docker. In a native environment, we could keep going.

GitMensch commented

Hm, if it is not working in most environments then it may not be that useful.
Concerning the edit: yes - attaching to a running Docker process can be postponed (for longer).

Do you know if it works on GNU/Linux native already?

brunopacheco1 commented

I didn't try, but if WSL2 is really a VM, I would say it should work, but we cannot relly on assumptions. Perhaps @OlegKunitsyn can check that.

GitMensch commented

May I suggest to PR your current changes and just not describe the attach part or (possibly better) add a note next to it, that it is experimental and works in a very limited scope + at this issue as reference).

I'll test in GNU/Linux and Win32 native as soon as it is available in a (testing) release.

OlegKunitsyn commented

I didn't try, but if WSL2 is really a VM, I would say it should work, but we cannot relly on assumptions. Perhaps @OlegKunitsyn can check that.

Of course. Can you make the branch?

OlegKunitsyn commented

I've created a boilerplate of attach/detach. On my Ubuntu I see the error ../sysdeps/unix/sysv/linux/read.c: No such file or directory Please check this out.
nofile.txt

GitMensch commented

I've created a boilerplate of attach/detach. On my Ubuntu I see the error ../sysdeps/unix/sysv/linux/read.c: No such file or directory Please check this out.

That itself is a common no-issue; the part of the log says:

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
stdout: ~"0x00007f97eb2a0fb2 in __GI___libc_read (fd=0, buf=0x55e778fbf3c0, nbytes=1024) at ../sysdeps/unix/sysv/linux/read.c:26\n"
&"26\t../sysdeps/unix/sysv/linux/read.c: No such file or directory.\n"
*stopped,frame={addr="0x00007f97eb2a0fb2",func="__GI___libc_read",args=[{name="fd",value="0"},{name="buf",value="0x55e778fbf3c0"},{name="nbytes",value="1024"}],file="../sysdeps/unix/sysv/linux/read.c",fullname="/build/glibc-YYA7BZ/glibc-2.31/io/../sysdeps/unix/sysv/linux/read.c",line="26",arch="i386:x86-64"},thread-id="1",stopped-threads="all",core="3"

The message mainly says: "Hey I'm GDB and now reached a place where you could go on - but I don't have the sources for this available".

I guess you've started a GnuCOBOL program outside of vscode before and tried to use its pid, correct? Can you attach with plain GDB?

brunopacheco1 commented

I create a PR that should by-pass this error message, forcing the extension to continue whenever a stop event comes, but there isn't a visible COBOL line.

  • 👍1
GitMensch commented

Thanks for #49!
@brunopacheco1 Can you please review if the configuration (and possibly documentation) for this is similar to the native-debug extension? It seems reasonable to me to use the same where possible.

brunopacheco1 commented

Thanks for sharing this native-debug extension. Pretty much the attach mechanism is the same.

Anyway, for some unknown reason, attaching to a PID is not working correctly in this new branch, I keep receiving a GDB warning saying the break-point couldn't be inserted.

I believe this is a minor issue, I'll keep investigating.

brunopacheco1 commented

It looks like it was a concurrency issue, once attach command was delayed, pushing break-points started working correctly.

So, basically the status is exactly the same as I've got previously (besides proper VSCode action buttons). The variable panel is not working due to missing custom cob_field eval MI events.

OlegKunitsyn commented

Same to me - I can't test attach yet. I've pushed minor conflicting changes. Please take a look, then we could merge to the master. This is beta! :)

  • 👍1
brunopacheco1 commented

Thanks for sharing this native-debug extension. Pretty much the attach mechanism is the same.

Anyway, for some unknown reason, attaching to a PID is not working correctly in this new branch, I keep receiving a GDB warning saying the break-point couldn't be inserted.

I believe this is a minor issue, I'll keep investigating.

@OlegKunitsyn Answering your question in the PR thread, I moved the target-attach command, to the start() function because I was facing an error during break-point insertion (something like warning cannot insert breakpoint 1. cannot access memory at address) and moving it fixed the issue.

But checking out the extension @GitMensch has shared WebFreak001/code-debug, it looks it should work as you've done before without delays. So, I believe either there is something wrong in the current implementation or it doesn't work as expected in a WSL2 VM.

I'll do the following, I will test the code-debug extension and see the attach on WSL2 is behaving the same there.

  • 👍1
OlegKunitsyn commented

@brunopacheco1 The code wasn't "mine" but looking at search results I have not found an initial author/license tag for referencing.

see the attach on WSL2

That's would be great! Workarounds against unknown root cause are risky. I need to fix missing stuff on my system in order to test attach properly.

OlegKunitsyn commented

I've installed missing sources but still can't have the breakpoint reached: set the breakpoint on line 9 on helloinput.cbl, execute helloinput binary in the terminal, get the pid, launch debugger, enter "PID to attach", switch back to the terminal, enter a word. Then I expect the breakpoint in debug console. Correct?
undefined.txt

brunopacheco1 commented

I am able to attach and debug the code accordingly. Please, take a look in the attachment.

image

  • 🚀1
brunopacheco1 commented

BTW, it is working after merging master to the attach branch, where there is not cob_display function.

brunopacheco1 commented

I'll do the following, I will test the code-debug extension and see the attach on WSL2 is behaving the same there.

I was testing this code-debug extension and it is not allowing me to attach to a running process, but due to other exceptions (possibly not related to this necessity of delaying the command -target-attach when running on WSL2). I'll report the bug there.

OlegKunitsyn commented

BTW, it is working after merging master to the attach branch, where there is not cob_display function.

Confirmed. I see variables as well. Thank you! Shall I merge and release?

brunopacheco1 commented
GitMensch commented

Does this already include the option for remote debugging or attaching to a local process only?

brunopacheco1 commented

Attach to a local process only. I didn't go further as we were blocked by the calling cob_display issue. Should we implement that first before merging or can we track remote debugging on another issue?

GitMensch commented

Merge and release is reasonable at any time. Additional it would be good to tackle the remote debugging independent to the cob_display issue.

brunopacheco1 commented

@GitMensch Were you expecting something specific when you requested this feature? I mean, I just committed some changes to enable the extension to attach to a gdbserver, but I believe this connection is pretty basic (comparing to remote debugging on code-debug.

I added a remoteDebugger launch variable (either this one or pid is mandatory in order to attach to a running process). But if you are looking for something like connecting to a remote debug host through ssh, for instance, these changes may not be enough.

GitMensch commented

Support of tunneling through ssh is nice (and very likely something needed in some COBOL environments where everything is closed but still accessible via ssh with a specific user/key from a specific ip address (range). [Yes, I do have something very specific in mind ;-)]
But in general your changes look simple and good - do they work in a local environment?
Could you possibly add a wiki entry or gist or something describing what to do where when you have your "developer machine" and a "server running the COBOL code"? I guess only the server needs GnuCOBOL (and gdbserver) then, only the client the debugging extensions (and gdb) - do both need GnuCOBOL? Does the client need access to the generated C sources or only the server? What to do in what order... (I have obviously no practice in using gdbserver yet, but I'm very sure this is what most "corporate" environments will need to do).

brunopacheco1 commented

Yes, it is working and so far what I did was the following:

  • Started the process;
  • Checked the PID and attached the gdbserver to it running gdbserver :{AVAILABLE_PORT} --attach {PID};
  • Set the remoteDebugger pointing to localhost and the gdbserver port;

And voilà, everything worked without surprises.

But the server needs the executable and the gdbserver to have a debuggable running process. And the developer machine needs the source code and the GC, to compile COBOL to C and prepare the source map (line and variable names), for breakpoints insertions and watch the variables, and GDB to connect to the remote debugger.

A Wiki would be nice. @OlegKunitsyn Could we add it to Github? Not sure if it is possible and how.

Support of tunneling through ssh is nice

Sure, I will take a look at the code-debug extension, to understand what is needed to enable SSH.

OlegKunitsyn commented

Wiki is enabled now.

GitMensch commented

... so I've tried it and after 10 minutes found that remoteDebugger is not yet released...
Looking forward to this and the new wiki entries.

... at least I have gdbserver running, attaching to the process, and a local gdb instance attaching to this, both via port and via stdio pipes and also adjusted the source path to the place where the generated .c files are stored.
So everything set up for testing the remote debugging (note: to debug from a Windows machine there's another thing necessary: a win32 gdb version with the correct x86_64 linux target).

GitMensch commented

@brunopacheco1 said

I will take a look at the code-debug extension, to understand what is needed to enable SSH.

That is very useful, if this works out you'd only need a ssh connection (and a client, but even Win10 comes with an openssh client/server now) as the complete debugging is done only on the server side this way (no local gdb on the client and no gdbserver on the ... server needed, only the server will have a gdb which then obviously also matches the target environment). Using an approach like this will allow to debug from vscode under Win10 to a remote GNU/Linux machine.

brunopacheco1 commented

I was checking the native-code-debug extension and it looks quite complete and powerful, as it allows to execute or attach to a process through an SSH connection, forwarding the remote X11 IO and much more, but for this initial purpose I think this is too much (as we say in Rio, it's using a cannon to kill an ant).

Basically, my idea was just to do the ssh port tunneling, forwarding the remote gdbserverPort into a localhost gdbserverPort, then connect the debugger as it would connect to a localhost gdbserver. It would require two extra launch configurations, or three if required, the remote host to connect and the remote gdbserverPort. If any extra configuration is required to enable SSH (like custom a keyfile path), the user can specify an extra field sshArgs.

What do you think? It would keep things simple and it would add tons of extra possibilities (and issues).

brunopacheco1 commented

I tried running everything manually just to see if the gdbserver connection works and by the log below it looks promising. I've got an error when trying to insert the breakpoint, but every other command had the output as expected.

ssh.txt

brunopacheco1 commented

I found the issue. It was a compilation problem, I didn't pay attention on GC version and I recompiled the code in the remote machine, but once I uploaded the locally compiled executable the attaching using an ssh tunnel worked properly.

I'm going further with this solution then if you think it's reasonable.

GitMensch commented

I'm going further with this solution then if you think it's reasonable.

Hm. as you want to remote debug it would be more reasonable to use both GC and its configured C compiler on the remote machine - then copying the results to the local machine (= the other way around).
Still: this will only work when the local GDB has the targets "target" installed, otherwise it won't be able to debug it.

... which is the reason that "doing everything over ssh as native-debug does it" is the "better" solution (it actually is good to have both remote options).

brunopacheco1 commented

This was the working log, BTW.
working_ssh_tunnel.txt

The connection to a micro instance on Google Cloud is quite slow, and I've got a warning about slowness when transferring files from remote.

brunopacheco1 commented

Hm. as you want to remote debug it would be more reasonable to use both GC and its configured C compiler on the remote machine - then copying the results to the local machine (= the other way around).

Ah, I got it. Well, when I thought about remote debugging, the first thing that came to my mind was connecting directly to a running program on production, therefore connecting through an ssh tunnel.

brunopacheco1 commented

In the way you've mentioned, for me sounds more like coding remotely, something similar to using WSL1/2 for coding on a Windows machine.

brunopacheco1 commented

@GitMensch Is this what you meant?

GitMensch commented

Well, when I thought about remote debugging, the first thing that came to my mind was connecting directly to a running program on production, therefore connecting through an ssh tunnel.

With using gdbserver, which was the first thing that came to my mind it is something like this, but the local GDB must have access to the binary and the sources and it must have one of the targets (I guess gdb supports multiple ones) that the local server uses, and the local gdb must be able to connect to the gdbserver. This connection can be part of a tunnel (SSH/VPN/...).

In the way you've mentioned, for me sounds more like coding remotely, something similar to using WSL1/2 to code in a Windows machine.

Yes, that's what it is, just without the proprietary extensions that are in WS1 and WSL2 to allow it - and actually it would allow debugging within WS1/WSL2 from VSCodium, too, which is another bonus :-)
This option mainly is: connect via SSH to the server and instruct the server's gdb to debug - neither gdbserver on the server needed, nor a local gdb (with a matching target).

Both are reasonable but the second seems to be the more "attractive" one.

brunopacheco1 commented

Okay, so everything is clear now.

If you don't mind, I would prefer splitting this two options into two different issues.

I believe connecting to a gdbserver is related to this issue so I'll finish this ssh tunneling feature, and later on (after setting variables perhaps), I could tackle that.

BTW, could you open the new issue?

brunopacheco1 commented

I believe this issue can be closed. Any extra support can be tracked by a new issue.

GitMensch commented

@brunopacheco1 Can you recheck what the issue is and open a new one for that?

I have the following working with plain gdb:

  • on the server:
    • start the application via script (internally uses cobcrun), that one waits with ACCEPT
    • on a second terminal: gdbserver $clientname:$somelocalport --attach $somepid
  • on the remote "debugging machine":
    • gdb, then within:
    • target extended-remote $remotename:$somepid
    • set sysroot remote:
    • set substitute-path /opt/app /mnt/srv/app

Note: the sources are actually not writable on the client side.

This works fine in plain gdb. Using the remote option of the last release (no idea if it uses extended-remote (which is documented to not be available everywhere) or remote (ideally we can toggle that in the launch configuration, maybe have the second as default and the first as option), I need to change cobcargs to --help to circumvent the rebuild #63, but then am prompted with:

COBOL file /mnt/srv/app/src/common/ROOT.cob compiled with exit code: 0
Error: ENOENT: no such file or directory, open '/home/simon/vsworkspace/ROOT.c'
getThreads

So we likely need also a way to specify the folder of the C sources?