containers/conmon-rs

Attach detaches from container too early

jakecorrenti opened this issue · 2 comments

jakecorrenti commented

What happened?

I am currently trying to integrate conmon-rs into Podman. I am trying to get the attach functionality working. When I run the command sudo bin/podman run --log-driver=k8s-file --name=demo -d alpine top, the container successfully starts and the logs display that there are bytes being read:

2022-08-11T18:14:42.317436Z DEBUG backend:create_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="1c4c62f4-c230-485f-895e-aee634fe43a6"}:promise:stdout: conmonrs::container_io: 238: Read 304 bytes
2022-08-11T18:14:47.322829Z DEBUG backend:create_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="1c4c62f4-c230-485f-895e-aee634fe43a6"}:promise:stdout: conmonrs::container_io: 238: Read 304 bytes
2022-08-11T18:14:52.327820Z DEBUG backend:create_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="1c4c62f4-c230-485f-895e-aee634fe43a6"}:promise:stdout: conmonrs::container_io: 238: Read 304 bytes

When I try and attach to the container doing sudo bin/podman attach demo, it attaches and briefly displays the output of top:

Mem: 15176388K used, 50390388K free, 1378696K shrd, 4192K buff, 9559136K cached
CPU:   1% usr   0% sys   0% nic  98% idle   0% io   0% irq   0% sirq
Load average: 0.26 0.45 0.47 1/1736 1
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    1     0 root     R     1604   0%  10   0% top

However, after this output displays, the container immediately detaches even though the logs show that conmon-rs is still reading bytes.

Log output when attaching:

2022-08-11T18:16:01.775594Z DEBUG backend: conmonrs::rpc: 50: Got a version request
2022-08-11T18:16:01.775814Z DEBUG backend:attach_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="fe2068d0-b224-47ac-9a90-a321b23225ef"}: conmonrs::rpc: 238: Got a attach container request
2022-08-11T18:16:01.775826Z DEBUG backend:attach_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="fe2068d0-b224-47ac-9a90-a321b23225ef"}:promise: conmonrs::attach: 116: Creating attach socket: /var/lib/containers/storage/overlay-containers/a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5/userdata/attach
2022-08-11T18:16:01.775897Z DEBUG backend:attach_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="fe2068d0-b224-47ac-9a90-a321b23225ef"}:promise:attach: conmonrs::attach: 158: Start listening on attach socket
2022-08-11T18:16:01.775968Z DEBUG backend:attach_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="fe2068d0-b224-47ac-9a90-a321b23225ef"}:promise:attach: conmonrs::attach: 163: Got new attach stream connection
2022-08-11T18:16:02.402098Z DEBUG backend:create_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="1c4c62f4-c230-485f-895e-aee634fe43a6"}:promise:stdout: conmonrs::container_io: 238: Read 304 bytes
2022-08-11T18:16:02.402554Z DEBUG backend:attach_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="fe2068d0-b224-47ac-9a90-a321b23225ef"}:promise:attach:write_loop: conmonrs::attach: 254: Wrote stdout packet 0/1 to client
2022-08-11T18:16:02.402594Z DEBUG backend:attach_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="fe2068d0-b224-47ac-9a90-a321b23225ef"}:promise:attach:write_loop: conmonrs::attach: 254: Wrote stdout packet 1/1 to client
2022-08-11T18:16:02.402956Z DEBUG backend:attach_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="fe2068d0-b224-47ac-9a90-a321b23225ef"}:promise:attach:read_loop: conmonrs::attach: 203: Stopping read loop because no more data to read
2022-08-11T18:16:07.407616Z DEBUG backend:create_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="1c4c62f4-c230-485f-895e-aee634fe43a6"}:promise:stdout: conmonrs::container_io: 238: Read 304 bytes
2022-08-11T18:16:12.413399Z DEBUG backend:create_container{container_id="a186f94a8b5cd7953e3fe191a0bbf2dfbb8e4199c2dd3c1dd3a05c6ac44674f5" uuid="1c4c62f4-c230-485f-895e-aee634fe43a6"}:promise:stdout: conmonrs::container_io: 238: Read 304 bytes

Matt Heon and I believe there is a synchronization issue somewhere server side.

What did you expect to happen?

I expected the container to remain attached and continue outputting the top data

How can we reproduce it (as minimally and precisely as possible)?

The code I've been writing to integrate conmon-rs into Podman can be found in this PR: containers/podman#14930

Anything else we need to know?

This has been tried with prior versions of conmon-rs and didn't work either.

conmon-rs version

$ conmonrs --version
version: 0.1.0
tag: none
commit: 55285cb357adbbdf4ebd8d54076ca8db6da69682
build: 2022-08-11 18:26:23 +00:00
rustc 1.62.0 (a8314ef7d 2022-06-27)

OS version

# On Linux:
$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="36 (Workstation Edition)"
ID=fedora
VERSION_ID=36
VERSION_CODENAME=""
PLATFORM_ID="platform:f36"
PRETTY_NAME="Fedora Linux 36 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:36"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=36
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=36
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation
$ uname -a
Linux fedora 5.18.16-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 3 15:44:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Additional environment details (AWS, VirtualBox, physical, etc.)

$ podman version

Client: Podman Engine
Version: 4.2.0-dev
API Version: 4.2.0-dev
Go Version: go1.18.2
Git Commit: 1ada01a038fceaa9e94beb2de6e4593df03be7fa-dirty
Built: Tue Jun 7 12:48:55 2022
OS/Arch: linux/amd64

$ go version
go version go1.18.4 linux/amd64

$ rustup --version
rustup 1.24.3 (ce5817a94 2021-05-31)

$ rustc --version
rustc 1.62.0 (a8314ef7d 2022-06-27)

saschagrunert commented

Indeed, this looks like a bug on the conmon-rs side 🤔

haircommander commented

I started taking a look at this (we hit attach issues in cri-o too)

  • 👍1