x11vnc CPU usage on 100% and VNC Connecting takes several minutes on selenoid UI
milance1998 opened this issue · 5 comments
Hello,
I'm using selenoid and selenoid-ui on my Rocky Linux 9 server, everything looks OK, docker logs of selenoid, selenoid ui, selenoid/chrome container when started but when I create selenoid/chrome session it takes several minutes to display the session UI on selenoid UI
This is the view for 3-5 minutes:
and after that, after 3-5 minutes, session UI is displayed.
While waiting for the VNC connection, I inspected selenoid/chrome container and the CPU usage was on 100% by the x11vnc:
As soon as session UI is displayed, CPU usage drops to the normal (not 100%)
This is the case on Rocky Linux 9 server but not on Ubuntu, tried different selenoid configurations but wasn't successful..
Configuration I'm using now:
{
"chrome": {
"default": "127.0",
"versions": {
"127.0": {
"image": "selenoid/chrome:127.0",
"port": "4444",
"path": "/",
"tmpfs": {
"/tmp": "size=512m"
},
"env": ["ENABLE_VNC=true", "SCREEN_WIDTH=1920", "SCREEN_HEIGHT=1080", "SCREEN_DEPTH=24"],
"shmSize" : 268435456,
"mem": "512m",
"cpu": "4.0"
}
}
}
}Selenoid logs:
2024/09/04 12:31:19 [-] [NEW_REQUEST] [unknown] [192.168.0.144]
2024/09/04 12:31:19 [-] [NEW_REQUEST_ACCEPTED] [unknown] [192.168.0.144]
2024/09/04 12:31:19 [69] [LOCATING_SERVICE] [chrome] [127.0]
2024/09/04 12:31:19 [69] [USING_DOCKER] [chrome] [127.0]
2024/09/04 12:31:19 [69] [CREATING_CONTAINER] [selenoid/chrome:127.0]
2024/09/04 12:31:19 [69] [STARTING_CONTAINER] [selenoid/chrome:127.0] [7121d69ec5613dcbb8c777a7f6aed6cdfd96f291a073670c394c40476947fdb7]
2024/09/04 12:31:19 [69] [CONTAINER_STARTED] [selenoid/chrome:127.0] [7121d69ec5613dcbb8c777a7f6aed6cdfd96f291a073670c394c40476947fdb7] [0.14s]
2024/09/04 12:31:19 [69] [SERVICE_STARTED] [selenoid/chrome:127.0] [7121d69ec5613dcbb8c777a7f6aed6cdfd96f291a073670c394c40476947fdb7] [0.31s]
2024/09/04 12:31:19 [69] [PROXY_TO] [7121d69ec5613dcbb8c777a7f6aed6cdfd96f291a073670c394c40476947fdb7] [http://172.17.0.4:4444/]
2024/09/04 12:31:19 [69] [SESSION_ATTEMPTED] [http://172.17.0.4:4444/] [1]
2024/09/04 12:31:19 [69] [SESSION_CREATED] [a12187ec49043c3abe46b15cd624a705] [1] [0.84s]
2024/09/04 12:31:22 [72] [CONTAINER_LOGS] [7121d69ec5613dcbb8c777a7f6aed6cdfd96f291a073670c394c40476947fdb7]
2024/09/04 12:31:22 [73] [VNC_ENABLED] [a12187ec49043c3abe46b15cd624a705]
2024/09/04 12:53:06 [-] [NEW_REQUEST] [unknown] [192.168.0.144]
2024/09/04 12:53:06 [-] [NEW_REQUEST_ACCEPTED] [unknown] [192.168.0.144]
2024/09/04 12:53:06 [336] [LOCATING_SERVICE] [chrome] [127.0]
2024/09/04 12:53:06 [336] [USING_DOCKER] [chrome] [127.0]
2024/09/04 12:53:06 [336] [CREATING_CONTAINER] [selenoid/chrome:127.0]
2024/09/04 12:53:07 [336] [STARTING_CONTAINER] [selenoid/chrome:127.0] [ac6d632cc57f6f6e8c59a3c29f3784691a33bc7f70dcf9975ecb7aa2d9184b18]
2024/09/04 12:53:07 [336] [CONTAINER_STARTED] [selenoid/chrome:127.0] [ac6d632cc57f6f6e8c59a3c29f3784691a33bc7f70dcf9975ecb7aa2d9184b18] [0.19s]
2024/09/04 12:53:07 [336] [SERVICE_STARTED] [selenoid/chrome:127.0] [ac6d632cc57f6f6e8c59a3c29f3784691a33bc7f70dcf9975ecb7aa2d9184b18] [0.31s]
2024/09/04 12:53:07 [336] [PROXY_TO] [ac6d632cc57f6f6e8c59a3c29f3784691a33bc7f70dcf9975ecb7aa2d9184b18] [http://172.17.0.5:4444/]
2024/09/04 12:53:07 [336] [SESSION_ATTEMPTED] [http://172.17.0.5:4444/] [1]
2024/09/04 12:53:07 [336] [SESSION_CREATED] [b01d05103c20fd08eea5287b37517f30] [1] [0.90s]
2024/09/04 12:53:11 [339] [CONTAINER_LOGS] [ac6d632cc57f6f6e8c59a3c29f3784691a33bc7f70dcf9975ecb7aa2d9184b18]
2024/09/04 12:53:11 [340] [VNC_ENABLED] [b01d05103c20fd08eea5287b37517f30]
Some of selenoid-ui logs:
2024/09/04 12:26:16 [WS_PROXY] [/ws/logs/a3b0dead0b058a37d056f247997aa8b0] [ws://192.168.0.184:4445/logs/a3b0dead0b058a37d056f247997aa8b0]
2024/09/04 12:26:16 [WS_PROXY] [/ws/vnc/a3b0dead0b058a37d056f247997aa8b0] [ws://192.168.0.184:4445/vnc/a3b0dead0b058a37d056f247997aa8b0]
2024/09/04 12:30:32 [WS_PROXY] [/vnc/a3b0dead0b058a37d056f247997aa8b0] [CLOSED]
2024/09/04 12:30:32 [WS_PROXY] [/logs/a3b0dead0b058a37d056f247997aa8b0] [CLOSED]
2024/09/04 12:31:22 [WS_PROXY] [/ws/logs/a12187ec49043c3abe46b15cd624a705] [ws://192.168.0.184:4445/logs/a12187ec49043c3abe46b15cd624a705]
2024/09/04 12:31:22 [WS_PROXY] [/ws/vnc/a12187ec49043c3abe46b15cd624a705] [ws://192.168.0.184:4445/vnc/a12187ec49043c3abe46b15cd624a705]
2024/09/04 12:53:04 Client added. 4 registered clients
2024/09/04 12:53:11 [WS_PROXY] [/ws/logs/b01d05103c20fd08eea5287b37517f30] [ws://192.168.0.184:4445/logs/b01d05103c20fd08eea5287b37517f30]
2024/09/04 12:53:11 [WS_PROXY] [/ws/vnc/b01d05103c20fd08eea5287b37517f30] [ws://192.168.0.184:4445/vnc/b01d05103c20fd08eea5287b37517f30]
chrome container logs:
19
2024/09/04 12:31:19 [INIT] [Listening on :7070]
Waiting X server...
Logging to: /dev/null
Waiting X server...
Starting ChromeDriver 127.0.6533.72 (9755e24ca85aa18ffa16c743f660a3d914902775-refs/branch-heads/6533@{#1760}) on port 4444
All remote connections are allowed. Use an allowlist instead!
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
--- x11vnc loop: 1 ---
--- x11vnc loop: waiting for: 175
PORT=5900
@milance1998 this is a known CentOS \ RedHat specific issue related to their kernel \ security settings. Switching to another distribution should fix this.
@milance1998 this is a known CentOS \ RedHat specific issue related to their kernel \ security settings. Switching to another distribution should fix this.
@vania-pooh Thanks a lot, also asked this on rocky linux forum, got answer:
On RedHat-based distributions you should set LimitNOFILE=1048576 for containerd.service
Will investigate more that way, and if nothing - will switch to another distro.
Thanks for this thread - it helped me to fix the same issue on RHEL 9.
The issue lies at containerd/docker service configurations and was fixed back in 2023, however the most recent version of containerd for RedHat as of now still has the line that was removed by this pull request: https://github.com/containerd/containerd/pull/8924/files
So the solution is as simple as removing the LimitNOFILE=infinity line from /usr/lib/systemd/system/containerd.service
Additionally, it worth checking the docker.service file at the same location, as per https://github.com/moby/moby/pull/45534/files
It was good in my config, but who knows...
Thanks to this post, where the same issue happens with selenium-chrome image: https://bbs.archlinux.org/viewtopic.php?pid=2135855#p2135855
Just came across this and @igorscripco 's suggestion fixed my issue. Thanks!