UnknownError: session deleted because of page crash from tab crashed
elgalu opened this issue · 58 comments
Chrome crashes on docker-selenium instances (works ok with real machines) on certain high GPU intensive UI tests
{code}
UnknownError: unknown error: session deleted because of page crash
from tab crashed
(Session info: chrome=43.0.2357.81)
(Driver info: chromedriver=2.15.322448 (52179c1b310fec1797c81ea9a20326839860b7d3),platform=Linux 3.13.0-32-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 53 milliseconds
Build info: version: '2.45.0', revision: '5017cb8', time: '2015-02-26 23:59:50'
System info: host: 'zelenium', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-32-generic', java.version: '1.7.0_80'
Session ID: b12eb3c43351dad58746798c40078ef9
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities [{platform=LINUX, acceptSslCerts=true, javascriptEnabled=true, browserName=chrome, chrome={userDataDir=/tmp/.com.google.Chrome.11yHIh}, rotatable=false, locationContextEnabled=true, mobileEmulationEnabled=false, version=43.0.2357.81, takesHeapSnapshot=true, cssSelectorsEnabled=true, databaseEnabled=false, handlesAlerts=true, browserConnectionEnabled=false, webStorageEnabled=true, nativeEvents=true, applicationCacheEnabled=false, takesScreenshot=true}]
at IncomingMessage.emit (events.js:129:20)
at _stream_readable.js:908:16
at process._tickCallback (node.js:355:11)
==== async task ====
WebDriver.executeScript()
.....
{code}
Works ok in Firefox.
Note: playing with certain Chrome options like --no-sandbox didn't help
capabilities: {
browserName: 'chrome',
chromeOptions: {
args: [
'--disable-gpu',
'--disable-impl-side-painting',
'--disable-gpu-sandbox',
'--disable-accelerated-2d-canvas',
'--disable-accelerated-jpeg-decoding',
'--no-sandbox',
'--test-type=ui',
],
},
},
Same here, using selenium/standalone-chrome:2.46.0
Same here. I have been trying many docker images built for running selenium+protractor +xvfb (includes one of @elgalu ;) ). Look like this is dues to docker. Everything ok on real machine or vagrant+virtualbox.
I had to switch to firefox, and it does not happen anymore.
Yes, Firefox works ok fortunately.
I guess Chrome guys would be able to fix the problem if someone gets together a sample project that reproduces the problem and host it somewhere.
When I run my suite without test cases that download files, I don't see this error. Has anyone else tried this?
Chrome 42 - 44 crashes when manually open http://www.google.com/adwords
Is there issue already opened for Chrome browser in docker. Should I submit a new one for Chromium?
We are running into similar issue with our application same as with http://www.google.com/adwords
We use own docker configuration similar to selenium/node-chrome-debug
Steps to reproduce manually on public selenium image:
docker run -d -p 4444:4444 --name selenium-hub selenium/hub
docker run -it -p 5900:5900 --link selenium-hub:hub selenium/node-chrome-debug
start vnc viewver on default docker IP 127.0.0.1:5900
or for boot2docker IP at 192.168.69.103:5900
(default password is 'secret')
Right mouse click. Open Applications\Terminal Emulators\XTerm
run following commands
sudo su - seluser
google-chrome
In google chrome window type
http://www.google.com/adwords
It loads page and then shows: Aw, Snap!
It is reproducible in all Chrome browsers starting 42 (till latest 44)
Currently on Chrome/44.0.2403.155
It is not reproducible on the our old docker-selenium image with older Chrome/41.0.2272.89
Please refer correct duplicate or let me know where I can submit this issue?
I can provide additional details, logs if needed?
Best chance for this to be fixed here:
https://code.google.com/p/chromium/issues/detail?id=519952
FYI, I have found that recent version(s) of Chrome seem to crash in Docker containers, on certain pages, due to too small /dev/shm. May or may not be relevant to this issue.
OMG @issuj you figured this out, I just tried below code and Chrome no longer crashes:
Simpy mount -v /dev/shm:/dev/shm
Or, longer, create a big shm
- Started in privileged mode:
docker run --privileged - Fix small /dev/shm size
docker exec $id sudo umount /dev/shm
docker exec $id sudo mount -t tmpfs -o rw,nosuid,nodev,noexec,relatime,size=512M tmpfs /dev/shmHowever it would be nice to avoid privileged mode.
Any ideas @stonemaster @rongearley @jvermillard ?
Other refs:
moby/moby#3505 (comment)
moby/moby#4981 (comment)
moby/moby#2606 (comment)
We were having the same issue, fixed by remounting /dev/shm with bigger size.
Awesome! It works for us!
Quoting @kkochubey1 solution:
It also works if use -v /dev/shm:/dev/shm option to share host /dev/shm
In this case you do not need --privileged mode.
Does anyone know how to connect /dev/shm using docker-compose. Couldn't find anything in the reference
volumes:
- /dev/shm:/dev/shm
^doesn't work on mac.
@afthar It works like a charm on my mac machine. Docker version 1.8.1, build d12ea79
seleniumnode:
image: selenium/standalone-chrome:2.47.1
ports:
- "4444:4444"
volumes:
- /dev/shm:/dev/shm
@arkaitzgarro works for me as well.
UPD: works in local docker environment. If your CI does not allow sharing /dev/shm - this will not help.
@afthar Works for me on a Mac with docker 1.9.1, like @arkaitzgarro mentioned.
Is there a way to do this in Kubernetes on GCP?
How big should it be (64M is the default it seems)? Going to try using --shm-size with docker build.
I got the same error even if my container has
volumes:
- "/dev/shm:/dev/shm"
more specifically
hub2:
image: selenium/standalone-chrome:2.45.0
volumes:
- "/dev/shm:/dev/shm"
I was having this same issue on Ubuntu (digitalocean) without Docker. To fix it I added this line to /etc/fstab: tmpfs /dev/shm tmpfs defaults,size=256m 0 0
check the size with: df -k | grep shm
then remounted with: mount -o remount tmpfs (or reboot)
Good point @qualityshepherd ; sharing /dev/shm:/dev/shm won't help if the host shm is tiny in the 1st place. Will reopen as "check if shm is too small then fail docker run"
Oh, and it looks like they nixed /dev/shm in favor of /run/shm in Ubuntu 11.10. So I can fix the issue by just soft linking thusly: sudo ln -s /run/shm /dev/shm
Nope.
With latest docker is now possible to set the shm size via docker run option --shm-size=2g where 2g is 2GB but can be changed:
https://docs.docker.com/engine/reference/run/
I have tried expanding the size of /dev/shm but still it is not working for me :( I'm trying to run headless test with selenium ruby bindings and chrome drivers on Digital Ocean Ubuntu 14.04. Chrome driver version is 2.29 and and google-chrome version is 57.0
But whener I try to start a new session with these commands
headless=Headless.new
headless.start
driver = Selenium::WebDriver.for :chrome
Net::ReadTimeout: Net::ReadTimeout
I tried the same on a docker container on my local machine and still I am facing the same issue.
I can run it successfully on my local machine(without using docker container) and AWS server as well. Please help me to fix this :(
How do I specify:
volumes:
- /dev/shm:/dev/shm
If I'm using a kubernetes yaml file to spin up the container?
Hi @amirplanet,
Can you please share your yaml file? We need to double check because we are not really sure how to do that. We might need to research a bit together.
Perhaps you can create a new issue with this? It would be easier to track it. There is already a lot of info in this one.
@diemol Thanks a lot for replying. Here is the current yaml file that works, but the node crashes for the same reason this ticket describes:
https://gist.github.com/amirplanet/79b06ab0e1b2f942cc90eb7dd99408ff
Any help on how to specify the shm would be extremely helpful.
For others using docker-compose, you can add the following line to docker-compose.yml:
shm_size: 128M
This fixed it for me. From https://docs.docker.com/compose/compose-file/#domainname-hostname-ipc-mac_address-privileged-read_only-shm_size-stdin_open-tty-user-working_dir
@dhobbs Where specifically did you add it to in the YML? I tried adding it to the hub definition like this but it didn't work. In the mean time I'm iterating on it, increasing the size if the test continues to fail on Jenkins in the same manner.
hub:
image: selenium/standalone-chrome-debug:3.4.0-einsteinium
volumes:
- /dev/shm:/dev/shm
ports:
- "4444"
- "38471:5900"
shm_size: 128M
@anjunatl That location in the yml looks fine. Did you add it to your nodes too? For my use case, I've had to use a couple gigs as well.
@seangerhardt-wf Mybad, hub is a misnomer now. I removed the hub/node architecture to limit the debug space when I first encountered this problem - that container is the standalone chrome container.
I'm still seeing failures with shm_size: 1G so I think I may need to try the tmpfs trick from this previous comment
With our test infrastructure, I think I'll need it ideally to be configured through the docker-compose.yml file or as a script that gets run in the container though - Maybe through entrypoint or command but it looks like those override the default, and I'm pretty sure that's not what I want to do. It looks like tmpfs is a docker-compose command, but I don't know how to specify the size for it from the sparse docs there.
I'll try this and see if it works:
bbt-chrome:
image: selenium/standalone-chrome-debug:3.4.0-einsteinium
ports:
- "4444"
- "38471:5900"
shm_size: 1G
tmpfs: /dev/shm:rw,nosuid,nodev,noexec,relatime,size=1G
I couldn't get my last code snippet to work, I got ERROR: for bbt-chrome Cannot start service bbt-chrome: linux mounts: Duplicate mount point '/dev/shm'. Got the same error when I tried adding the volume mount info back into the definition. So maybe I can't do that from the docker-compose side of things, is there any way to add additional commands to docker-compose instead of replacing them?
You have 3 options to do this:
1. Volumes
volumes:
- /dev/shm:/dev/shm
2. shm_size
shm_size: 1G
3. tmpfs
All 3 all mutually exclusive.
So you pick just 1, not all.
I prefer the first one however if you are running dind (docker in docker) the first option will only work if the shm device was shared from the host, upstream=>downstream. And if that shm is big enough in the host.
@elgalu I used 1 to solve it the first time I encountered this. As our teams added more tests, I believe we grew past what our Jenkins' docker node instance's /dev/shm had to offer. I'm also fairly sure the Jenkins nodes are virtualized but it's not DinD.
I'm not noticing the browser page go gray anymore, so I have a feeling my problem might be more with protractor and/or selenium itself than the docker container supporting Chrome.
Thanks for your help!
docker run --privileged --rm -ti --name zalenium -p 4444:4444 -p 5555:5555 -v /dev/shm:/dev/shm -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/videos:/home/seluser/videos dosel/zalenium start --chromeContainers 5
using the above to start the docker instance.
when I run in parallel it is not working. Getting page crash issue
can you please guide on how to solve this issue
@hemanthsridhar try the one-liner to start Zalenium:
curl -sSL https://raw.githubusercontent.com/dosel/t/i/p | bash -s start -i --chromeContainers 5
Should take care of dealing with the shm for you.
What's the shm size of the host?
@elgalu i dont know. I just did whatever you asked me to do. fyi the curl command. how to check the shm size?
my system :
OS : Mac
Ram : 16 GB
Sorry I don't know what's wrong with your environment, perhaps someone else can help.
Much of the answer I see here is related to Docker Compose. How would you do it in K8s using YAML file ? The only option I have and can do is using tmpfs. This is what I tried and still the issue remains.
volumeMounts:
- mountPath: /selenium-node-chrome
name: selenium-node-chrome
volumes:
- name: selenium-node-chrome
emptyDir:
medium: Memory
Error -
org.openqa.selenium.WebDriverException: unknown error: session deleted because of page crash from unknown error: cannot determine loading status from tab crashed (Session info: chrome=53.0.2785.143) (Driver info: chromedriver=2.24.417424 (c5c5ea873213ee72e3d0929b47482681555340c3),platform=Linux 3.10.0-514.6.1.el7.x86_64 x86_64) (WARNING: The server did not provide any stacktrace information) Command duration or timeout: 996 milliseconds Build info: version: '2.53.1',
@keshavneupane - Yup, I'm also facing the same issue without any success.
What you mention is the current workaround
https://docs.openshift.org/latest/dev_guide/shared_memory.html
Since it is not really supported yet kubernetes/kubernetes#28272
I have tried that workaround and it works for me, but I also need to leave the request limit at least with 2Gi, maybe try with 1Gi first and see how it behaves.
I also experience this issue when running tests in gitlab CI:
There is another way to make it work. In the browser capabilities, add --disable-dev-shm-usage. This will force Chrome to use the /tmp directory instead. This may slow down the execution though since disk will be used instead of memory. The flag is available as of Chrome 65.
Saved my life! Just shm_size: 2G in my docker-compose file did the trick. 💯
Should this not be resolved within the base container definition?
Is there a way to do this in Kubernetes on GCP?
spec:
volumes:
- name: shm-vol
hostPath:
path: /dev/shm
containers:
volumeMounts:
- name: shm-vol
mountPath: /dev/shm
Hi. I'm still having this issue on some of the pods spawned by the zalenium. Where should I set this "/dev/shm:/dev/shm" in my yaml when deploying zalenium to K8S?
@baflQA, this should work https://stackoverflow.com/questions/46085748/define-size-for-dev-shm-on-container-engine/46434614#46434614
Not sure if k8s has a different way to do it these days.
Just to add my few cents about this issue.
None of the solutions given on the internet worked for me. Including increasing /dev/shm.
The problem in my case was a memory leak in the application that wasn't discovered by real human or browser but was mainly caused by Mink driver.
I had in my HTML a following piece of code
<input type="text" name="content" value="{{ a long HTML template string here }}" />When this view was tested, Mink tried to execute a step I follow "Close", which was searching the DOM to find Close button. But because the long HTML template was there, I assume Mink could not handle HTML in HTML and it caused memory leak which in result crashed the chrome driver with the error.
To resolve this, I had to change the way I store the HTML template (via ajax, not form) and no more crashing.
Not sure if k8s has a different way to do it these days.
Using this stackoverflow solution on Kubernetes, worked like a charm
I encountered this issue when I calling the executeScript method to try to get an object from the web page and I'm not using selenium in docker. I found a workaround for this, that is calling JSON.stringify() in the web page before returning the object. Something like below.
const dataStr = await browser.driver.executeScript('return JSON.stringify(obj)');
// Parse the JSON string.
const data = JSON.parse(dataStr);OMG @issuj you figured this out, I just tried below code and Chrome no longer crashes:
Simpy mount
-v /dev/shm:/dev/shmOr, longer, create a big shm
- Started in privileged mode:
docker run --privileged- Fix small /dev/shm size
docker exec $id sudo umount /dev/shm docker exec $id sudo mount -t tmpfs -o rw,nosuid,nodev,noexec,relatime,size=512M tmpfs /dev/shmHowever it would be nice to avoid privileged mode.
Any ideas @stonemaster @rongearley @jvermillard ?Other refs:
moby/moby#3505 (comment)
moby/moby#4981 (comment)
moby/moby#2606 (comment)
How do I run these commands? and what is $id in this case?
how about windows ?