rustdoc-gui tests sporadically failing
ehuss opened this issue · 18 comments
The src/test/rustdoc-gui tests have been sporadically failing on CI:
These may have started with #93597
cc @GuillaumeGomez
There are various different errors:
/checkout/src/test/rustdoc-gui/search-tab-selection-if-current-is-empty.goml search-tab-selection-if-current-is-empty... FAILED
[ERROR] (line 6) TimeoutError: waiting for selector "#titles" failed: timeout 30000ms exceeded: for command `wait-for: "#titles"`
/checkout/src/test/rustdoc-gui/search-result-display.goml search-result-display... FAILED
[ERROR] (line 6) TimeoutError: waiting for selector "#titles" failed: timeout 30000ms exceeded: for command `wait-for: "#titles"`
/checkout/src/test/rustdoc-gui/src-font-size.goml An exception occured: read ECONNRESET
== STACKTRACE ==
Error
at innerRunTestCode (/node-v14.4.0-linux-x64/lib/node_modules/browser-ui-test/src/index.js:468:16)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async main (/checkout/src/tools/rustdoc-gui/tester.js:226:13)
/checkout/src/test/rustdoc-gui/mobile.goml An exception occured: Failed to launch the browser process!
Inconsistency detected by ld.so: dl-tls.c: 493: _dl_allocate_tls_init: Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed!
TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md
== STACKTRACE ==
Error
Error
at innerRunTestCode (/node-v14.4.0-linux-x64/lib/node_modules/browser-ui-test/src/index.js:468:16)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
That is concerning... If it happens too many times, let's revert it.
There is also #94417.
It looks like rustdoc-gui tests are now consistently failing with:
2022-03-09T18:18:21.7017663Z /checkout/src/test/rustdoc-gui/mobile.goml mobile... FAILED
2022-03-09T18:18:21.7018773Z [ERROR] (line 27) Error: Evaluation failed: assert didn't fail: for commandcompare-elements-position-near-false: ("#preferred-light-theme .setting-name", "#preferred-light-theme .choice", {"y": 16})
I'm checking locally.
@ehuss Is it still the case?
It seems to still be happening regularly:
2021-11-23T17:42:34 https://github.com/rust-lang-ci/rust/actions/runs/1495820343: [ERROR] (line 4) TimeoutError: waiting for selector
2021-12-14T00:47:00 https://github.com/rust-lang-ci/rust/actions/runs/1575507908: [ERROR] (line 17) TimeoutError: waiting for selector
2022-01-22T19:25:41 https://github.com/rust-lang-ci/rust/actions/runs/1733757860: /checkout/src/test/rustdoc-gui/search-result-description.goml An exception occured: Failed to launch the browser process!
2022-01-26T19:44:04 https://github.com/rust-lang-ci/rust/actions/runs/1752548375: /checkout/src/test/rustdoc-gui/impl-default-expansion.goml An exception occured: Failed to launch the browser process!
2022-01-31T08:11:10 https://github.com/rust-lang-ci/rust/actions/runs/1771336059: [ERROR] (line 6) TimeoutError: waiting for selector
2022-02-04T05:46:21 https://github.com/rust-lang-ci/rust/actions/runs/1793315545: /checkout/src/test/rustdoc-gui/trait-sidebar-item-order.goml An exception occured: Failed to launch the browser process!
2022-02-05T22:03:59 https://github.com/rust-lang-ci/rust/actions/runs/1800584018: /checkout/src/test/rustdoc-gui/src-font-size.goml An exception occured: read ECONNRESET
2022-02-07T19:15:06 https://github.com/rust-lang-ci/rust/actions/runs/1808171078: [ERROR] (line 6) TimeoutError: waiting for selector
2022-02-08T21:52:54 https://github.com/rust-lang-ci/rust/actions/runs/1814714051: [ERROR] (line 6) TimeoutError: waiting for selector
2022-02-08T22:32:08 https://github.com/rust-lang-ci/rust/actions/runs/1814879899: /checkout/src/test/rustdoc-gui/implementors.goml An exception occured: Failed to launch the browser process!
2022-02-14T11:19:42 https://github.com/rust-lang-ci/rust/actions/runs/1840526472: /checkout/src/test/rustdoc-gui/search-filter.goml An exception occured: Failed to launch the browser process!
2022-02-21T07:21:35 https://github.com/rust-lang-ci/rust/actions/runs/1874627943: [ERROR] (line 6) TimeoutError: waiting for selector
2022-02-22T02:47:16 https://github.com/rust-lang-ci/rust/actions/runs/1879224701: /checkout/src/test/rustdoc-gui/search-result-description.goml An exception occured: Failed to launch the browser process!
2022-03-05T14:31:20 https://github.com/rust-lang-ci/rust/actions/runs/1938283620: /checkout/src/test/rustdoc-gui/toggled-open-implementations.goml An exception occured: Failed to launch the browser process!
2022-03-07T21:18:49 https://github.com/rust-lang-ci/rust/actions/runs/1947870857: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-09T06:18:46 https://github.com/rust-lang-ci/rust/actions/runs/1955562234: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-09T18:18:21 https://github.com/rust-lang-ci/rust/actions/runs/1958791332: /checkout/src/test/rustdoc-gui/toggle-docs-mobile.goml An exception occured: Failed to launch the browser process!
2022-03-09T18:49:35 https://github.com/rust-lang-ci/rust/actions/runs/1958984691: /checkout/src/test/rustdoc-gui/jump-to-def-background.goml An exception occured: Failed to launch the browser process!
2022-03-10T05:52:57 https://github.com/rust-lang-ci/rust/actions/runs/1961326088: /checkout/src/test/rustdoc-gui/toggle-docs.goml An exception occured: Failed to launch the browser process!
2022-03-20T08:34:38 https://github.com/rust-lang-ci/rust/actions/runs/2011183164: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-21T18:32:43 https://github.com/rust-lang-ci/rust/actions/runs/2017851280: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-24T09:11:02 https://github.com/rust-lang-ci/rust/actions/runs/2033118524: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-27T12:24:59 https://github.com/rust-lang-ci/rust/actions/runs/2047654016: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-27T15:02:11 https://github.com/rust-lang-ci/rust/actions/runs/2048045097: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-28T05:47:21 https://github.com/rust-lang-ci/rust/actions/runs/2050300242: [ERROR] (line 6) TimeoutError: waiting for selector
The situation seems to have been much better since #95013: there is just one error to fix now. \o/
@GuillaumeGomez Just FYI, there's still timeouts happening after #95470:
/checkout/src/test/rustdoc-gui/search-filter.goml search-filter... FAILED
Error: (line 6) TimeoutError: waiting for selector "#titles" failed: timeout 30000ms exceeded: for command `wait-for: "#titles"`
https://github.com/rust-lang-ci/rust/actions/runs/2086146075
Thanks! Then I'll need for another way to wait for the search results.
Since #97139, I think there are no more spurious failures? I think the bug came from puppeteer waitFor function. I removed my usage of it in GuillaumeGomez/browser-UI-test#310 when I was working on something else. I'll close the issue and if a spurious failure appears again, please re-open it.
Nevermind, seems like new bugs appeared recently. I'll take another look.
A new spurious failure appeared in #102744.
Another flaky GUI test #104647 (comment)
That was
/checkout/src/test/rustdoc-gui/code-sidebar-toggle.goml code-sidebar-toggle... FAILED
It seems to be mostly:
/checkout/src/test/rustdoc-gui/basic-code.goml basic-code... FAILED
Error: (line 3) Error: Execution context was destroyed, most likely because of a navigation.: for command `assert-count: (".src-line-numbers", 1)`
So about "execution context was destroyed": it seems that when puppeteer gets a element from the DOM, it can get destroyed in the meantime (for whatever reason), resulting in the current issue. A solution would be to instead move everything directly into the browser context. I'll check that in the next weeks to see if it improves the situation.
Same spurious failure in #106315. So my fix didn't solve this issue unfortunately...