[BUG] Integration tests for dashboards failing with the image/latest we are currently using
Divyaasm opened this issue · 12 comments
Is your feature request related to a problem? Please describe
Integtests dashboards jenkins workflow uses the docker image latest (v4 == v4.3 at the time of writing this)
This is failing for a few of the dashboard plugins with the following error
Sending interrupt signal to process
script returned exit code 143.
Sample build
Plugins failed:
opensearch-dashboards
alertingDashboards
anomalyDetectionDashboards
customImportMapDashboards
indexManagementDashboards
observabilityDashboards
securityAnalyticsDashboards
securityDashboards
The issue is seen with the new image which has been recently uploaded, but the the previous versions of the docker image are working fine. (previous version == 4.2)
Describe the solution you'd like
Debug the reason for the failing tests or revert the image latest to use the previous version for integ-testing
Describe alternatives you've considered
No response
Additional context
No response
Will take a look on whether the changes of the user and home cause this issue.
Using OSD tar on 2.12 7163 build on x64 as an example.
Using 4.3 image will keep stucking on this step of without-security test for dashboards-maps:
Running: plugins/custom-import-map-dashboards/documentsLayer.spec.js (2 of 4)
Documents layer
In 4.2 it passes with this:
Running: plugins/custom-import-map-dashboards/documentsLayer.spec.js (2 of 4)
Documents layer
✓ Add new documents layer with configuration (110069ms)
✓ Open saved map with documents layer
2 passing (2m)
Seems like issues with the chromium browser, or compatibility between Cypress 9.5.4 and Chromium version over 114:
% docker exec -it osd-rockylinux-4.2 chromium-browser --version
Chromium 114.0.5735.198 Fedora Project
% docker exec -it osd-rockylinux-4.3 chromium-browser --version
Chromium 118.0.5993.117 Fedora Project
The cypress test will stuck on the Chromium version > 114.
I even use the very old dockerfile that build 4.2 to build a new image here: https://github.com/opensearch-project/opensearch-build/blob/35048a49164337843efcf128a2b4245dc1559b8c/docker/ci/dockerfiles/current/test.rockylinux8.opensearch-dashboards.x64.arm64.dockerfile
And it still stuck with the same issue, version is:
% docker exec -it osd-rockylinux-4.2-rebuild chromium-browser --version
Chromium 120.0.6099.199 Fedora Project
The 4.2 Rebuild Logs are here:
opensearch_dashboards.log
opensearch.log
cypress.log
I switch back to electron and the test passes.
$ env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=false --browser electron --spec 'cypress/integration/plugins/custom-import-map-dashboards/*'
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 9.5.4 │
│ Browser: Electron 94 (headless) │
│ Node Version: v18.16.0 (/usr/share/opensearch/.nvm/versions/node/v18.16.0/bin/node) │
│ Specs: 4 found (plugins/custom-import-map-dashboards/add_saved_object.spec.js, plugin │
│ s/custom-import-map-dashboards/documentsLayer.spec.js, plugins/custom-import-m │
│ ap-dashboards/import_vector_map_tab.spec.js, plugins/custom-import-map-dashboa │
│ rds/opensearchMapLayer...) │
│ Searched: cypress/integration/plugins/custom-import-map-dashboards/* │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: plugins/custom-import-map-dashboards/add_saved_object.spec.js (1 of 4)
Add flights dataset saved object
✓ check if maps saved object of flights dataset can be found and open (76695ms)
1 passing (1m)
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 1 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 0 │
│ Video: true │
│ Duration: 1 minute, 20 seconds │
│ Spec Ran: plugins/custom-import-map-dashboards/add_saved_object.spec.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
(Video)
- Started processing: Compressing to 32 CRF
- Finished processing: /tmp/tmpjmuih98k/customImportMapDashboards/cypress/videos/p (8 seconds)
lugins/custom-import-map-dashboards/add_saved_object.spec.j
s.mp4
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: plugins/custom-import-map-dashboards/documentsLayer.spec.js (2 of 4)
Documents layer
✓ Add new documents layer with configuration (114497ms)
✓ Open saved map with documents layer (5851ms)
2 passing (2m)
There is no way to install chromium 114 now as the latest available on fedora project repo is already 120.
The 120 chromium will also cause this issue on security enabled test:
Running: plugins/custom-import-map-dashboards/documentsLayer.spec.js (2 of 4)
0 passing (2ms)
Warning: We failed processing this video.
This error will not alter the exit code.
TimeoutError: operation timed out
at afterTimeout (/usr/share/opensearch/.cache/Cypress/9.5.4/Cypress/resources/app/packages/server/node_modules/bluebird/js/release/timers.js:46:19)
at Timeout.timeoutTimeout [as _onTimeout] (/usr/share/opensearch/.cache/Cypress/9.5.4/Cypress/resources/app/packages/server/node_modules/bluebird/js/release/timers.js:76:13)
at listOnTimeout (node:internal/timers:557:17)
at processTimers (node:internal/timers:500:7)
Firefox issues: cypress-io/cypress#18919
yarn run v1.22.21
$ env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=false --browser firefox --spec 'cypress/integration/plugins/custom-import-map-dashboards/*'
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 9.5.4 │
│ Browser: Firefox 115 (headless) │
│ Node Version: v18.16.0 (/usr/share/opensearch/.nvm/versions/node/v18.16.0/bin/node) │
│ Specs: 4 found (plugins/custom-import-map-dashboards/add_saved_object.spec.js, plugin │
│ s/custom-import-map-dashboards/documentsLayer.spec.js, plugins/custom-import-m │
│ ap-dashboards/import_vector_map_tab.spec.js, plugins/custom-import-map-dashboa │
│ rds/opensearchMapLayer...) │
│ Searched: cypress/integration/plugins/custom-import-map-dashboards/* │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: plugins/custom-import-map-dashboards/add_saved_object.spec.js (1 of 4)
Add flights dataset saved object
Still waiting to connect to Firefox, retrying in 1 second (attempt 18/62)
Cypress failed to make a connection to Firefox.
This usually indicates there was a problem opening the Firefox browser.
Error: could not find CRI target
at lazyAssLogic (/usr/share/opensearch/.cache/Cypress/9.5.4/Cypress/resources/app/node_modules/lazy-ass/index.js:110:14)
at lazyAss (/usr/share/opensearch/.cache/Cypress/9.5.4/Cypress/resources/app/node_modules/lazy-ass/index.js:115:28)
at findStartPage (/usr/share/opensearch/.cache/Cypress/9.5.4/Cypress/resources/app/packages/server/lib/browsers/protocol.js:56:28)
at runMicrotasks (<anonymous>:null:null)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
~
Chrome 121 show the same issue as chromium120.
Approaches:
-
Upgrade Cypress from 9.5.4 to latest version (Long Term)
- Pros:
- Resolve the issue from the root cause
- Update to latest Cypress for more features and functions
- Cons:
- Significant amount of work on both the core changes and all plugins to support it, already tried by Craig and Security Team contractor for several times, both stale: opensearch-project/opensearch-dashboards-functional-test#408
- Will not be done for 2.12.0 release
- Pros:
-
Change FTrepo to use Electron instead of Chromium
- Pros:
- Quick Fix
- Proves working
- Cons:
- Directly change FTrepo which cause all users of FTrepo force to change from Chromium to Electron
- Have issues with Electron not showing all the issues that Chromium shows
- Pros:
-
(Recommended) Change opensearch-build to define
BROWSER_PATH=electronenv vars to override FTrepo settings- Pros:
- Quick Fix
- Proves working
- Will not change FTrepo and only change opensearch-build related workflows
- Cons:
- Have issues with Electron not showing all the issues that Chromium shows
- Pros:
-
Change to other browsers apart from Electron/Chromium
- Chrome has the same error as Chromium
- Firefox have issues being detected by Cypress 9.5.4
- Edge is basically using the Chromium core
@kavilla @joshuarrrr @ashwin-pc @seraphjiang @AMoo-Miki We are thinking of using electron for 2.12.0 instead of Chromium due to the issues mentioned above. Can you provide your inputs?