False positive (zero exit code) returned when versions mismatch

Question

False positive (zero exit code) returned when versions mismatch

yagudaev opened this issue 4 years ago · 7 comments

We had an incident on our CI when we cleared cache on buildkite and got a version mismatch between cypress and npm.

The error we saw looks like this:

Log looked like this:

+ npm bin
--
  | + /app/node_modules/.bin/knapsack-pro-cypress --spec cypress/integration/**/*.spec.js --reporter junit --reporter-options mochaFile=artifacts/cypress/results-[hash].xml
  |  
  | 2020-09-24T23:05:26.483Z [@knapsack-pro/core] info: POST https://api.knapsackpro.com/v1/queues/queue
  | DEPRECATION WARNING: `preview_path` will be removed in v3.0.0. Use `preview_paths` instead. (called from <top (required)> at /app/config/environment.rb:5)
  | WARNING: Stimulus Reflex requires caching to be enabled. Caching allows the session to be modified during ActionCable requests.
  | To enable caching in development, run:
  | rails dev:cache
  | DEPRECATION WARNING: Initialization autoloaded the constants API, API::Auth, API::BaseController, API::GraphqlController, Search::Query, Middleware, Middleware::VersionHeader, ActionText::ContentHelper, and ActionText::TagHelper.
  |  
  | Being able to do this is deprecated. Autoloading during initialization is going
  | to be an error condition in future versions of Rails.
  |  
  | Reloading does not reboot the application, and therefore code executed during
  | initialization does not run again. So, if you reload API, for example,
  | the expected changes won't be reflected in that stale Module object.
  |  
  | These autoloaded constants have been unloaded.
  |  
  | Please, check the "Autoloading and Reloading Constants" guide for solutions.
  | (called from <top (required)> at /app/config/environment.rb:5)
  |  
  | 2020-09-24T23:05:27.197Z [@knapsack-pro/core] info: 200 OK
  |  
  | Request ID:
  | 68593e37-feba-46e5-b71d-20ec94d029ed
  |  
  | Response body:
  | {
  | queue_name: '1335:0e2799c3a307ced616ceaf2a6dbceb6e',
  | build_subset_id: null,
  | test_files: [
  | {
  | path: 'cypress/integration/seller/spa_customer/view-customers.spec.js',
  | time_execution: 70.619
  | }
  | ]
  | }
  | It looks like this is your first time using Cypress: 5.2.0
  |  
  | ✖  Verifying Cypress can run /root/.cache/Cypress/5.2.0/Cypress
  |  
  | (node:3551) UnhandledPromiseRejectionWarning: Error: Cypress failed to start.
  |  
  | This is usually caused by a missing library or dependency.
  |  
  | The error below should indicate which dependency is missing.
  |  
  | https://on.cypress.io/required-dependencies
  |  
  | If you are using Docker, we provide containers with all required dependencies installed.
  |  
  | ----------
  |  
  | /root/.cache/Cypress/5.2.0/Cypress/Cypress: error while loading shared libraries: libgbm.so.1: cannot open shared object file: No such file or directory
  |  
  | ----------
  |  
  | Platform: linux (Debian - 10)
  | Cypress Version: 5.2.0
  | at /app/node_modules/@knapsack-pro/cypress/node_modules/cypress/lib/errors.js:393:15
  | at tryCatcher (/app/node_modules/bluebird/js/release/util.js:16:23)
  | at Promise._settlePromiseFromHandler (/app/node_modules/bluebird/js/release/promise.js:547:31)
  | at Promise._settlePromise (/app/node_modules/bluebird/js/release/promise.js:604:18)
  | at Promise._settlePromise0 (/app/node_modules/bluebird/js/release/promise.js:649:10)
  | at Promise._settlePromises (/app/node_modules/bluebird/js/release/promise.js:729:18)
  | at Promise._fulfill (/app/node_modules/bluebird/js/release/promise.js:673:18)
  | at Promise._resolveCallback (/app/node_modules/bluebird/js/release/promise.js:466:57)
  | at Promise._settlePromiseFromHandler (/app/node_modules/bluebird/js/release/promise.js:559:17)
  | at Promise._settlePromise (/app/node_modules/bluebird/js/release/promise.js:604:18)
  | at Promise._settlePromise0 (/app/node_modules/bluebird/js/release/promise.js:649:10)
  | at Promise._settlePromises (/app/node_modules/bluebird/js/release/promise.js:729:18)
  | at Promise._fulfill (/app/node_modules/bluebird/js/release/promise.js:673:18)
  | at Promise._resolveCallback (/app/node_modules/bluebird/js/release/promise.js:466:57)
  | at Promise._settlePromiseFromHandler (/app/node_modules/bluebird/js/release/promise.js:559:17)
  | at Promise._settlePromise (/app/node_modules/bluebird/js/release/promise.js:604:18)
  | (node:3551) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
  | (node:3551) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
  | + TESTS_EXIT_STATUS=0

Our configuration looked like this:

set -x

export RAILS_ENV=test \
       KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \
       RAILS_ENABLE_TEST_LOG=true \
       CYPRESS=true \
       UV_THREADPOOL_SIZE=1024; # Option for node, helps with local DNS requests and cypress

# run migrations
bundle exec rake db:create
bundle exec rake db:migrate

# Run puma in background
bundle exec bin/puma -p 3001 -e test &

# Run knapsack-pro-cypress with junit reporter
$(npm bin)/knapsack-pro-cypress --spec "cypress/integration/**/*.spec.js" \
            --reporter "junit" \
            --reporter-options "mochaFile=artifacts/cypress/results-[hash].xml"

# Save knapsack-pro-cypress exit status
TESTS_EXIT_STATUS=$?

# Upload artifacts
buildkite-agent artifact upload "artifacts/**/*.xml"
buildkite-agent artifact upload "cypress/screenshots/**/*"
buildkite-agent artifact upload "cypress/videos/**/*"
buildkite-agent artifact upload "log/*"

# return exit status of cypress
exit $TESTS_EXIT_STATUS

The line where we execute knapsack should actually be using yarn instead of npm. So I think this had a lot to do with it.

We were also using the version of knapsack pro before you introduced the peer dependency of cypress (#32).

The problem to investigate is why did knapsack pro return a 0 exit code where clearly there is a failure? Is it a problem with the knapsack runner? Or is it a problem with cypress not catching uncaught exceptions?

If it is knapsack, my best guess is that that the queue mode simply didn't see any jobs and didn't take into account this particular error scenario. Thus it reported it as a success.

Answer 1 · 2020-09-25T17:13:13.000Z

This is how we get info from Cypress if tests are green or red:

https://github.com/KnapsackPro/knapsack-pro-cypress/blob/8942e0430e9b529ab27cf877b15b2d2964f89222/src/knapsack-pro-cypress.ts#L37,L61

const onSuccess: onQueueSuccessType = async (queueTestFiles: TestFile[]) => {
  const testFilePaths: string[] = queueTestFiles.map(
    (testFile: TestFile) => testFile.path
  );
  const { runs: tests, totalFailed } = await cypress.run({
    ...cypressCLIOptions,
    spec: testFilePaths,
  });

  // when Cypress crashed
  if (typeof tests === 'undefined') {
    return {
      recordedTestFiles: [],
      isTestSuiteGreen: false,
    };
  }

  const recordedTestFiles: TestFile[] = tests.map((test: any) => ({
    path: test.spec.relative,
    time_execution:
      // test.stats.wallClockDuration - Cypress 3.x and 4.x
      // test.stats.duration - Cypress 5.x
      (test.stats.wallClockDuration || test.stats.duration) / 1000,
  }));

  return {
    recordedTestFiles,
    isTestSuiteGreen: totalFailed === 0,
  };
};

When Cypress cashes we should detect it:

  // when Cypress crashed
  if (typeof tests === 'undefined') {
    return {
      recordedTestFiles: [],
      isTestSuiteGreen: false,
    };
  }

From your error:

(node:3551) UnhandledPromiseRejectionWarning: Error: Cypress failed to start.

I'm guessing that Cypress was never started and never return anything to Knapsack. What's weird the process was totally stopped because I don't see any more logs rather than backtrace. Maybe this escaped from the process so knapsack could not have a chance to catch that something failed.

How can I reproduce this error? Then maybe I could figure out how to catch the error and make sure knapsack pro sets excite code 1 (to mark CI build as failed).
You mentioned yarn. You can replace npm with yarn and it should work just fine I believe. You can use npm or yarn to install the @knapsack-pro/cypress package.
I noticed --spec "cypress/integration/**/*.spec.js" in:

# Run knapsack-pro-cypress with junit reporter
$(npm bin)/knapsack-pro-cypress --spec "cypress/integration/**/*.spec.js" \
            --reporter "junit" \
            --reporter-options "mochaFile=artifacts/cypress/results-[hash].xml"

It will be ignored. You need to tell Knapsack Pro what tests should be run instead of telling Cypress because Knapsack Pro decides what tests should be run. See https://knapsackpro.com/faq/question/how-to-run-tests-only-from-specific-directory-in-cypress

Answer 2 · 2020-09-25T22:38:04.000Z

Yeah, I switched us to using yarn bin since. I thought that would solve it.

Interestingly enough I found another error when I was trying to upgrade, again false positive but for a different reason. We didn't have a library installed for Cypress 5.2.0 on the CI.

Answer 3 · 2020-09-25T23:05:41.000Z

Cypress has a new dependency now, libgmb-dev. So if you don't have that in your docker image it should still return with a 0 code

It looks like this is your first time using Cypress: 5.2.0
--
  |  
  | ✖  Verifying Cypress can run /root/.cache/Cypress/5.2.0/Cypress
  |  
  | (node:1057) UnhandledPromiseRejectionWarning: Error: Cypress failed to start.
  |  
  | This is usually caused by a missing library or dependency.
  |  
  | The error below should indicate which dependency is missing.
  |  
  | https://on.cypress.io/required-dependencies
  |  
  | If you are using Docker, we provide containers with all required dependencies installed.
  |  
  | ----------
  |  
  | /root/.cache/Cypress/5.2.0/Cypress/Cypress: error while loading shared libraries: libgbm.so.1: cannot open shared object file: No such file or directory
  |  
  | ----------
  |  
  | Platform: linux (Debian - 10)

Answer 4 · 2020-09-28T16:47:33.000Z

I suspect the problem is that cypress.run is running in an async way and when it fails then error won't break the main process.

  const { runs: tests, totalFailed } = await cypress.run({
    ...cypressCLIOptions,
    spec: testFilePaths,
  });

I will try to catch an exception and fail the main process. Maybe this would help.

try {
  const { runs: tests, totalFailed } = await cypress.run({
    ...cypressCLIOptions,
    spec: testFilePaths,
  });
} catch (e) {
  // fail here when cypress.run fails. Maybe this will help
}

Answer 5 · 2020-09-29T21:10:06.000Z

You are probably right @ArturT I'll raise an issue with Cypress. I'll close this one for now.

Answer 6 · 2020-09-29T22:09:08.000Z

I believe I still need to fix this on my side. I need to catch any exception and just make knapsack pro fail with exit code 1. I have this on my TODO list to test this scenario.

Answer 7 · 2022-06-17T13:09:15.000Z

I've released @knapsack-pro/cypress 5.1.0 that handles exceptions from the cypress.run process and correctly exits the Knapsack Pro process with exit status 1 so that the CI build fails correctly.