salesforce/sa11y

fix(jest): codecov, automatic check tests to work with Jest v27

Closed this issue · 4 comments

Upgrading from Jest v26 to v27 results in test failures from automatic check tests that exercise the custom hook. Investigate and fix. (Found in #62)

 ● automatic checks registration › should run when called directly without setup

    Hooks cannot be defined inside tests. Hook of type "afterEach" is nested within "should run when called directly without setup".

    > 68 |         afterEach(async () => {
         |         ^
      69 |             await automaticCheck(opts);
      70 |         });
      71 |     }

      at eventHandler (node_modules/jest-circus/build/eventHandler.js:107:11)
      at registerSa11yAutomaticChecks (packages/jest/src/automatic.ts:68:9)
      at Object.<anonymous> (packages/jest/__tests__/automatic.test.ts:37:9)

Workaround:

  • Use testRunner: 'jest-jasmine2' in jest config as mentioned in Jest 27: New Defaults for Jest, 2021 edition
  • Will use this workaround for now
    • but jasmine2 test runner will be removed in next major Jest release
    • hence this needs to be fixed before then

Noticing also that codecov is incorrectly reported as missing in certain areas even with testRunner: 'jest-jasmine2'. Don't see any updates to istanbul with v27 or any ref to codecov related changes in Jest 27 Milestone · GitHub. Not sure what could be causing it. Reverting back to v26 for now.

Also noticing that the missing codecov reports are inconsistent

  1. between the jest console reporter and the HTML lcov-report, with the console reporting more missing lines (that are actually covered by tests)
  2. between subsequent runs - After adding /* istanbul ignore next */ to reported missing lines, subsequent run errors with missing codecov in different lines that were not previously reported.

Will test again with future releases of Jest to check if this issue gets fixed.