No coverage report generated for large project
cklogs opened this issue · 59 comments
Hi there,
This library is now recommended for code coverage when using v11 of the angular-cli. We are currently using karma-coverage-istanbul-reporter because we have yet to get successful coverage reports with karma-coverage. Seeing, however, that there are deprecation warnings when using the istanbul reporter library with the v11 angular-cli, we would very much like to move off it.
Perhaps someone here can help discern what is wrong with our configuration or point us in the right direction.
I will include the karma.config, and test output debug logs for two different projects.
The first project (PROJECT-1) is a bare project that was initialized with ng new command. It was updated to Angular 11 using the ng update command from angular-cli. The point of including this project is to serve as a 'control' for the second project.
The second project (PROJECT-2) comes from a relatively large production codebase. In this codebase is roughly 5000 unit tests.
Important points
- The angular.json test configuration block, tsconfig.spec.ts, and karma.config for both PROJECT-1 and PROJECT-2 are exactly the same. I will post the shared karma.config below.
- v11 angular-cli still has memory management issues at build time, similar to those mentioned here. For build/run/test on PROJECT-2 we have to use the --max_old_space_size flag. Note that we are running with latest version of the angular-cli and node v12.x.
PROJECT-1 tests are run with: ng test.
PROJECT-2 tests are run with: node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng test.
- When running tests on PROJECT-1, I get consistent coverage reports. I never see coverage reports for PROJECT-2.
- The only discernable difference between PROJECT-1 and PROJECT-2 is the
--max_old_space_sizeconfiguration property mentioned about.
Below is the karma config file shared by the two projects and the debug logs that are output when running tests.
Karma config
// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html
module.exports = function (config) {
config.set({
basePath: '',
frameworks: ['jasmine', '@angular-devkit/build-angular'],
plugins: [
require('karma-spec-reporter'),
require('karma-jasmine'),
require('karma-chrome-launcher'),
require('karma-coverage'),
require('@angular-devkit/build-angular/plugins/karma')
],
client: {
clearContext: false // leave Jasmine Spec Runner output visible in browser
},
coverage: {
dir: require('path').join(__dirname, '../coverage/karma-coverage'),
reports: ['html', 'lcovonly', 'text-summary']
},
reporters: ['dots', 'spec', 'coverage'],
specReporter: {
maxLogLines: 5, // limit number of lines logged per test
suppressErrorSummary: true, // do not print error summary
suppressFailed: false, // do not print information about failed tests
suppressPassed: true, // do not print information about passed tests
suppressSkipped: true, // do not print information about skipped tests
showSpecTiming: false, // print the time elapsed for each spec
failFast: false // test would finish with error when a first fail occurs.
},
customLaunchers: {
'CustomChromeHeadless': {
base: 'ChromeHeadless',
flags: [
'--disable-gpu',
'--disable-web-security',
'--no-sandbox',
'--remote-debugging-port=9222'
],
debug: true
}
},
browsers: ['CustomChromeHeadless'],
browserDisconnectTolerance: 2,
browserNoActivityTimeout: 300000, // five minutes
browserDisconnectTimeout: 60000,
captureTimeout: 300000,
hostname: '127.0.0.1',
port: 9876,
colors: true,
logLevel: config.LOG_DEBUG,
autoWatch: false,
singleRun: true,
restartOnFileChange: false
});
};
PROJECT-1-debug-output.txt
PROJECT-2-debug-output.txt
In the PROJECT-2 output you will see only ~20 tests ran. This is because I used an fdescribe to run a single test file so as to reduce the noise a little. The same result--no coverage--happens with or without this change.
The point of failure seems to be here:
Thanks in advance for your support.
@cklogs, karma-coverage needs to be configured using the coverageReporter property, which is missing from your configuration.
Can you please try to change the configuration like the below;
// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html
module.exports = function (config) {
config.set({
basePath: '',
frameworks: ['jasmine', '@angular-devkit/build-angular'],
plugins: [
require('karma-spec-reporter'),
require('karma-jasmine'),
require('karma-chrome-launcher'),
require('karma-coverage'),
require('@angular-devkit/build-angular/plugins/karma')
],
client: {
clearContext: false // leave Jasmine Spec Runner output visible in browser
},
+ coverageReporter: {
+ dir: require('path').join(__dirname, './coverage/karma-coverage'),
+ subdir: '.',
+ reporters: [
+ { type: 'html' },
+ { type: 'text-summary' }
+ ]
+ },
- coverage: {
- dir: require('path').join(__dirname, '../coverage/karma-coverage'),
- reports: ['html', 'lcovonly', 'text-summary']
- },
reporters: ['dots', 'spec', 'coverage'],
specReporter: {
maxLogLines: 5, // limit number of lines logged per test
suppressErrorSummary: true, // do not print error summary
suppressFailed: false, // do not print information about failed tests
suppressPassed: true, // do not print information about passed tests
suppressSkipped: true, // do not print information about skipped tests
showSpecTiming: false, // print the time elapsed for each spec
failFast: false // test would finish with error when a first fail occurs.
},
customLaunchers: {
'CustomChromeHeadless': {
base: 'ChromeHeadless',
flags: [
'--disable-gpu',
'--disable-web-security',
'--no-sandbox',
'--remote-debugging-port=9222'
],
debug: true
}
},
browsers: ['CustomChromeHeadless'],
browserDisconnectTolerance: 2,
browserNoActivityTimeout: 300000, // five minutes
browserDisconnectTimeout: 60000,
captureTimeout: 300000,
hostname: '127.0.0.1',
port: 9876,
colors: true,
logLevel: config.LOG_DEBUG,
autoWatch: false,
singleRun: true,
restartOnFileChange: false
});
};More information about karma-coverage configuration can be found here: https://github.com/karma-runner/karma-coverage/blob/master/docs/configuration.md
I tried exactly as is suggested above and coverage reports are still not being generated.
Perhaps there is something more I'm missing?
Without a reproduction it's hard to tell what's happening. You can try to debug
karma-coverage/lib/reporter.js
Line 224 in 1d34a56
I'm still trying to demystify what is going on but I have observed the following:
For these particular methods:
karma-coverage/lib/reporter.js
Lines 185 to 201 in 1d34a56
The browsers object on line 185 is defined as:
BrowserCollection {
browsers: [],
emitter: EventEmitter {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
[Symbol(kCapture)]: false
}
}
This means that if (browsers) { executes.
However, this.onBrowserStart is never called during the entire execution of the tests.
Once the tests have all ran, this function is called:
karma-coverage/lib/reporter.js
Lines 203 to 210 in 1d34a56
The parameter browser is defined as:
Browser {
id: '35804940',
fullName: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/89.0.4389.90 Safari/537.36',
name: 'Chrome Headless 89.0.4389.90 (Windows 10)',
lastResult: BrowserResult {
startTime: 1617310253880,
total: 0,
success: 6032,
failed: 0,
skipped: 0,
totalTime: 628830,
netTime: 437749,
error: false,
disconnected: false
},
disconnectsCount: 0,
activeSockets: [
Socket {
_events: [Object: null prototype],
_eventsCount: 8,
_maxListeners: undefined,
nsp: [Namespace],
client: [Client],
acks: Map {},
fns: [],
flags: {},
_rooms: Set {},
server: [Server],
adapter: [Adapter],
id: 'eEVk_hCsxP6b3WzbAAAD',
connected: true,
disconnected: false,
handshake: [Object],
[Symbol(kCapture)]: false
}
],
noActivityTimeout: 300000,
singleRun: true,
clientConfig: {
args: [],
useIframe: true,
runInParent: false,
captureConsole: true,
clearContext: true,
originalArgs: []
},
collection: BrowserCollection {
browsers: [ [Circular] ],
emitter: Server {
_events: [Object: null prototype],
_eventsCount: 17,
_maxListeners: undefined,
log: [Logger],
loadErrors: [],
_injector: [Injector],
_boundServer: [Server],
_fileList: [FileList],
[Symbol(kCapture)]: false
}
},
emitter: Server {
_events: [Object: null prototype] {
load_error: [Function],
exit: [Array],
run_complete: [Array],
browser_complete: [Array],
file_list_modified: [Array],
run_start: [Array],
browser_start: [Array],
browser_error: [Array],
browser_log: [Array],
spec_complete: [Array],
spec_failure: [Function],
browsers_change: [Function],
browser_register: [Function],
stop: [Function],
browser_restart_failure: [Function],
browser_complete_with_no_more_retries: [Function],
browser_process_failure: [Function]
},
_eventsCount: 17,
_maxListeners: undefined,
log: Logger {
category: 'karma-server',
context: {},
parseCallStack: [Function: defaultParseCallStack]
},
loadErrors: [],
_injector: Injector {
_providers: [Object: null prototype],
_instances: [Object: null prototype],
get: [Function: get],
invoke: [Function: invoke],
instantiate: [Function: instantiate],
createChild: [Function: createChild]
},
_boundServer: Server {
_events: [Object: null prototype],
_eventsCount: 3,
_maxListeners: undefined,
_connections: 0,
_handle: [TCP],
_usingWorkers: false,
_workers: [],
_unref: false,
allowHalfOpen: false,
pauseOnConnect: false,
_connectionKey: '4:0.0.0.0:9876',
[Symbol(kCapture)]: false,
[Symbol(asyncId)]: 13
},
_fileList: FileList {
_patterns: [Array],
_excludes: [Array],
_emitter: [Circular],
_preprocess: [AsyncFunction: preprocess],
buckets: [Map],
_refreshing: [Promise],
_emitModified: [Function]
},
[Symbol(kCapture)]: false
},
socket: Socket {
_events: [Object: null prototype] {
start: [Function],
info: [Function],
karma_error: [Function],
result: [Function],
complete: [Function],
error: [Function],
register: [Function],
disconnect: [Function]
},
_eventsCount: 8,
_maxListeners: undefined,
nsp: Namespace {
_events: [Object: null prototype],
_eventsCount: 1,
_maxListeners: undefined,
sockets: [Map],
_fns: [],
_rooms: Set {},
_flags: {},
_ids: 0,
server: [Server],
name: '/',
adapter: [Adapter],
[Symbol(kCapture)]: false
},
client: Client {
sockets: Map {},
nsps: Map {},
server: [Server],
conn: [Socket],
encoder: Encoder {},
decoder: [Decoder],
id: '2DlQ3r3gH89vkFHCAAAA',
onclose: [Function: bound onclose],
ondata: [Function: bound ondata],
onerror: [Function: bound onerror],
ondecoded: [Function: bound ondecoded],
connectTimeout: undefined
},
acks: Map {},
fns: [],
flags: {},
_rooms: Set {},
server: Server {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
_nsps: [Map],
parentNsps: Map {},
_path: '/socket.io',
clientPathRegex: /^\/socket\.io\/socket\.io(\.min|\.msgpack\.min)?\.js(\.map)?$/,
_connectTimeout: 45000,
_serveClient: true,
_parser: [Object],
encoder: Encoder {},
_adapter: [class Adapter extends EventEmitter],
sockets: [Namespace],
opts: [Object],
eio: [Server],
httpServer: [Server],
engine: [Server],
[Symbol(kCapture)]: false
},
adapter: Adapter {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
nsp: [Namespace],
rooms: [Map],
sids: [Map],
encoder: Encoder {},
[Symbol(kCapture)]: false
},
id: 'tpLs3SFHdGeiHnz_AAAB',
connected: false,
disconnected: true,
handshake: {
headers: [Object],
time: 'Thu Apr 01 2021 13:50:53 GMT-0700 (Pacific Daylight Time)',
address: '127.0.0.1',
xdomain: false,
secure: false,
issued: 1617310253837,
url: '/socket.io/?EIO=4&transport=polling&t=NYFD0BQ',
query: [Object: null prototype],
auth: {}
},
[Symbol(kCapture)]: false
},
timer: {
setTimeout: [Function: setTimeout],
clearTimeout: [Function: clearTimeout]
},
disconnectDelay: 60000,
log: Logger {
category: 'Chrome Headless 89.0.4389.90 (Windows 10)',
context: {},
parseCallStack: [Function: defaultParseCallStack]
},
noActivityTimeoutId: Timeout {
_idleTimeout: 300000,
_idlePrev: [TimersList],
_idleNext: [TimersList],
_idleStart: 983207,
_onTimeout: [Function],
_timerArgs: undefined,
_repeat: null,
_destroyed: false,
[Symbol(refed)]: true,
[Symbol(asyncId)]: 1889860,
[Symbol(triggerId)]: 1889856
},
pendingDisconnect: Timeout {
_idleTimeout: -1,
_idlePrev: null,
_idleNext: null,
_idleStart: 385875,
_onTimeout: null,
_timerArgs: undefined,
_repeat: null,
_destroyed: true,
[Symbol(refed)]: true,
[Symbol(asyncId)]: 1809518,
[Symbol(triggerId)]: 1809514
},
state: 'CONNECTED'
}
coverageMaps are defined as:
[Object: null prototype] {}
The result is a large object containing coverage results for each file. One such example:
'C:\\app\\shared\\helpers\\url-serializer.ts': {
path: 'C:\\app\\shared\\helpers\\url-serializer.ts',
statementMap: [Object],
fnMap: [Object],
branchMap: [Object],
s: [Object],
f: [Object],
b: [Object],
inputSourceMap: [Object],
_coverageSchema: '1a1c01bbd47fc00a2c39e90264f33305004495a9',
hash: 'b12799a1103f348e9c0f91bb2cb5bbf81c79d909'
}
Is there anything more you would like to see?
@alan-agius4 When you have a chance, could you please review my updates?
Advance thanks!
I'll leave the triaging to someone from the karma-coverage team.
To the maintainers of this repo:
I have a small suspicion that,
- the sheer size of our project and,
- the associated long compile time while running tests
have something to do with coverage failing to be generated. But this is only a hunch.
I cannot share or anonymize our code base, but if there is a large open repository that you would suggest I try and reproduce this issue from, I will happily try.
Aside from that, if you have any insights on the above issue, please let me know here or contact me directly.
Thanks again.
We also have a big project with 3000+ unit tests. And I can see coverage reports generated only on powerful machines. On our ci server and on slow machines it's not generated.
@WebMex
Thank you for the additional data point!
Do you plan to continue using karma-coverage, or have you sought out a different library? We have not been able to move onto karma-coverage (from karma-coverage-instanbul-reporter) because of the above issue, but we are open to other coverage generators.
@cklogs I have rolled back to karma-coverage-instanbul-reporter so far and waiting for the fix. Despite the warning message from angular it is still possible to use it.
So far I have had little luck getting feedback from any of the maintainers of this library.
Given karma-coverage is now the defacto standard coverage library for Angular, it may be necessary to raise this up to the greater Angular team.
Please advise on how to most effectively do this.
Thanks in advance for your help!
Thanks to your tip, I tested karma-coverage on a faster box (Ryzen 7 PRO / 32 GB RAM), and I am getting consistent coverage results there. Still need this fix for our deployment servers to generate coverage, but I'm hoping these multiple data points give the maintainers a lead on the issue.
This would be a hard one to track down without a reproduction. I tried it on a seed project and generated a large number of tests but I didn't manage to replicate.
Unfortunately, I am not too familiar with this plugin to get to the bottom of this without being able to reproduce the issue.
I am seeing the same issue. If I run the test on my mac locally, I'm seeing the lcov.info file always get generated. However, running the test with even a greater --max_old_space_size size specified and run by Jenkins on a linux box intermittently does not generate the lcov.info file. There are no reports of exceptions or anything else I can see that explain why the report is not generated.
When the coverage isn't generated I see something like:
[2021-06-29T21:37:31.258Z] TOTAL: 6266 SUCCESS
[2021-06-29T21:37:31.258Z]
[2021-06-29T21:37:31.258Z] Finished in 13 mins 7.178 secs / 9 mins 58.633 secs @ 21:37:31 GMT+0000 (Coordinated Universal Time)
[2021-06-29T21:37:31.258Z]
[2021-06-29T21:37:31.258Z] SUMMARY:
[2021-06-29T21:37:31.258Z] ✔ 6266 tests completed
[2021-06-29T21:37:31.258Z] ℹ 28 tests skipped
When I run it locally and the file is generated I get:
TOTAL: 6266 SUCCESS
Finished in 14 mins 0.813 secs / 10 mins 58.549 secs @ 15:34:47 GMT-0600 (Mountain Daylight Time)
SUMMARY:
✔ 6266 tests completed
ℹ 28 tests skipped
29 06 2021 15:35:04.211:WARN [launcher]: Chrome was not killed in 2000 ms, sending SIGKILL.
Oddly, it seems the Chrome was not killed is usually reported when the lcov.info file is generated, but if that is not reported, the coverage file was not generated. So it almost seems like the node process is exiting before the karma-coverage plugin is done generating the report.
Could the problem be that the process can exit before the plugin/reporter has completed writing the file?
@alan-agius4
Perhaps this project might be a good place to run a smoke test of this issue: https://github.com/angular/angular-cli-stress-test. I was able to get it roughly running just by dropping the files into a newly spun up Angular 12 project.
I could not get coverage reports generating on the codebase. You will likely be able to reproduce the issue so long as your machine is not too powerful.
Please can this be done? We have the same issue. We have about 13000 unit tests. Sometimes the coverage report gets generated, however it mostly fails. We run them on a Linux docker image trion/ng-cli-karma.
It's upsetting to be "punished" for having too many tests.
I have done a previous investigation on this, and I found that it was related to the speed/number of tests. It failed in Istanbul, I believe it was when merging the coverage summaries that the result was just suddenly empty and then no coverage were created
Unfortunately this started happening ~75% of the time for us, but I could not reproduce it locally. Only solution I could find was to uninstall and go back to an unmaintained, but much more reliable solution for now. This is a blocker, we need it to be 100% reliable.
This looks to be an issue with Karma:
karma-runner/karma#945
Unfortunately it seems that Karma is a dead project. That issue was opened in 2014 and was closed since no one worked on the issue
It is absolutely not dead! We work with limited resources, so we cannot address every issue immediately and sadly that one got closed a long time ago; I will be adding it to my working queue. Also, your collaboration is welcome.
I am experiencing the same issue after addressing the following message from our test run:
'karma-coverage-istanbul-reporter' usage has been deprecated since version 11.
Please install 'karma-coverage' and update 'karma.conf.js.' For more info, see https://github.com/karma-runner/karma-coverage/blob/master/README.md
I am going to revert the change in my project, and follow this issue, in hopes it gets resolved soon.
Some additional observations: I have seen the coverage report generated one time successfully, and if I scope the files in my test runner to a smaller subset of .spec.ts files, it is generated every time. This clearly seems to be a memory or cpu issue. Whats troubling to me is that the process exits as though it successfully generated the report!
Update: Reverting back to karma-coverage-istanbul-reporter 100% fixed the issue, and our coverage reports are being reliably generated again, despite the deprecation warning. I can confirm there are issues with code coverage reports and:
Angular:
$ ng --version
_ _ ____ _ ___
/ \ _ __ __ _ _ _| | __ _ _ __ / ___| | |_ _|
/ △ \ | '_ \ / _` | | | | |/ _` | '__| | | | | | |
/ ___ \| | | | (_| | |_| | | (_| | | | |___| |___ | |
/_/ \_\_| |_|\__, |\__,_|_|\__,_|_| \____|_____|___|
|___/
Angular CLI: 12.2.1
Node: 14.15.5
Package Manager: npm 6.14.11
OS: darwin x64
Angular: 12.2.1
... animations, cdk, cli, common, compiler, compiler-cli, core
... forms, language-service, material, material-moment-adapter
... platform-browser, platform-browser-dynamic, platform-server
... router
Package Version
---------------------------------------------------------
@angular-devkit/architect 0.1202.1
@angular-devkit/build-angular 12.2.1
@angular-devkit/core 12.2.1
@angular-devkit/schematics 12.2.1
@angular/flex-layout 12.0.0-beta.34
@schematics/angular 12.2.1
rxjs 6.6.7
typescript 4.3.5
webpack 5.50.0
karma stuffs from package.json:
$ cat package.json | grep karma
"karma-spec-reporter": "^0.0.32",
"karma": "~6.3.4",
"karma-chai": "^0.1.0",
"karma-chrome-launcher": "~3.1.0",
"karma-cli": "^2.0.0",
"karma-coverage": "~2.0.3",
"karma-jasmine": "~4.0.0",
"karma-jasmine-html-reporter": "^1.5.0",
"karma-junit-reporter": "^2.0.1",
"karma-phantomjs-launcher": "^1.0.4",
"karma-typescript": "^5.5.1",
I hope this helps
I think I am seeing this problem as well. Intermittently, the coverage directory isn't created. I don't know that my project is large or not... not sure of the baseline!
$ ./node_modules/.bin/cloc --include-lang=TypeScript src/app/ vendored/
381 text files.
380 unique files.
148 files ignored.
github.com/AlDanial/cloc v 1.90 T=1.48 s (165.3 files/s, 117159.1 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
TypeScript 245 2320 1451 169847
-------------------------------------------------------------------------------
SUM: 245 2320 1451 169847
-------------------------------------------------------------------------------
An example of failure:
✓ should submit form
✓ should submit form API data
✓ should check permission for edit
✓ should edit & delete API key
TOTAL: 182 SUCCESS
TOTAL: 182 SUCCESS
Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 0 (skipped 1) SUCCESS (59.589 secs / 17.681 secs)
TOTAL: 182 SUCCESS
____ _ _ ____ ____ _____ ____ ____
/ ___|| | | |/ ___/ ___| ____/ ___/ ___|
\___ \| | | | | | | | _| \___ \___ \
___) | |_| | |__| |___| |___ ___) |__) |
|____/ \___/ \____\____|_____|____/____/
$ ls -la coverage/
ls: cannot access coverage/: No such file or directory
Cleaning up file based variables
An example of success:
Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 183 (skipped 1) SUCCESS (17.465 secs / 15.649 secs)
TOTAL: 182 SUCCESS
TOTAL: 182 SUCCESS
Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 183 (skipped 1) SUCCESS (17.465 secs / 15.649 secs)
TOTAL: 182 SUCCESS
08 10 2021 11:15:39.085:WARN [launcher]: ChromeHeadless was not killed in 2000 ms, sending SIGKILL.
____ _ _ ____ ____ _____ ____ ____
/ ___|| | | |/ ___/ ___| ____/ ___/ ___|
\___ \| | | | | | | | _| \___ \___ \
___) | |_| | |__| |___| |___ ___) |__) |
|____/ \___/ \____\____|_____|____/____/
$ ls -la coverage/
total 832
drwxrwxr-x 4 gitlab-runner gitlab-runner 152 Oct 8 11:15 .
drwxrwxr-x 8 gitlab-runner gitlab-runner 4096 Oct 8 11:15 ..
-rw-rw-r-- 1 gitlab-runner gitlab-runner 658117 Oct 8 11:15 cobertura.txt
drwxrwxr-x 3 gitlab-runner gitlab-runner 182 Oct 8 11:15 report-html
drwxrwxr-x 3 gitlab-runner gitlab-runner 42 Oct 8 11:15 report-lcov
-rw-rw-r-- 1 gitlab-runner gitlab-runner 160020 Oct 8 11:15 report-lcovonly.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner 697 Oct 8 11:15 teamcity.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner 302 Oct 8 11:15 text-summary.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner 14499 Oct 8 11:15 text.txt
What is kind of interesting is that I get the SIGKILL in the case in which it works.... but maybe that's just karma killing the browser as it writes out the reports.
I also had no problems when using Istanbul previously.
Due to the warning, I'm just going to arbitrarily try setting the timeout higher in karma.conf.js and see what happens... I can't justify why it might help, but I'd like to see the warning go away anyway:
processKillTimeout: 60 * 1000
update: it didn't help. I still get intermittent failures to create the coverage output.
update 2: reducing the number of kinds of reports to just text/text-summary might have helped. Waiting for more data.
update 3: seems to still happen. Maybe when cicd machine is under higher load. Will need to go back to istanbul version more than likely.
I have exactly the same issue. The coverage reports are generated only when the following messasge is logged:
ChromeHeadless was not killed in 2000 ms, sending SIGKILL.
Is there a way to force karma to always send this? Or any other updates on this issue?
From a quick look it seems that the problem is that onRunComplete returns a Promise, but Karma does not wait for for it. As a result Karma shutdown will race with the writing of the coverage report.
Unfortunately, without a reproduction it is hard to confirm or reject the theory. I wonder if somebody having the problem can run tests using this branch and see if it solves the problem? The only change there is that Karma will not call process.exit once done, thus allowing all async activities to complete.
npm i devoto13/karma#test-coverage
UPD Looks like that promise is actually awaited in the onExit callback. Another place, which is not synchronized is this one 🤔 I think it still will be useful to perform the test, so we can confirm that the issue is caused by the race and focus on locating and fixing this race.
I wonder if somebody having the problem can run tests using this branch and see if it solves the problem?
Trying now. I'll do a few runs on this and report back to see how it does with our code. Thanks
First run worked. Our CI server is busy today and the tests can take a while, so I'll run a few more runs on the branch and see if it is working 100% of the time across several runs.
I tried the branch that @devoto13 posted and it does seem to fix the problem 👍
Out of 20 executions the coverage report was generated in all of them.
I did not have any reproductions of the issue during my checks on the branch. We are going to try to use it for all our code until it is released and I'll report back if anyone fails to have coverage generated. Thank you!
@arobinson The branch was only intended for testing the race, I think it will always exit with 0 exit code (even when tests have failed).
Haven't found any more races, so hopefully, #463 is the final fix. You can also give it a try on your codebase by installing it from the branch using the below command:
$ npm i devoto13/karma-coverage#fix-race
I think it will always exit with 0 exit code
FYI, I got an exit code of 1 with a test failure even with that branch
Using the fix-race branch for #463, I tried it on our software and the build failed:
Test report file /tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/junit/TESTS-Chrome_Headless_92.0.4515.159_(Linux_x86_64).xml was length 0
The tests started fine:
[2022-01-20T16:15:59.383Z] - Generating browser application bundles (phase: setup)...
[2022-01-20T16:16:01.932Z] Test results are being saved to: /tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results
[2022-01-20T16:16:01.932Z] 20 01 2022 16:16:01.915:INFO [karma-server]: Writing browser console to file: /tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/console.log
[2022-01-20T16:16:01.932Z]
[2022-01-20T16:16:01.932Z] START:
[2022-01-20T16:16:02.188Z] 20 01 2022 16:16:01.938:INFO [karma-server]: Karma v6.3.11 server started at http://localhost:9876/
[2022-01-20T16:16:02.188Z] 20 01 2022 16:16:01.939:INFO [launcher]: Launching browsers ChromeHeadless with concurrency unlimited
[2022-01-20T16:16:02.188Z] 20 01 2022 16:16:01.949:INFO [launcher]: Starting browser Chrome
[2022-01-20T16:18:08.605Z] 20 01 2022 16:18:05.217:WARN [launcher]: Chrome has not captured in 120000 ms, killing.
[2022-01-20T16:18:08.605Z] 20 01 2022 16:18:07.315:WARN [launcher]: Chrome was not killed in 2000 ms, sending SIGKILL.
[2022-01-20T16:18:20.800Z] 20 01 2022 16:18:19.493:INFO [launcher]: Trying to start Chrome again (1/2).
[2022-01-20T16:19:42.190Z] ✔ Browser application bundle generation complete.
[2022-01-20T16:19:42.190Z] 20 01 2022 16:19:33.627:INFO [Chrome Headless 92.0.4515.159 (Linux x86_64)]: Connected on socket uvg6glKjq65iOVWJAAAB with id 95294800
[2022-01-20T16:20:04.076Z] AppComponent Test Suite
[2022-01-20T16:20:04.076Z] ✔ Initial Rendering
But at the end, the tests failed badly:
[2022-01-20T16:31:58.196Z] ✔ 7564 tests completed
[2022-01-20T16:31:58.196Z] ℹ 35 tests skipped
[2022-01-20T16:32:13.033Z] 20 01 2022 16:32:12.590:WARN [launcher]: Chrome was not killed in 2000 ms, sending SIGKILL.
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.792:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.792:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z] errno: -17,
[2022-01-20T16:32:13.968Z] code: 'EEXIST',
[2022-01-20T16:32:13.968Z] syscall: 'mkdir',
[2022-01-20T16:32:13.968Z] path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.794:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.794:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z] errno: -17,
[2022-01-20T16:32:13.968Z] code: 'EEXIST',
[2022-01-20T16:32:13.968Z] syscall: 'mkdir',
[2022-01-20T16:32:13.968Z] path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z] errno: -17,
[2022-01-20T16:32:13.968Z] code: 'EEXIST',
[2022-01-20T16:32:13.968Z] syscall: 'mkdir',
[2022-01-20T16:32:13.968Z] path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z] errno: -17,
[2022-01-20T16:32:13.968Z] code: 'EEXIST',
[2022-01-20T16:32:13.968Z] syscall: 'mkdir',
[2022-01-20T16:32:13.968Z] path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }
That EEXIST error was repeated 34,494 times in the job log before Jenkins stopped recording the output. Looks like it got caught in an infinite loop
Thanks for the feedback @arobinson. Looks like there is still a race between a check for a directory existence and creation in the helper, which is executed concurrently for all reporters. I'll look into fixing it.
Interesting that it got into an infinite loop though 🤔
Here is another attempt to fix this:
$ npm i devoto13/karma-coverage#fix-race
$ npm i devoto13/karma#fix-helper
which should address the infinite loop and the race when creating the same directory concurrently. I've tried it on the fresh Angular project and it seems to work now.
@alan-agius4
I've noticed since upgrading our project to Angular 13 that, when running ng test, the karma-coverage-instanbul-reporter deprecation warning has gone away.
Do you know if this was this intentional? I combing through the v13 commits to angular-cli, I could not find where it was changed. I thought you might have some insight.
Yea, that’s intentional as we no longer support it.
that said, you can still use it, if you provide the reporter configuration yourself.
Testing the latest patches, I did get code coverage numbers without any issues. I'll try a couple of more times to make sure
Did 3 runs, all collected coverage correctly
@jginsburgn Looks like my changes resolve the coverage flakiness. Please take a look at the corresponding PRs (one and two) when you have some time.
@jginsburgn Looks like my changes resolve the coverage flakiness. Please take a look at the corresponding PRs (one and two) when you have some time.
Checking them now. Sorry about my delay.
🎉 This issue has been resolved in version 2.1.1 🎉
The release is available on:
Your semantic-release bot 📦🚀
I upgraded and I believe I'm still not getting a coverage directory in cicd. Locally, yes, but in cicd, no. The main difference being the load being higher on the cicd machine from other jobs running in parallel.
$ grep karma-coverage yarn.lock
karma-coverage-istanbul-reporter@^3.0.3:
resolved "http://build-artifactory.eng.vmware.com/artifactory/api/npm/npm/karma-coverage-istanbul-reporter/-/karma-coverage-istanbul-reporter-3.0.3.tgz#f3b5303553aadc8e681d40d360dfdc19bc7e9fe9"
karma-coverage@^2.1.1:
resolved "http://build-artifactory.eng.vmware.com/artifactory/api/npm/npm/karma-coverage/-/karma-coverage-2.1.1.tgz#7434f8736841e08de3e6c302632e8d239634f388"
(I'll also try without the istanbul one also in node_modules)
$ more karma.conf.js
const path = require("path")
const fs = require("fs")
function get_backend_url() {
const config_file = path.resolve(process.cwd(), ".karma-backend-config.json")
try {
const data = fs.readFileSync(config_file)
const config = JSON.parse(data)
return config.LIVE_SERVER_URL
} catch (error) {
return "http://localhost:12345"
}
}
const backend_url = get_backend_url()
module.exports = function (config) {
config.set({
failOnSkippedTests: false,
basePath: "",
frameworks: ["jasmine", "@angular-devkit/build-angular"],
processKillTimeout: 60 * 1000,
plugins: [
require("karma-jasmine"),
require("karma-chrome-launcher"),
require("karma-coverage"),
require("@angular-devkit/build-angular/plugins/karma"),
require("karma-spec-reporter"),
],
logLevel: config.LOG_INFO,
client: {
clearContext: false, // leave Jasmine Spec Runner output visible in browser
jasmine: {
random: false,
timeoutInterval: 300000,
},
},
// doesn't always work: https://github.com/karma-runner/karma-coverage/issues/434
coverageReporter: {
dir: path.join(__dirname, "coverage"),
reporters: [
{ type: "html", subdir: "report-html" },
{ type: "lcov", subdir: "report-lcov" },
{ type: "text-summary", subdir: ".", file: "text-summary.txt" },
],
fixWebpackSourcePaths: true,
},
// https://stackoverflow.com/questions/54744584/karma-disconnected-because-no-message-in-10000-ms
captureTimeout: 300000,
browserDisconnectTolerance: 3,
browserDisconnectTimeout: 300000,
browserNoActivityTimeout: 300000,
reporters: [
"progress",
"spec",
"coverage",
],
specReporter: {
suppressSkipped: false,
},
hostname: "0.0.0.0",
port: 9876,
colors: true,
autoWatch: true,
browsers: ["ChromeHeadless"],
singleRun: false,
restartOnFileChange: true,
proxies: {
"/api/": backend_url + "/api/",
"/ajax/": backend_url + "/ajax/",
"/saml2/": backend_url + "/saml2/",
},
})
}
@rrauenza Are you also using karma 6.3.15 which contains the second part of the fix?
If so I wonder if this can be related to the issue reported in karma-runner/karma#3745, where onBrowserStart is not called. Would you be willing to add some logs to the on* hooks in
karma-coverage/lib/reporter.js
Line 189 in 9e8492c
onBrowserStart should be a good place to start. With this information, I may be able to pinpoint the issue and make the fix.
The thing with onBrowserStart is that it is sent over the socket.io connection by the browser and I suspect that socket.io may lose messages while upgrading to the WebSocket connection when the host/browser is under heavy load. There were several issues reported that seem related, but it is tricky to reproduce and confirm from our side.
@arobinson I wonder if you had time to upgrade karma and karma-coverage in your project and can confirm whether the flaky coverage issues was resolved or it is still flaky for you?
No, karma plugins are not used to have peerDependencies, that's something I would like to better address in the future.
Unfortunately the code coverage is often not generated for me when the unit tests run on a Azure Devops pipeline (always works on local). I'm using the latest packages:
- karma: 6.3.17
- karma-coverage: 2.2.0
- @angular-devkit/build-angular and @angular/cli: 13.2.5
Here is a comparison of the end of the logs in case it helps. Since the source code is the same on both executions, I wonder if the difference in the order of the logged actions is the cause of the issue.
With test coverage generated 👍
Without test coverage: 👎
@ozkoidi From your logs, you seem to be using the https://github.com/joeljeske/karma-parallel plugin which may be the reason. Can you please try to run without it and see if you still get the same problem?
I'm not familiar with that plugin and will try to look at it when I have time to see if it does something suspicious.
@devoto13, thanks for your suggestion. I removed the karma-parallel package and the result is still the same, so no need for you to look into that package.
In case it helps the specs of the Microsoft hosted agents running the tests are: 2 core CPU, 7 GB of RAM, and 14 GB of SSD disk space. I think those VM should be enough to run the test properly but I don't know what else could make the test not generate coverage reports randomly. Updated logs:
With test coverage generated 👍
Without test coverage 👎
@ozkoidi Thank you for the quick feedback. Can you please install karma and karma-coverage from these two branches and make more sample runs on your CI?
$ npm i devoto13/karma#test-coverage
$ npm i devoto13/karma-coverate#test-coverage
The first one removes the call to the process.exit on Karma shut down, so if the problem is resolved by the change, it means that we still have a race somewhere in the coverage generation routine. Per #434 (comment). Here is the code.
The second one is based on #434 (comment). It adds logs to all karma-coverage hooks so that we can compare good and bad runs and see if we can get some clue about what is different. Here is the code.
Thank you @devoto13, I tried your branches yesterday and the result was the same (no coverage generated half the time).
With Karma's logLevel set to debug the logs are huge so I put here just the beginning and end of the two types of run.
I include the reduced and cleaned logs below but as far as I can see the differences are:
| Coverage report generated | No coverage report generated |
|---|---|
| Logs show [coverage]: onBrowserComplete completed | No logs about onBrowserComplete being called |
In "[coverage]: onRunComplete called BrowserCollection" the Browser object's pendingDisconnect property is null |
In "[coverage]: onRunComplete called BrowserCollection" the Browser object's pendingDisconnect contains a Timeout object |
| The launcher sends the SIGKILL | no SIGKILL is sent |
| [coverage]: Writing coverage is present before onExit completed | no "Writing coverage" debug log before onExit completed |
| [reporter.junit]: JUnit results written | no "JUnit results written" log |
With test coverage generated 👍
> ng test --no-watch --code-coverage
- Generating browser application bundles (phase: setup)...
[config]: Loading config /home/vsts/work/1/s/Client/karma.conf.js
[config]: autoWatch set to false, because of singleRun
.
.
.
INFO [coverage]: onBrowserComplete completed
DEBUG [launcher]: CAPTURED -> BEING_KILLED
DEBUG [launcher]: BEING_KILLED -> BEING_FORCE_KILLED
DEBUG [Chrome Headless 99.0.4844.51 (Linux x86_64)]: CONNECTED -> DISCONNECTED
TOTAL: 3231 SUCCESS
Chrome Headless 99.0.4844.51 (Linux x86_64): Executed 3231 of 3234 (skipped 3)�[32m SUCCESS�[39m (1 min 38.135 secs / 59.497 secs)
�[32mTOTAL: 3231 SUCCESS�[39m
10 03 2022 16:10:38.524:INFO [coverage]: onRunComplete called BrowserCollection {
browsers: [
Browser {
id: '25015465',
fullName: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/99.0.4844.51 Safari/537.36',
name: 'Chrome Headless 99.0.4844.51 (Linux x86_64)',
lastResult: [BrowserResult],
disconnectsCount: 0,
activeSockets: [Array],
noActivityTimeout: 210000,
singleRun: true,
clientConfig: [Object],
collection: [BrowserCollection],
emitter: [Server],
socket: [Socket],
timer: [Object],
disconnectDelay: 210000,
log: [Logger],
noActivityTimeoutId: Timeout {
\_idleTimeout: 210000,
\_idlePrev: [TimersList],
\_idleNext: [Timeout],
\_idleStart: 278390,
\_onTimeout: [Function (anonymous)],
\_timerArgs: undefined,
\_repeat: null,
\_destroyed: false,
[Symbol(refed)]: true,
[Symbol(kHasPrimitive)]: false,
[Symbol(asyncId)]: 680445,
[Symbol(triggerId)]: 680443
},
pendingDisconnect: null,
state: 'DISCONNECTED'
}
],
emitter: EventEmitter {
\_events: [Object: null prototype] {},
\_eventsCount: 0,
\_maxListeners: undefined,
[Symbol(kCapture)]: false
}
}
INFO [coverage]: onRunComplete completed
DEBUG [karma-server]: Run complete, exiting.
DEBUG [launcher]: Disconnecting all browsers
DEBUG [launcher]: BEING_FORCE_KILLED -> BEING_FORCE_KILLED
DEBUG [proxy]: Destroying proxy agents
INFO [coverage]: onExit called
=============================== Coverage summary ===============================
Statements : 91.97% ( 23967/26059 )
Branches : 79.05% ( 9673/12237 )
Functions : 88.99% ( 6694/7522 )
Lines : 92.2% ( 22669/24586 )
================================================================================
WARN [launcher]: ChromeHeadless was not killed in 2000 ms, sending SIGKILL.
DEBUG [coverage]: Writing coverage to /home/vsts/work/1/s/Client/coverage
DEBUG [launcher]: Process ChromeHeadless exited with code null and signal SIGTERM
DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-25015465
DEBUG [coverage]: Writing coverage to /home/vsts/work/1/s/Client/coverage
DEBUG [coverage]: Writing coverage to /home/vsts/work/1/s/Client/coverage
INFO [coverage]: onExit completed
DEBUG [reporter.junit]: JUnit results written to "/home/vsts/work/1/s/Client/TESTS-Chrome*Headless_99.0.4844.51*(Linux_x86_64).xml".
DEBUG [launcher]: Finished all browsers
DEBUG [launcher]: BEING_FORCE_KILLED -> FINISHED
DEBUG [launcher]: FINISHED -> FINISHED
DEBUG [karma-server]: Received stop event, exiting.
DEBUG [launcher]: Disconnecting all browsers
DEBUG [launcher]: FINISHED -> BEING_FORCE_KILLED
DEBUG [proxy]: Destroying proxy agents
INFO [coverage]: onExit called
INFO [coverage]: onExit completed ##[section]Finishing: Test
Without test coverage 👎
> ng test --no-watch --code-coverage
- Generating browser application bundles (phase: setup)...
DEBUG [config]: Loading config /home/vsts/work/1/s/Client/karma.conf.js
DEBUG [config]: autoWatch set to false, because of singleRun
.
.
.
DEBUG [launcher]: CAPTURED -> BEING_KILLED
DEBUG [launcher]: BEING_KILLED -> BEING_FORCE_KILLED
DEBUG [Chrome Headless 99.0.4844.51 (Linux x86_64)]: CONNECTED -> DISCONNECTED
TOTAL: 3231 SUCCESS
Chrome Headless 99.0.4844.51 (Linux x86_64): Executed 3231 of 0 (skipped 3)�[32m SUCCESS�[39m (2 mins 15.127 secs / 59.651 secs)
�[32mTOTAL: 3231 SUCCESS�[39m
INFO [coverage]: onRunComplete called BrowserCollection {
browsers: [
Browser {
id: '13380493',
fullName: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/99.0.4844.51 Safari/537.36',
name: 'Chrome Headless 99.0.4844.51 (Linux x86_64)',
lastResult: [BrowserResult],
disconnectsCount: 0,
activeSockets: [Array],
noActivityTimeout: 210000,
singleRun: true,
clientConfig: [Object],
collection: [BrowserCollection],
emitter: [Server],
socket: [Socket],
timer: [Object],
disconnectDelay: 210000,
log: [Logger],
noActivityTimeoutId: Timeout {
\_idleTimeout: 210000,
\_idlePrev: [TimersList],
\_idleNext: [Timeout],
\_idleStart: 285367,
\_onTimeout: [Function (anonymous)],
\_timerArgs: undefined,
\_repeat: null,
\_destroyed: false,
[Symbol(refed)]: true,
[Symbol(kHasPrimitive)]: false,
[Symbol(asyncId)]: 681197,
[Symbol(triggerId)]: 681196
},
pendingDisconnect: Timeout {
\_idleTimeout: -1,
\_idlePrev: null,
\_idleNext: null,
\_idleStart: 187011,
\_onTimeout: null,
\_timerArgs: undefined,
\_repeat: null,
\_destroyed: true,
[Symbol(refed)]: true,
[Symbol(kHasPrimitive)]: false,
[Symbol(asyncId)]: 646232,
[Symbol(triggerId)]: 646172
},
state: 'DISCONNECTED'
}
],
emitter: EventEmitter {
\_events: [Object: null prototype] {},
\_eventsCount: 0,
\_maxListeners: undefined,
[Symbol(kCapture)]: false
}
}
INFO [coverage]: onRunComplete completed
DEBUG [karma-server]: Run complete, exiting.
DEBUG [launcher]: Disconnecting all browsers
DEBUG [launcher]: BEING_FORCE_KILLED -> BEING_FORCE_KILLED
DEBUG [proxy]: Destroying proxy agents
INFO [coverage]: onExit called
INFO [coverage]: onExit completed
DEBUG [launcher]: Process ChromeHeadless exited with code null and signal SIGTERM
DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-13380493
DEBUG [launcher]: Finished all browsers
DEBUG [launcher]: BEING_FORCE_KILLED -> FINISHED
DEBUG [launcher]: FINISHED -> FINISHED
DEBUG [karma-server]: Received stop event, exiting.
DEBUG [launcher]: Disconnecting all browsers
DEBUG [launcher]: FINISHED -> BEING_FORCE_KILLED
DEBUG [proxy]: Destroying proxy agents
INFO [coverage]: onExit called
INFO [coverage]: onExit completed ##[section]Finishing: Test
Thanks for the details @ozkoidi! I don't think pendingDisconnect is relevant, it probably happened sometime during the test execution and karma has recovered from it given that the timer is inactive (_idleTimeout: -1).
Missing onBrowserComplete called looks relevant though. junit reporter is affected by this as well. Unfortunately, I was not able to figure out how exactly could it happen from reading the code... There is a relatively sync chain of browser_complete -> browser_complete_with_no_more_retries -> run_complete -> Karma shut down, but I don't understand where it could race, so that browser_complete event is not reaching plugins at all. Without the process.kill the pending event listeners should prevent Node process from exiting and they should be called even if out of order 🤔
Can you please use the two below branches and share logs for the two cases to further investigate this problem?
$ npm i devoto13/karma#test-coverage-1
$ npm i devoto13/karma-coverate#test-coverage-1
If possible can you also attach unedited logs as files as timestamps may be useful in figuring out what is going on?
Hello again @devoto13, I tested your latest branches in the pipelines and I get the same behavior.
I attach here a zip file with the full unit tests log output (Karma's logLevel set to LOG_DEBUG).
The zip contains 2 text files (coverage-devoto-1.txt and no-coverage-devoto-1.txt). For confidentiality reasons I had to rename the tests but I don't think that should be an impediment.
I hope you find these logs useful to find the problem, thanks for your time.
Thank you for the help debuggin it, @ozkoidi! Are you sure you were running the code from test-coverage-1 branches though? I don't see any of the logs added in devoto13/karma@727469e for example.
And specific test names are irrelevant, I was primarily interested to look into the timing of the newly added logs.
Yes, I even uninstalled and re-installed karma and karma-coverage again today to be sure. You can see it in this extract from my package-lock.json:
"karma": {
"version": "git+ssh://git@github.com/devoto13/karma.git#c411e999b433a1c135305953658f9662571312a4",
"dev": true,
"from": "karma@github:devoto13/karma#test-coverage-1",
...
"karma-coverage": {
"version": "git+ssh://git@github.com/devoto13/karma-coverage.git#a7f67f27edd9ea515bfcf39e6fef2259f89360c7",
"dev": true,
"from": "karma-coverage@github:devoto13/karma-coverage#test-coverage-1",
...
I run the pipeline several times and searched for the logs you added but indeed, they never show up on the output.
If there is anything else I could do to help please let me know as this issue blocks our pipelines often.
Okey, thanks @ozkoidi! So it seems that the browser_complete event is not coming from the browser over the socket.io connection, that's interesting. I'll look more into it and get back to you if I need more information.
Thanks again for your help in debugging this.
I had the very same issue and was trying for weeks now to solve it. I was able constantly solved it with adding to karma.config.js following 2 options
browserSocketTimeout: 8 * 60 * 40000,
pingTimeout: 8 * 60 * 40000,
so the resulted file is
// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html
const isDocker = require('is-docker');
module.exports = (config) => {
config.set({
basePath: '',
frameworks: ['jasmine', '@angular-devkit/build-angular'],
plugins: [
require('karma-jasmine'),
require('karma-chrome-launcher'),
require('karma-jasmine-html-reporter'),
require('karma-junit-reporter'),
require('karma-coverage'),
require('@angular-devkit/build-angular/plugins/karma'),
],
client: {
jasmine: {
random: false,
stopOnFailure: true,
failFast: true,
},
clearContext: false, // leave Jasmine Spec Runner output visible in browser
},
junitReporter: {
outputDir: './reports',
outputFile: 'karma.xml',
useBrowserName: false,
},
coverageReporter: {
dir: './reports/coverage',
subdir: '.',
includeAllSources: true,
reporters: [
{ type: 'cobertura' },
{ type: 'lcovonly' },
{ type: 'html' },
{ type: 'text' },
],
check: {
// thresholds for all files
global: {
statements: 76.25,
branches: 55.95,
functions: 71.4,
lines: 75.8,
}
},
},
reporters: ['progress', 'kjhtml', 'junit', 'coverage'],
port: 9876,
colors: true,
logLevel: config.LOG_INFO,
autoWatch: true,
// to avoid DISCONNECTED messages
browserDisconnectTimeout: 400000, // default 2000
browserDisconnectTolerance: 6, // default 0
browserNoActivityTimeout: 8 * 60 * 40000, //default 10000
captureTimeout: 8 * 60 * 40000, //default 60000
browserSocketTimeout: 8 * 60 * 40000,
pingTimeout: 8 * 60 * 40000,
browsers: ['ChromeCustom'],
singleRun: false,
customLaunchers: {
ChromeCustom: {
base: 'ChromeHeadless',
flags: isDocker() ? ['--disable-gpu', '--no-sandbox'] : [],
},
},
});
};Thank you so much @danbilokha! After adding browserSocketTimeout and pingTimeout it hasn't failed a single time so far 🎉
I already had big values for browserDisconnectTimeout, browserNoActivityTimeout and played with browserDisconnectTolerance in the past but didn't help. However adding the properties you mentioned made the coverage reports to always be generated 👍
Even though is not an actual fix, it does unblock our pipelines, so thanks again for your suggestion :)
had same issue while running unit tests through linux docker container with low cpu specs. using browserSocketTimeout and pingTimeout worked!!!
for anyone searching like me below are our app configs.
angular v14
with nrwl nx v14
using karma