Parallel invocations should not be failing the build
cspotcode opened this issue · 8 comments
I'm trying to set up coveralls reporting so that we do multiple parallel invocations, and the final aggregate will pass / fail based on coverage increase/decrease.
Here is where I'm trying to set this up.
TypeStrong/ts-node#1040
The workflow configuration:
https://github.com/TypeStrong/ts-node/blob/a1f5bc8d99f5473ac1716d0c0cc9a5e12f7c0f8a/.github/workflows/continuous-integration.yml#L55-L75
I run a matrix of tests with parallel: true and continue-on-error: true. I have a single parallel-finished: true invocation to aggregate the results. Only the latter should pass/fail.
Am I don't something wrong? Why are the parallel invocations failing my build? I want them to succeed if coverage is successfully reported, ignoring any increase / decrease.
I think I have the exact same problem.
OGGM/oggm#1033
All the parts of the parallel build fail the build, despite the final result being successful.
How does one tell it not to report commit status?
The documentation says that this happens when you set the flag-name, but not setting that breaks the parallel build even more, making it fail to associate individual parts to the whole build.
@TimoRoth for what it's worth, we ended up going with CodeCov because it fit our use-case better. We were seeing some confusing behavior from coveralls -- reporting coverage decreased / increased when it seemed like it had actually stayed the same -- and we needed path rewriting in the coverage report, too.
@TimoRoth Your problem with it failing to associate individual parts was due to you using the non-unique GITHUB_RUN_NUMBER as a supposed to be unique service_job_id at https://github.com/coveralls-clients/coveralls-python/pull/227/files#diff-aff091311b4a5401e4fbbbedcf1e7b91R102. When I use an empty string (I hope None should work as well but I wanted to try this without touching python-coveralls), the builds are associated just fine.
It's not the only fix necessary, however, as coveralls won't let me finalize the build via webhook using GITHUB_TOKEN, but that's an unrelated issue I think.
When I tried that, it lead to coveralls not associating any but one of the partial jobs with my code, losing large chunks of the coverity results.
@TimoRoth Well that is exactly what I'm experiencing when I use your code unmodified. Setting GITHUB_RUN_NUMBER to empty is what makes it work again. It is possible that setting the flag name is another workaround, however, as I see you tried that as well.
So coveralls must have changed something on their end then.
Changing it like that will also break finishing the parallel job with the official coveralls action.
Cause they set(at https://github.com/coverallsapp/github-action/blob/master/src/run.ts#L47) COVERALLS_SERVICE_JOB_ID to GITHUB_RUN_ID.
Which then laters(at https://github.com/coverallsapp/github-action/blob/master/node_modules/coveralls/lib/getOptions.js#L150) gets passed as 'service_job_id'.
While seemingly leaving 'service_job_number' entirely unset.
Something does not seem to match up there.
Changing it like that will also break finishing the parallel job with the official coveralls action.
Cause they set(at https://github.com/coverallsapp/github-action/blob/master/src/run.ts#L47) COVERALLS_SERVICE_JOB_ID to GITHUB_RUN_ID.
Which then laters(at https://github.com/coverallsapp/github-action/blob/master/node_modules/coveralls/lib/getOptions.js#L150) gets passed as 'service_job_id'.
While seemingly leaving 'service_job_number' entirely unset.
Something does not seem to match up there.
Well I can't get finishing the parallel job to work even without the change so I am not sure if the change I'm suggesting is the one that breaks it... :-)
I'm thinking that perhaps I'm experiencing lemurheavy/coveralls-public#1407 or something because the webhook endpoint doesn't give me an error for the build id whereas it does if I manually corrupt the build id.
You're right though that they do indeed pass the non-unique GITHUB_RUN_IDas service_job_id, which probably explains why their documentation says that flag-name is required for parallel jobs, as it really wouldn't work otherwise.
There's something we're missing, either way. May be a bug on their side, may be something else entirely. :-/
Have there been any advancements since July? We have some jobs failing due to decreased coverage every time we add code that will only be run in one job.
I've seen some repositories where coveralls only shows the summary (here for example). They all don't specify a flag-name in parallel runs as opposed to what the documentation of this action says.
We did the same, and it worked in this PR:
However, after merging into master, the runs only report one job each which greatly decreases our total coverage.