regression in passing environments via TOXENV

[rienafairefr]

Hey devs,

in our project here we use tox to unit test, and we experienced a drop in code coverage which went unnoticed, and I finally bisected the problem to b6de343, first released in version 3.7.0

The issue is that when passing environments to test (via TOXENV, but the problem might happen when passing the environments to tox differently, not sure), then duplicated environments are not run.

Example:

Before, TOXENV="py27-tests,coverage_upload,py35-tests,coverage_upload" tox would run two tests, uploading the coverage after each python version

Afterwards, up to the most recent version, the coverage_upload env is run only once. I'm not sure where the test environments are dropped or de-duplicated, but it's happening.

I'll probably rewrite the tox.ini so I don't rely on tox running environments twice like that, but I think you should be aware :-)

[gaborbernat]

I would say the expectation here would be to define coverage_upload depends on the others; then that would run last and should upload all existing coverages 🤔

[rienafairefr]

As far as I remember, the issue was coverage_upload overriding each run, rewriting each time.
We can aggregate coverages locally, upload once, or upload many times and let codecov do the aggregation. Both approaches can be valid I think.

[gaborbernat]

I was thinking of coverage_upload uploads each coverage report separately, sequentially; rather than calling multiple time the env and caring about order.