Set codecov threshold and target
ivaylonikolov7 opened this issue · 1 comments
Problem
Currently codecov
fails if the test coverage drops slightly. This is an issue as our tests seem to be non-deterministic and sometimes coverage drops non-signifcantly by -0.04%, -0.1%, 0.5% etc I couldn't investigate if this is an issue with codecov itself not following all test paths or there is some randomness generated in our tests too.
This creates the perception that our main
branch is broken because the CI says it failed when it actually didn't.
Solution
My solution is that we should set a codecov
threshold of 1% so even if our tests drop slightly the CI shouldn't have this ugly red cross.
But I would like to also set a target that the maintainers of SDK should aim for so this 1% threshold doesn't get abused and we don't fall significantly in our tests coverage. Currently we have approximately 84% coverage so a realistic starting target would be 80%. In my opinion we should aim long term for an 90% target but this might take some time to reach. I think that more than 90% coverage would mean that we will have bloated tests, duplication, time wasted and tests that are hard to read.
Alternatives
disable codecov <3
complete via #2575