LCOV 2.1 not compatible with Codecov
albinahlback opened this issue · 9 comments
At FLINT we have used LCOV along with Codecov to get code coverage. Yesterday I tried to switch to LCOV 2.1, but that did not work on Codecov's end. Not sure who's to blame, I just know that switching back to LCOV 2.0 works. Here are the commits, changing it back to LCOV 2.0: flintlib/flint@13e69eb and flintlib/flint@b63929b.
Any guess to why this is?
While still on the topic: Do you know why --omit-lines in LCOV 2.0 seems to work with Codecov? We want to avoid including aborting functions, and so we push --omit-lines "flint_throw", but that doesn't seem to have an effect in Codecov. I think it worked first, but doesn't seem to work now.
Without some more context, I fear that I have no idea :-(
lcov/2.1 is more picky than 2.0, and also checks some additional things; this could be tripping you up.
Do you have a log which shows errors and/or warnings? Can you download v2.1 and then execute the capture and/or genhtml steps in your sandbox - to see what is happening?
Unrelated - but I noticed in your changelog that you are omitting lines which contain only close brace.
lcov --filter brace ... will do something similar - but will leave braces which it thinks may be significant (e.g., implicit 'else' destinations). While I definitely agree that this is a hack - it is a slightly less dangerous hack than blanket removal.
Thanks for your quick response!
Unrelated - but I noticed in your changelog that you are omitting lines which contain only close brace.
lcov --filter brace ...will do something similar - but will leave braces which it thinks may be significant (e.g., implicit 'else' destinations). While I definitely agree that this is a hack - it is a slightly less dangerous hack than blanket removal.
I think we only had a problem with functions ending with an if-else statement, hence the regex ^}$ (I think it was mainly void type functions). But thanks for the tip!
Do you have a log which shows errors and/or warnings? Can you download v2.1 and then execute the capture and/or genhtml steps in your sandbox - to see what is happening?
Here is the Codecov error (yes, not a very good error code). Here is the associated runner. Based on the runner log, it looks exactly the same as with 2.0, so it appears as if it works.
But I will download both versions locally and try to identify the difference.
I fear that I don't know what I am looking at :-( - but when I check the 'Ubuntu gcc code coverage x10" page in the above link - it appears that geninfo was able to capture the coverage data (though, with quite a few 'inconsistent' warnings).
Which build failed? Or is this job after reverting back to lcov/2.0?
This was the one with 2.1. I am currently running all tests locally to see what the difference is on my system.
I ran both LCOV 2.0 and LCOV 2.1 on my system, and I pasted the output from them below. I took the releases here on Github.
Command-line output from LCOV 2.0
$ lcov -j $(expr $(nproc) + 1) --capture --rc geninfo_unexecuted_blocks=1 --ignore-errors unused --omit-lines "flint_throw" --omit-lines '^}$' --exclude "/usr/*" --exclude "src/*/inlines.c" --exclude "src/test/main.c" --exclude "src/test/t-*.c" --exclude "src/*/test/main.c" --exclude "src/*/test/t-*.c" --exclude "src/generic_files/exception.c" --exclude "src/generic_files/profiler.c" --exclude "src/profiler.h" --directory build --output-file build/lcov_2.0/coverage.info
Capturing coverage data from build
geninfo cmd: '/usr/local/bin/geninfo build --output-filename build/lcov_2.0/coverage.info --ignore-errors unused --rc geninfo_unexecuted_blocks=1 --exclude /usr/* --exclude src/*/inlines.c --exclude src/test/main.c --exclude src/test/t-*.c --exclude src/*/test/main.c --exclude src/*/test/t-*.c --exclude src/generic_files/exception.c --exclude src/generic_files/profiler.c --exclude src/profiler.h --omit-lines flint_throw --omit-lines ^}$ --parallel 5 --memory 0'
Found gcov version: 13.2.1
Using intermediate gcov format
Writing temporary data to /tmp/geninfo_datl5_N
Scanning build for .gcda files ...
Found 5121 data files in build
files remaining: 4921 (61.77 files/s - predict 1.33 minutes remaining)
files remaining: 4721 (54.94 files/s - predict 1.43 minutes remaining)
files remaining: 4521 (49.44 files/s - predict 1.52 minutes remaining)
files remaining: 4321 (44.30 files/s - predict 1.63 minutes remaining)
files remaining: 4121 (36.97 files/s - predict 1.86 minutes remaining)
files remaining: 3921 (33.41 files/s - predict 1.96 minutes remaining)
files remaining: 3721 (28.77 files/s - predict 2.16 minutes remaining)
files remaining: 3521 (26.61 files/s - predict 2.21 minutes remaining)
files remaining: 3321 (23.51 files/s - predict 2.35 minutes remaining)
files remaining: 3121 (22.07 files/s - predict 2.36 minutes remaining)
files remaining: 2921 (19.62 files/s - predict 2.48 minutes remaining)
files remaining: 2721 (18.91 files/s - predict 2.40 minutes remaining)
files remaining: 2521 (18.24 files/s - predict 2.30 minutes remaining)
files remaining: 2321 (16.80 files/s - predict 2.30 minutes remaining)
files remaining: 2121 (16.22 files/s - predict 2.18 minutes remaining)
files remaining: 1921 (15.71 files/s - predict 2.04 minutes remaining)
files remaining: 1721 (14.75 files/s - predict 1.95 minutes remaining)
files remaining: 1521 (13.90 files/s - predict 1.82 minutes remaining)
files remaining: 1321 (13.10 files/s - predict 1.68 minutes remaining)
files remaining: 1121 (12.82 files/s - predict 1.46 minutes remaining)
files remaining: 921 (12.54 files/s - predict 1.22 minutes remaining)
files remaining: 721 (12.29 files/s - predict 0.98 minutes remaining)
files remaining: 521 (12.01 files/s - predict 0.72 minutes remaining)
files remaining: 321 (11.72 files/s - predict 0.46 minutes remaining)
files remaining: 121 (11.46 files/s - predict 0.18 minutes remaining)
Excluded data for 3326 files due to include/exclude options
Finished .info-file creation
Omitted 18013 total lines matching 2 '--omit-lines' patterns
geninfo: WARNING: ('unused') 'exclude' pattern 'src/generic_files/exception.c' is unused.
geninfo: WARNING: ('unused') 'exclude' pattern 'src/generic_files/profiler.c' is unused.
(use "geninfo --ignore-errors unused,unused ..." to suppress this warning)
Command-line output from LCOV 2.1
$ lcov -j $(expr $(nproc) + 1) --capture --rc geninfo_unexecuted_blocks=1 --ignore-errors unused --omit-lines "flint_throw" --omit-lines '^}$' --exclude "/usr/*" --exclude "src/*/inlines.c" --exclude "src/test/main.c" --exclude "src/test/t-*.c" --exclude "src/*/test/main.c" --exclude "src/*/test/t-*.c" --exclude "src/generic_files/exception.c" --exclude "src/generic_files/profiler.c" --exclude "src/profiler.h" --directory build --output-file build/lcov_2.1/coverage.info
Capturing coverage data from build
geninfo cmd: '/usr/local/bin/geninfo build --output-filename build/lcov_2.1/coverage.info --ignore-errors unused --rc geninfo_unexecuted_blocks=1 --exclude /usr/* --exclude src/*/inlines.c --exclude src/test/main.c --exclude src/test/t-*.c --exclude src/*/test/main.c --exclude src/*/test/t-*.c --exclude src/generic_files/exception.c --exclude src/generic_files/profiler.c --exclude src/profiler.h --omit-lines flint_throw --omit-lines ^}$ --parallel 5'
Found gcov version: 13.2.1
Using intermediate gcov format
geninfo: WARNING: (usage) branch filter enabled but branch coverage not enabled
(use "geninfo --ignore-errors usage,usage ..." to suppress this warning)
Recording 'internal' directories:
/home/albin/cpkg/flint/build
Writing temporary data to /tmp/geninfo_datXM5e
Scanning build for .gcda files ...
Found 5121 data files in build
using: chunkSize: 819, nchunks:7, intervalLength:256
elapsed:0.4m: remaining:4302 files 2.3m: 31.27 files/s 0.03 s/file (interval:31.27 f/s 0.03 s/f)
elapsed:0.4m: remaining:3483 files 0.9m: 62.16 files/s 0.03 s/file (interval:5216.93 f/s 0.03 s/f)
elapsed:0.4m: remaining:2664 files 0.5m: 92.54 files/s 0.03 s/file (interval:4082.04 f/s 0.03 s/f)
elapsed:0.4m: remaining:1845 files 0.3m: 122.31 files/s 0.03 s/file (interval:3503.05 f/s 0.03 s/f)
elapsed:0.5m: remaining:1026 files 0.1m: 147.52 files/s 0.03 s/file (interval:840.45 f/s 0.03 s/f)
elapsed:0.7m: remaining:0 files 0.0m: 124.43 files/s 0.03 s/file (interval:76.59 f/s 0.02 s/f)
Finished processing 5121 GCDA files
Apply filtering..
Filter: chunkSize 100 nChunks 52
Finished filter file processing
Finished .info-file creation
Summary coverage rate:
source files: 5191
lines.......: 87.4% (266609 of 305178 lines)
functions...: 79.7% (13941 of 17498 functions)
Omitted 36026 total lines matching 2 '--omit-lines' patterns
Message summary:
1 warning message:
usage: 1
However, by a simple diff I can see that there is no difference between the outputs, which should mean that the error is either on my part (most likely, I suppose) or Codecov.
I will look into it some more, then perhaps close it.
Looks like there is a counting bug in either lcov/2.0 or lcov/2.1 as well: the number of omitted line matches is exactly doubled. Looks suspicious (though maybe not hugely important).
I don't think there are any format differences between the data generated by the two version - but you might want to diff those as well (maybe you already have).
Looks like there is a counting bug in either lcov/2.0 or lcov/2.1 as well: the number of omitted line matches is exactly doubled. Looks suspicious (though maybe not hugely important).
Yes, I noticed it as well. I did run LCOV directly one after the other -- does this affect the result in any way?
I don't think there are any format differences between the data generated by the two version - but you might want to diff those as well (maybe you already have).
I'm a bit of a novice when it comes to this. I checked the command-line output as well as the file related to --output-filename. What other type of files should I check?
Yes, I noticed it as well. I did run LCOV directly one after the other -- does this affect the result in any way?
No - should not make any difference.
If you diffed the stdout/stderr and also your coverage.info file - then that's it. Nothing else to look at (unless you are interested to really get into the weeds of temporary/intermediate files).
I'm not familiar with codecov - but I suspect that it is only interested in the .info file - maybe also the stdout and stderr logs as well, if it is checking for unexpected messages/unexpected behaviour.
It almost certainly does not care about details of processing order, record order in the output files, etc.
Thanks for your all of your inputs. It looks like it is on Codecov's end in this case.