Change code coverage provider?

[emschwartz]

Right now we're using kcov / cargo-kcov (a C++ program) for collecting code coverage statistics. The advantage of kcov is that it seems to be fairly widely used for compiled languages. The disadvantage is that it's running into compilation issues right now and it's not designed specifically for rust so it sometimes tracks the coverage incorrectly.

As an alternative, we could switch to using tarpaulin instead. It is written in Rust and designed for Rust projects. The disadvantage is that it's not as widely used as kcov (as far as I can tell).

Thoughts?

[emschwartz]

A further advantage of tarpaulin is that since it's installed through cargo, it would be cached automatically by circle (see here). I tried to cache the kcov files but after spending a couple of hours on it, I couldn't get past issues related to making cached binaries executable and I gave up on it. Caching these should shave off a minute or so from CI build times.

[gakonst]

Any ideas on what broke codecov initially? Been broken at least since https://circleci.com/gh/emschwartz/interledger-rs/307?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

[emschwartz]

From looking at the CI logs, it seems like the build has been running into two main issues:

1. Can't set personality: Operation not permitted - this seems to be the same problem reported in travis-ci/travis-ci#9061 and should be fixed by c7a4499
2. note: collect2: fatal error: ld terminated with signal 9 [Killed] compilation terminated - this is caused by the VM running out of memory. I increased the RAM for the container to solve the intermittent issue it was having just running cargo test but it's still failing to run it with kcov

On a separate note, there is a discussion in a cargo-make issue about caching kcov, which if implemented would solve the performance issue. If that is added, I would switch back to using cargo-make instead of cargo-kcov to run kcov. (Note that this repo used to use cargo-make to run everything but I opted to break out the build steps so they'd show up as separate steps in the Circle interface rather than using one of the built-in cargo-make workflows. Without using the workflows, the only thing cargo-make was really doing was building and running kcov and managing the installation of other tools like cargo-audit and rustfmt)

Currently, every time I run cargo-tarpaulin locally, it exits with a segfault so I'm less sold on switching to it.

[emschwartz]

After spending another day working on this issue, I am thoroughly frustrated.

One of the main problems is definitely:

• xd009642/tarpaulin#77
• https://discuss.circleci.com/t/cargo-tarpaulin-fails/30215

I tried:

• switching the circle config in a number of different ways
• switching to TravisCI (which is much too slow)
• switching to grcov (unclear exactly how to use it on a bigger rust project)

It seems like the best options now are:

1. Disable code coverage statistics collection for now
2. Run the Circle build on a VM (as opposed to in a docker image) as suggested in https://discuss.circleci.com/t/cargo-tarpaulin-fails/30215
   - this seems to take quite a bit longer to build
   - this might not work because only medium machines are available for the VM builds and it seemed from previous runs like we would need at least medium+ or large to make running kcov with all the tests work
3. Split the coverage job from everything else and run only that on a VM while running the rest in docker
4. Figure out if it's possible to run kcov using the docker instructions inside the docker build so that we can turn off the docker security features that block the necessary operation (as described in SimonKagstrom/kcov#151)
   - we would not be able to use cargo-kcov with this so we would need to manually call kcov with the right arguments

[emschwartz]

Might be worth looking at how clap uses kcov because they've got a working setup with Travis + Codecov

[rdaniels6813]

@emschwartz Not sure if this helps, but you can try running the coverage inside a CircleCI machine using docker similar to this: https://github.com/bittrd/rust-hello-world/blob/1ae74f6b675707b31d49ed867f30ee23e3774c49/.circleci/config.yml#L38

[gakonst]

Haven't been able to get progress on this, un-assigning until further notice to focus on bugfixing