JaCoCo agent shouldn't be activated on every build
szpak opened this issue · 10 comments
Only on demand to not introduce performance overhead on simple "mvn test".
This is rather standard practice using jacoco. It adds next to no overhead. Any significant overhead here is only because this project takes 10 seconds to build without the agent and 13 with it. In fact, about 3 seconds on jacoco agent is what I've seen across many larger builds. It isn't significant enough to remove for some intented gain. A lot of people use this agent within CI systems such as jenkins. So every build is expected to run the agent in order to promote to CI. This would be slow if the entire jacoco suite was run which it is not.
with the agent -> 13.399 s
without the agent -> 10.539 s
Am I missing something important here? I had the same concerns at my day job when asked to do this about a year ago in much larger projects. After seeing it added next to no overhead I made the switch. So curious as to why this is seen as a bad thing. Of course, tests can be entirely skipped and this drops to 5 seconds. Seems too quick to matter.
Unit tests give quick feedback about state of the project (if tests fail there is probably something wrong) - I want to run them together with code compilation from Maven. Code coverage on the other hand is a little more sophisticated technique and I don't check it after every change. 30% percent overhead on every build is 30%. I know people who like to run build from Maven very often (even when using IDE) and 3 second more to wait every time can be visible.
While every build on CI should use JaCoCo and generate report (with Coveralls or something else) I would like to have an ability to disable JaCoCo on "normal" local build (like mvn test). Probably it should be disabled by default and enabled on demand locally and always turned on CI server.
I agree this adds roughly 30% overhead on this project. Further I believe this project is missing a vital execution step which adds no additional time (jacoco report). I plan for a pull request shortly for that.
Now what I do not firmly believe is that the 30% overhead for jacoco is the same no matter the project. Taking an example project off git ('waffle'), it does not add any extra time at all. My own timing testings last year resulted in a couple of seconds on builds that took 2 minutes and up to 10 minutes on their own merrit. I'd have to run those timing again but sort of expect to match waffle with no overhead currently. Lots has changed in the maven world over last year so I'm sure that has a lot to do with me not picking up differences on those large projects. Have you ran anything outside of this project to get similar timings? I guess I'm not really speaking for this project but curious in general as this applies to other projects I work on very heavily.
I haven't used Maven in my projects for ~2 years, but the first time I spotted the issue in my Gradle build for the project building with unit tests about 2 minutes JaCoCo was adding several seconds to every build. I didn't profiled it, but it was probably due to agent and its code instrumentation to catch the method calls. It may vary depending on the project. Nevertheless I prefer to disable not needed operations to get the results a little bit earlier (especially if there are side effects - I assume in Maven it could be done somehow without hacks :) ).
For me this is:
with the agent and report -> 16.400 s
without the agent -> 8.576 s
Solution to building this quicker. use -T to build modules in parallel.
mvn clean install -T 2.0C
This seems to be a sweet spot. I've got an i7 chip. I tried higher number and got a slightly faster build but essentially at this level it builds in less than 8.5 seconds.
@hazendaz Good catch - in Gradle builds I have it enabled by default. Nevertheless it is only a workaround - parallel build without JaCoCo should be even faster :).
If jacoco is used for coveralls only, the simplest solution would be to switch to cobertura.
I use it one my project and it works fine without any overhead.
Should I open a pull request to see if coveralls behaves the same?
Really see no need. Its seconds we are talking about. Machine dependent of course. Plus cobertura doesn't support jdk8 fully. It still has open issues on that. Might not matter here though since this is really sort of eol if you will on jdk8. Although certainly worth checking out for speed comparisons.
--- Original Message ---
From: "Jan-Hendrik Peters" notifications@github.com
Sent: February 19, 2015 3:10 PM
To: "Codearte/catch-exception" catch-exception@noreply.github.com
Cc: "Jeremy Landis" jeremylandis@hotmail.com
Subject: Re: [catch-exception] JaCoCo agent shouldn't be activated on every build (#4)
If jacoco is used for coveralls only, the simplest solution would be to switch to cobertura.
I use it one my project and it works fine without any overhead.
Should I open a pull request to see if coveralls behaves the same?
Reply to this email directly or view it on GitHub:
#4 (comment)
Im closing this without change because branch 2.x took only 7.817 s to fully build project and there is no PR to merge :)