Proposal: multi-page html coverage report
silkentrance opened this issue · 10 comments
I am currently working on a multi page html coverage report to complement the existing single page report.
The report should adhere to the existing single page report's style, yet provide the user with navigable package index pages providing an overview on the package's overall coverage such as local hits and misses and coverage as well as total hits, misses and coverage across both the local files and those of the child packages and so on.
As for my own part, I am in desperate need of such a reporting facility as none of the existing tools such as istanbul would cover my projects implemented using coffee-script, nor would they generate the required reports based on the original coffee-script.
As such, I am using coffee-coverage to instrument my sources and from that I currently generate the single page html report, as it generates the required jscoverage data.
In order for this to become real, I need to make certain, backwards compatible, changes to the coverage report api and to the vows command itself, namely adding an additional -o, --out option to where to output the generated files to, which is currently always the CWD. In addition, I would like to add the following two options to the vows command, namely --coverage-html-single, which is actually an alias for --coverage-html, and --coverage-html-multi for the new multi page coverage report.
As for the API change, all reporters will have an additional parameter: outdir. While the plain text reporter will simply ignore the extra parameter, the other reporters will write their output to the specified directory. The directory will be created when missing, or, in case of the multi page report, the report might fail with an exception on non empty directories, preventing existing files from being overwritten.
In the long run, I would like to integrate this into a grunt task that would then configure vows so that it will consider user provided templates and stylesheets and other static resources such as scripts, images and so on.
I'd like to have some input on this, especially towards user defined style sheets, templates and so on. I am currently considering to use handlebars. If you have any objections to that, feel free to propose an alternate template engine.
i built the existing coverage reporting into vows, so i figure i should weigh in on this.
several months ago, @davglass and i had been talking about the future of vows in general. at the time, i had argued that we should continue moving the coverage forward into future versions of vows (essentially a rewrite), and he had strong opinions otherwise.
after all this time, my opinion has changed, and i am now firmly in the camp of dav. i'd rather see istanbul get extended to correctly cover coffee-script and with templates that i would like to see the existing coverage system (which is clunky) extended.
just my 2 cents.
Good point, thanks.
As of now, istanbul and others, namely jscoverage, do not support coffee-script.
Perhaps we could persuade the peops over at jscoverage and istanbul to incorporate coffee-coverage?
And, yes, the existing istanbul styles are awkward and the overall presentation diminishes its content.
that might be the best bet.
there is a pull request open with comments: gotwarlost/istanbul#167
but it looks like there is not yet source map support.
I think we should be enabling users to use whatever code coverage they want with vows via ecosystem modules. e.g. https://github.com/arikon/mocha-istanbul
@indexzero agreed, but feel that we would need to do it in a way that allows for reporters to be shared: similar to the shared testing framework reporter discussions.
this runs much deeper than vows, and i feel should be at the instrumentation level. i liken it to https://github.com/sindresorhus/jshint-stylish, except that it would entail post-test rendering of the coverage results.
what are your thoughts on that?
@JerrySievert regarding istanbul#167: they actually compile the coffee-script to javascript first and it seems that the initial initiative around this issue has somewhat come to a halt...
As for the post-test rendering, this is already part of for example the way that grunt-istanbul works, you will first instrument, then run your tests and then store a/o make the report.
That being said, coffee-coverage does create a coverage data structure that is incompatible with istanbul. Perhaps fixing this would solve our problem, instrumenting the code using coffee-coverage and making it collect the data in an instanbul compabtible way, then running vows and then having istanbul generate the reports?
That way, the coverage reporting part of vows would become obsolete, so to say...
without speaking too much for istanbul, i would be hopeful that adding source maps to it would help get past a lot of this - then they could compile without issue, as well has have true coverage.
Either way, I am continuing down this path and implement a multi page coverage report for my own use.
We would love to hear about it! You should check out the js-reporters org which is about writing test framework agnostic reporters: https://github.com/js-reporters
I think @indexzero is right that the best decision will be to let users use whatever code coverage tool they want to. This is the standard when it comes to testing and it will benefit the code base to adhere to it.