Coverage.jl

Release version:

Development version:

"Take Julia code coverage and memory allocation results, do useful things with them"

Code coverage: Julia can track how many times, if any, each line of your code is run. This is useful for measuring how much of your code base your tests actually test, and can reveal the parts of your code that are not tested and might be hiding a bug. You can use Coverage.jl to summarize the results of this tracking, or to send them to a service like Coveralls.io or Codecov.io.

Memory allocation: Julia can track how much memory is allocated by each line of your code. This can reveal problems like type instability, or operations that you might have thought were cheap (in terms of memory allocated) but aren't (i.e. accidental copying).

Working locally

Code coverage

Step 1: Navigate to your test directory, and start julia like this:

julia --code-coverage=user

or, if you're running Julia 0.4 or higher,

julia --code-coverage=user --inline=no

(Turning off inlining gives substantially more accurate results, but may slow down your tests.)

Step 2: Run your tests (e.g., include("runtests.jl")) and quit Julia.

Step 3: Navigate to the top-level directory of your package, restart Julia (with no special flags) and analyze your code coverage:

using Coverage
# defaults to src/; alternatively, supply the folder name as argument
coverage = process_folder()
# Get total coverage for all Julia files
covered_lines, total_lines = get_summary(coverage)
# Or process a single file
@show get_summary(process_file("src/MyPkg.jl"))

The fraction of total coverage is equal to covered_lines/total_lines.

To discover which functions lack testing, browse through the *.cov files in your src/ directory and look for lines starting with - or 0 - those lines were never executed. Numbers larger than 0 are counts of the number of times the respective line was executed.

Memory allocation

Start julia with

julia --track-allocation=user

Then:

Run whatever commands you wish to test. This first run is to ensure that everything is compiled (because compilation allocates memory).
Call clear_malloc_data() (or, if running julia 0.4 or higher, Profile.clear_malloc_data())
Run your commands again
Quit julia

Finally, navigate to the directory holding your source code. Start julia (without command-line flags), and analyze the results using

using Coverage
analyze_malloc(dirnames)  # could be "." for the current directory, or "src", etc.

This will return a vector of MallocInfo objects, specifying the number of bytes allocated, the file name, and the line number. These are sorted in increasing order of allocation size.

LCOV export

There are many tools to work with LCOV info-format files as generated by the geninfo tool. Coverage.jl can generate these files:

coverage = process_folder()
LCOV.writefile("coverage/lcov.info", coverage)

Cleaning up .cov files

When using Coverage.jl locally, over time a lot of .cov files can accumulate. Coverage.jl provides the clean_folder and clean_file methods to either clean up all .cov files in a directory (and subdirectories) or only clean the .cov files associated with a specific source file.

Tracking Coverage with Codecov.io

Codecov.io is a test coverage tracking tool that integrates with your continuous integration servers (e.g. TravisCI) or with HTTP POSTs from your very own computer at home.

Enable Codecov.io for your repository. If it is public on GitHub and you are using using TravisCI, this is all you need to do. You can sign into Codecov using your Github identity. You will be served a REPO_TOKEN. You'll need this if you're not using a CI solution.
Use the command line option when you run your tests

Either with something like julia --code-coverage test/runtests.jl, or
with something like julia -e 'Pkg.test("MyPkg", coverage=true)'

Add the following to the end of your .travis.yml or .appveyor.yml file. This line downloads this package, collects the per-file coverage data, then bundles it up and submits to Codecov. Coverage.jl assumes that the working directory is the package directory, so it changes to that first (so don't forget to replace MyPkg with your package's name!

On Travis CI:

after_success:
- julia -e 'cd(Pkg.dir("MyPkg")); Pkg.add("Coverage"); using Coverage; Codecov.submit(process_folder())'

On AppVeyor:

after_test:
- C:\projects\julia\bin\julia -e "cd(Pkg.dir(\"MyPkg\")); Pkg.add(\"Coverage\"); using Coverage; Codecov.submit(process_folder())"

If you're running coverage on your own machine and want to upload results to Codecov, make a bash script like the following:

#!/bin/bash
REPO_TOKEN=$YOUR_TOKEN_HERE julia -e 'cd(Pkg.dir("MyPkg")); using Coverage; Codecov.submit_token(process_folder())'

Tracking Coverage with Coveralls.io

Coveralls.io is a test coverage tracking tool that integrates with your continuous integration solution (e.g. TravisCI).

Enable Coveralls.io for your repository. If it is public on GitHub and you are using TravisCI, this is all you need to do. If you are using AppVeyor, you need to add a secure environment variable called REPO_TOKEN to your .appveyor.yml (see here). Your repo token can be found in your Coveralls repo settings. If neither of these are true, please submit an issue, and we can work on adding additional functionality for your use case.
You must be using Julia 0.3 or higher, which added the --code-coverage command line argument.
Use the command line option when you run your tests

Either with something like julia --code-coverage test/runtests.jl, or
with something like julia -e 'Pkg.test("MyPkg", coverage=true)'

Add the following to the end of your .travis.yml or .appveyor.yml file. This line downloads this package, collects the per-file coverage data, then bundles it up and submits to Coveralls. Coverage.jl assumes that the working directory is the package directory, so it changes to that first (so don't forget to replace MyPkg with your package's name!

On Travis CI:

after_success:
- julia -e 'cd(Pkg.dir("MyPkg")); Pkg.add("Coverage"); using Coverage; Coveralls.submit(process_folder())'

On AppVeyor:

after_test:
- C:\projects\julia\bin\julia -e "cd(Pkg.dir(\"MyPkg\")); Pkg.add(\"Coverage\"); using Coverage; Coveralls.submit(process_folder())"

Julia packages using Coverage.jl

Pull requests to add your package welcome (or open an issue)

platawiec/Coverage.jl