Set default --coverage-report in configuration file

Question

Set default --coverage-report in configuration file

adamjstewart opened this issue 4 months ago · 18 comments

Summary

When computing coverage, I almost always want to use --coverage-report=term-missing. Instead of typing this on the command line every time, I would like to store this default in pyproject.toml. At the moment, it's only possible to add this flag in:

[tool.pytest.ini_options]
addopts = "--cov=<project-name> --cov-report term-missing"

which isn't possible to override. Would it be possible to add a pytest-cov specific config value that changes the default?

Answer 1 · 2024-08-30T23:38:42.000Z

I think you want this:

[tool.pytest.ini_options]
addopts = "--cov=<project-name> --cov-report term-missing html"

Answer 2 · 2024-08-30T23:43:29.000Z

It's not possible to override this if I want a different option though.

Answer 3 · 2024-09-03T16:24:26.000Z

It's not possible to override this if I want a different option though.

I may be miss-understanding the docs but it seems to suggest you can put most whatever you want after the term-missing (there is an odd space in the documentation)

Specifically:

term- missing may be followed by “:skip-covered”. annotate, html, xml, json and lcov may be followed by “:DEST” where DEST specifies the output location. Use –cov-report= to not generate any output.

from here:

--cov-report=type
Type of report to generate: term, term-missing, annotate, html, xml, json, lcov (multi-allowed). term, term- missing may be followed by “:skip-covered”. annotate, html, xml, json and lcov may be followed by “:DEST” where DEST specifies the output location. Use –cov-report= to not generate any output.

Answer 4 · 2024-09-04T08:34:18.000Z

Sorry, I think the confusion stems from me including html in the initial edit instead of term-missing. I've edited the initial post to clarify.

By not possible to override, what I mean is that if you put term-missing in addopts, you can't remove/replace this on the command line, you can only append. So if I use:

> pytest --cov-report=xml

it will generate both xml and term-missing outputs.

Answer 5 · 2024-09-04T09:00:31.000Z

@adamjstewart yes, that's how it works. Which is why I started migrating to having the html coverage report set in tox.ini while XML is requested in CI, replacing it.

Answer 6 · 2024-09-04T09:02:32.000Z

By the way, I recommend having coverage-related options in .coveragerc so other tools using Coverage.py would also use the same settings. And only have --cov without args passed to pytest.

Answer 7 · 2024-09-04T09:10:57.000Z

Here's the most recent example of how I separate the args:

Answer 8 · 2024-09-04T09:14:22.000Z

@adamjstewart there's also a --cov-reset which can drop all cov args you've got on the CLI but not replace just one selectively: https://pytest-cov.rtfd.io/en/latest/config.html#reference. I can't imagine using it myself, though.

Answer 9 · 2024-09-04T09:26:31.000Z

Thanks @webknjaz! I haven't yet gotten around to adopting tox in my workflows, but I'll check it out someday. For now, I'm directly running pytest --cov=<project-name> --cov-report=term-missing from the command line and running pytest --cov=<project-name> --cov-report=xml in CI. I'm sure tox could save me a lot of typing...

I'm trying to keep as much configuration as possible in pyproject.toml and trying to minimize command-line args. I only have a couple of coverage-specific things, but I already have them in the [tool.coverage.report] section of pyproject.toml. I still need to add branch coverage, that's my next TODO.

I think you're right that what I want is a pyproject.toml like:

[tool.pytest.ini_options]
addopts = "--cov=<project-name> --cov-report=term-missing"

and CI/command-line like:

> pytest --cov-reset --cov=<project-name> --cov-report=
> pytest --cov-reset --cov=<project-name> --cov-report=xml

any time I want to override it. Not pretty, but it gets the job done. I guess what I'm really asking for is a way to set defaults for these values that can be optionally overridden on the command line instead of resetting everything completely.

Answer 10 · 2024-09-04T09:35:57.000Z

As an example of what I would like, if I use:

[tool.coverage.run]
source = ["<project-name>"]

and use:

> pytest --cov

on the command-line, it uses my default. But if I use:

> pytest --cov=<project.submodule>

it overrides this default. Is there a way to get this same behavior for the default report type, or is the only way to use addopts and --cov-reset?

Answer 11 · 2024-09-04T09:48:30.000Z

As far as I can tell, --cov-report is specific to pytest-cov, and is not included in coverage...

EDIT: I can influence this with show_missing.

Answer 12 · 2024-09-04T09:53:45.000Z

I'm now using the following pyproject.toml:

[tool.coverage.report]
show_missing = true

[tool.coverage.run]
source = ['<project-name>']

[tool.pytest.ini_options]
addopts = '--cov --'

and I think I'm pretty close to what I want. It still doesn't seem possible to override show-missing, but I think I've come to accept that.

EDIT: nevermind, --cov -- doesn't allow me to override --cov on the command line, and --cov ignores me when I ask it to only test one file.

Answer 13 · 2024-09-05T14:57:04.000Z

EDIT: nevermind, --cov -- doesn't allow me to override --cov on the command line, and --cov ignores me when I ask it to only test one file.

It should replace source if you do --cov=dir-path/. Also, coverage.py has source-pkgs which is more accurate than source. I recommend setting source = ['.'] and source_pkgs = ['importable_name'] in your config.

Answer 14 · 2024-09-05T14:58:24.000Z

If you want more flexibility, perhaps you need --cov-config= and configs for each case.

Answer 15 · 2024-09-05T15:01:02.000Z

I think I'm just going to go with:

[tool.coverage.report]
show_missing = true

[tool.coverage.run]
source = ['<project-name>']

and accept the fact that there is no way to enable coverage by default if you want to be able to customize it later.

Answer 16 · 2024-09-05T15:14:48.000Z

@adamjstewart make sure to use source_pkgs instead, though.

Still, I don't quite understand why --cov-reset doesn't help you.

Also, note that coverage.py itself has some config overrides. For example, you can put skip_covered = true in [report] and it'd normally be inherited by any non-terminal reports but setting it in [html] as skip_covered = false would override it for HTML reports but not XML or term.
That can be further overridden on the pytest-cov level.

Do you have a small demo project showing more details of what you're trying to have?

Answer 17 · 2024-09-06T15:27:25.000Z

Still, I don't quite understand why --cov-reset doesn't help you.

It does, it's just unfortunate that it's required at all. Instead of saying pytest for coverage and pytest --cov=foo.bar.baz for coverage of a single file, I need to use:

> pytest --cov-reset --cov=foo.bar.baz --cov-report=term-missing

Do you have a small demo project showing more details of what you're trying to have?

I have a massive production project: microsoft/torchgeo#2275

Answer 18 · 2024-09-10T15:16:34.000Z

Closing this issue, as:

[tool.coverage.report]
show_missing = true

is the solution to the initially asked question. I may open a separate issue someday to find a way to enable coverage by default (in a way that can be overridden).