Continuous Integration practices enable the frequent and verified (built and tested) contribution of code to shared repositories. But, as projects grow, so do their test suites and the time it takes to run them. Small and contained changes do not require all tests to be run, most of them simply aren't relevant. To improve developer productivity and reduce resource usage, we have implemented Partial Testing. By recording coverage data from previous runs, it identifies which tests should be run for each new change or pull-request.
To ensure that coverage data remains up-to-date, and that all tests are still occasionally run, master branch builds execute the entire test suite while recording coverage at the same time. The coverage data from the master branch is saved as a .coverage file (which is a sqlite DB file from coveragepy>=5.0, details available here) and non-master branches utilize that file to run only the relevant tests for the changes included in the branch.
For simplicity, we consider a file as the smallest possible changed unit (instead of doing it at the line level). That is, if a line has changed, it is treated as if the entire file has changed. This approach reduces the complexity of Partial Testing and provides a wider safety net against filtering out tests that were actually relevant.
As shown on the image above, once a file has been modified, a few different testing scenarios arise according to the type of the file. It is also possible that a combination is required. The different file types are as follows:
- Partial Testing only works with projects that are in git.
Includes any file that contains code and is not part of the test suite. They usually have the Python extension, .py, but could also have .cpp, .c, etc. instead.
When modified:
a) check the saved coverage data to find all relevant tests (test names).
b) For each test name, find which file has its definition.
c) feed pytest (or any other testing tool) the files from step b.
When a new file is added, the complete test suite must be run.
Includes any file that contains code and is part of the test suite, by default it includes all python files under tests/. They have the Python extension .py. When modified, run all the tests contained within the changed file.
When a new test file is added, all the tests within itself must be run. When one is deleted, no tests from it need to be run.
Additionally, some files under the tests directory might be utility files that get imported by other test files. Because of this, for modified test files, we need 2 steps:
a) check which tests use the file (just like non-test code files)
b) run the tests within the modified file
Includes any file that does not contain code but affects the way code is run. Examples: setup.cfg
When modified or when a new one is added, all tests need to be run.
Each project should have the ability to specify which files and/or file extensions should be considered special files, see below for details.
We have included conftest.py (pytest) under this category of files, even though it initially looks like a code file (.py) or a test file (it sits under the test directories).
Includes any file that does not contain code. For example, README or .md files.
When modified or a new one is added, no tests need to be run.
In Summary,
| Source Code Files | Test Files | Non-Code Files | Special/Config Files | |
|---|---|---|---|---|
| Extension | .py, .cpp, .c | .py | .md | .cfg |
| Modified | Run tests that use the file | Run tests a) within modified file b) that use the file | Don't run tests | Run all tests |
| Added New | Run all tests | Run tests within new file | Don't run tests | Run all tests |
| Deleted | Run tests that use the file | Do not run that file | Don't run tests | Run all tests |
Installation
pip install partialtesting
Once installed, the binaries partialtest and partialtesting (they are identical, the later is kept for compatibility) become available in your virtual environment:
$ partialtesting --help
Usage: partialtesting [OPTIONS]
Partial Testing (PT) identifies which tests need to be run for a given
change set, improving the speed of testing, developer productivity and
resource usage. To achieve this, PT relies on enriched coverage data
generated before hand, generally by a master-branch build running on a CI
(e.g. Jenkins).
PT expects to find the coverage data in a directory with the below
pattern: <coverage_dir>/<project_name>. Use the options for this script to
specify them
More information available in the README.
Options:
--coverage-dir TEXT Path to the saved coverage data. Set a default
path by setting the below in ~/.partialtesting:
[coverage] dir=<path>
--project-name TEXT Project name (e.g. numpy).The name will be used
to get the path to the coverage data for this
project:
<coverage_dir>/<project_name>/.../.coverage
[required]
This sections shows how to use partialtesting locally, but its convenience and benefits are maximized when used with a continuous integration system (CI), like Jenkins. That will be explained in the section below.
To use partialtesting locally, specify the directory in which the coverage data will be stored. That is, the coverage data generated during a previous run of the entire test suite or from a master build, more details in the CI section above.
As an example, say your project's code repository (git) is as follows:
$ tree
├── code1.py
├── code2.py
├── .coveragerc # explained below
└── tests
├── __init__.py
└── test_code1.py
and the files have the following content:
$ cat code1.py
def myfunc1():
print('in myfunc1')
def myfunc2():
print('in myfunc2')
myfunc3()
def myfunc3():
print('in myfunc3')
$ cat code2.py
def myfunc4():
print('in myfunc4')
$ cat tests/test_code1.py
import code1
def test_code1():
code1.myfunc1()
def test_code2():
code1.myfunc2()
Now, run the entire test suite to generate the coverage data that partialtesting will use.
$ pip install coverage>=5 pytest
$ PYTHONPATH='.' coverage --branch run -m pytest tests/
================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.8.1, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /Users/fersarr/workspace/fake_project
collected 2 items
tests/test_code1.py .. [100%]
=================================================================================== 2 passed in 0.01s ====================================================================================
Make sure you have coverage>=5.0 and a .coveragerc file telling coverage to record test contexts (read more here)
$ cat .coveragerc
[run]
dynamic_context = test_function
The coverage ... pytest command should have generated a .coverage file, which is a SQLite database:
$ sqlite3 .coverage ".tables"
arc coverage_schema line_bits tracer
context file meta
Now, move the .coverage file into a directory that will hold coverage data for this project:
$ mkdir -p saved_coverage/fake_project/ && cp .coverage saved_coverage/fake_project/
$ tree
├── saved_coverage
│ └── fake_project
│ └── .coverage
Okay, everything is set and ready for partialtesting to work, so let's modify a file and run PT.
Modified a code file by adding one line:
$ git diff
diff --git a/code1.py b/code1.py
index faecfed..40f81cc 100644
--- a/code1.py
+++ b/code1.py
@@ -1,5 +1,6 @@
def myfunc1():
print('in myfunc1')
+ print('changing myfunc1')
Run partialtesting:
$ partialtest --coverage-dir saved_coverage --project-name fake_project --compare-to-branch master
...
INFO:root:Partial Testing: using coverage file 'saved_coverage/fake_project/.coverage'
...
INFO:root:Creating file test_files_to_run.txt
INFO:root:Partial Testing: relevant test files:
tests/test_code1.py
As PT tells us, for the change we made, we need to run the tests in tests/test_code1.py.
As mentioned above, partialtesting can be used in a Continous Integration system (CI), like Jenkins to improve resource usage and reduce build time. The process is similar to running it locally as explained in the section above, but the difference lies in setting up the master branch to produce the coverage data .coverage and saving it in an accessible directory so that non-master branches can run partialtesting and use it.
- Run the full test suite with coverage in the master branch and save the
.coveragefile to a path:
$ coverage run -p --branch pytest tests/
$ cp .coverage jenkins/saved_coverage/project_x/907/ # build number 907
Reminder: add/create a .coveragerc file as explained above to save test contexts with the coverage data.
- Use
partialtestingon the non-master branch to get a list of the tests that should be run given the changes in the branch.
partialtesting --project-name project_x --coverage-dir /jenkins/saved_coverage/ --git-diff-use-head --special-files "'setup.py', 'setup.cfg'" --special-extensions "'.pkl', '.h5', '.csv'"
The list of tests that need to be run is output to the file test_files_to_run.txt (customisable via --output-file)
Feed those tests to pytest or your preferred testing tool.
Partial Testing has been under active development at Man Alpha Tech since 2019.
Original concept and implementation: Fernando Saravia Rajal
Contributors:
who-tests-what is here coverage v5
who-tests-what empty context names