/pytest-portion

Select a portion of the collected tests

Primary LanguagePythonBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

pytest-portion

PyPI version Python versions See Build Status on Travis CI

Select a portion of the collected tests, so you can run differents parts of your test suite in differents instances to scale horizontally.

Use case

Suppose you have a big, slow test suite, but you can trigger several CI workers to run different portions of it, in a sake lazy/simple way to parallelize it.

A basic, obvious way to do that is to explictily collect from different directories/modules

  • worker1: pytest tests/a (100 tests, ~4 minutes to finish)
  • worker2: pytest tests/b (20 tests, ~1 minute to finish)
  • worker3: pytest tests/c tests/d (30 tests, ~1 minute to finish)

The problem is that directory tests/a may have a lot more tests that tests/c plus test/d, so worker1 takes a lot more to finish.

With pytest-portion you can still split the tests in different instances, but letting the extension makes the selection in a more balanced way.

  • worker1: pytest --portion 1/3 tests (first 50 tests, ~2 minutes)
  • worker2: pytest --portion 2/3 tests (next 50 tests, ~2 minutes)
  • worker3: pytest --portion 3/3 tests (last 50 tests, ~2 minutes)

In this case, the tests of all the directories are collected, but only a third (a different one!) of them will be actually executed on each worker.

Note this balance is by number of tests, so if there is very slow tests in a particular portion, the duration may not be expected.

For a fine tuning, you could pass the portion in a more explicit way:

  • worker1: pytest --portion 0:0.5 tests (first half, 1st to 75th test)
  • worker2: pytest --portion 0.5:0.8 tests (next 30%, from 76th to 125º)
  • worker3: pytest --portion 0.8:1 tests (last 20%)

Installation

You can install "pytest-portion" via pip from PyPI:

$ pip install pytest-portion

Usage

Pass --portion <i/n> where:

  • n is the number of portions
  • i is the i-th portion to select (1 <= i <= n)

Note

If the number of tests collected is not divisible by n, the last portion will contain the rest. For instance, if you have test_1, test_2 and test_3, --portion 1/2 will run the first one, and --portion 2/2 the last 2.

Alternatively --portion start:end where:

  • start and end are the coefficient (between 0 and 1) that represent the segment of the collected tests

to select.

Contributing

Contributions are very welcome. Please ensure the coverage at least stays the same before you submit a pull request.

License

Distributed under the terms of the BSD-3 license, "pytest-portion" is free and open source software

Issues

If you encounter any problems, please file an issue along with a detailed description.

Acknowledgements