pytest-idempotent

Pytest plugin for testing the idempotency of a function.

Usage

pip install pytest-idempotent

Documentation

Suppose we had the following function, that we (incorrectly) assumed was idempotent. How would we write a test for this?

First, we can label the function with a decorator:

# abc.py
from pytest_idempotent import idempotent  # or use your own decorator! See below.

@idempotent
def func(x: list[int]) -> None:
    x += [9]

Note: this function is not idempotent because calling it on the same list x grows the size of x by 1 each time. To be idempotent, we should be able to run func more than once without any adverse effects.

We can write an idempotency test for this function as follows:

# tests/abc_test.py
import pytest

@pytest.mark.idempotent
def test_func() -> None:
    x: list[int] = []

    func(x)

    assert x == [9]

Adding the @pytest.mark.idempotent mark automatically splits this test into two - one that tests the regular behavior, and one that tests that the function can be called twice without adverse effects.

❯❯❯ pytest

================= test session starts ==================
platform darwin -- Python 3.9.2, pytest-6.2.5
collected 2 items

tests/abc_test.py .F                     [100%]

=====================  FAILURES ========================
------------- test_func[idempotency-check] -------------

    @pytest.mark.idempotent
    def test_func() -> None:
        x: list[int] = []

        func(x)

>       assert x == [9]
E       assert [9, 9] == [9]
E         Left contains one more item: 9
E         Use -v to get the full diff

tests/abc_test.py:19: AssertionError
=============== short test summary info ================
FAILED tests/abc_test.py::test_func[idempotency-check]
  - assert [9, 9] == [9]
============= 1 failed, 1 passed in 0.16s ==============

How It Works

Idempotency is a difficult pattern to enforce. To solve this issue, pytest-idempotent takes the following approach:

Introduce a decorator, @idempotent, to functions.
- This decorator serves as a visual aid. If this decorator is commonly used in the codebase, it is much easier to consider idempotency for new and existing functions.
- At runtime, this decorator is a no-op.
- At test-time, if the feature is enabled, we will run the decorated function twice with the same parameters in all test cases.
- We can also assert that the second run returns the same result using an additional parameter to the function's decorator: @idempotent(equal_return=True).
For all tests marked using @pytest.mark.idempotent, we run each test twice: once normally, and once with the decorated function called twice.
- Both runs need to pass all assertions.
- We return the first result because the first run will complete the processing. The second will either return exact the same result or be a no-op.
- To disable idempotency testing for a test or group of tests, add the Pytest marker: @pytest.mark.idempotent(enabled=False)

Enforcing Tests Use `@pytest.mark.idempotent`

By default, any test that calls an @idempotent function must also be decorated with the marker @pytest.mark.idempotent.

To disable idempotency testing for a test or group of tests, use: @pytest.mark.idempotent(enabled=False), or add the following config to your project:

def pytest_idempotent_enforce_tests() -> bool:
    return False

To disable enforced idempotency testing for a specific function, you can also pass the flag into the decorator:

# abc.py
from pytest_idempotent import idempotent

@idempotent(enforce_tests=False)
def func() -> None:
    return

Or, you can automatically add the marker based on the test name by adding to conftest.py:

# conftest.py
def pytest_collection_modifyitems(items):
    for item in items:
        if "idempotent" in item.nodeid:
            item.add_marker(pytest.mark.idempotent)

@idempotent decorator

By default, the @idempotent decorator does nothing during runtime. We do not want to add overhead to production code to run tests.

from typing import Any, Callable, TypeVar

_F = TypeVar("_F", bound=Callable[..., Any])


def idempotent(func: _F) -> _F:
    """
    No-op during runtime.
    This marker allows pytest-idempotent to override the decorated function
    during test-time to verify the function is idempotent.
    """
    return func

To use your own @idempotent decorator, you can override the pytest_idempotent_decorator function in your conftest.py to return the module path to your implementation.

# conftest.py
# Optional: you can define this to ensure the plugin is correctly installed
pytest_plugins = ["pytest_idempotent"]


def pytest_idempotent_decorator() -> str:
    # This links to my custom implementation of @idempotent.
    return "src.utils.idempotent"