What's the recommended way to move a test out of a notebook into its own file using ipytest while avoiding caching issues?

[Wizek]

The readme says this:

ipytest allows you to run Pytest in Jupyter notebooks. ipytest aims to give access to the full pytest experience and to make it easy to transfer tests out of notebooks into separate test files.

Emphasis mine.

Maybe I'm too new to pytest, ipytest, or even python, but could I see what's a recommended example way to do this with on an ongoing development basis?

In my jupyter notebook I tried like this:

ipytest.run( addopts=['--assert=plain', 'tests/'] )

And this seems to work at discovering the additional test file, but strange caching issues ensue.

e.g. if I start with this:

def test_one():
    assert 2 == 3

it correctly reports the failure:

============================================ FAILURES =============================================
____________________________________________ test_one _____________________________________________

    def test_one():
>       assert 2 == 3
E       AssertionError

tests\test_misc.py:2: AssertionError
===================================== short test summary info =====================================
FAILED tests/test_misc.py::test_one - AssertionError

However when I change it to this and save:

def test_one():
    assert 2 == 2

I get this which is quite strange:

============================================ FAILURES =============================================
____________________________________________ test_one _____________________________________________

    def test_one():
>       assert 2 == 2
E       AssertionError

tests\test_misc.py:2: AssertionError
===================================== short test summary info =====================================
FAILED tests/test_misc.py::test_one - AssertionError

I tried these things so far to try to improve caching:

ipytest.clean()
ipytest.force_reoload()

%%ipytest -qq
%reload_ext autoreload
%autoreload 1

And these might have helped a bit, since for printing the file does get refreshed, but not for the actual test run.

Any ideas how else I could try? A simple canonical example of how it's recommended that ipytest is used when moving some tests out of a notebook without running into caching issues might be all that I would need.

[chmp]

Hi @Wizek,

Thanks for reporting this issue and thanks for pointing out this gap with goal and reality!

In principle, "force_reload" would work, but you need to specify which modules to reload. When pytest executes tests in subfolders it adds each test folder as a separate entry in the python path (docs). Therefore, each module is imported under its own name without any package structure. E.g., when your directory layout would read:

Notebook.ipynb
tests/test_module1.py
tests/test_module2.py

The test files would be imported as test_module1 and test_module2. To trigger the reload, you would need to specify both modules independently in the call to force_reload. In your case, the following invocation should to the trick:

ipytest.force_reload("test_module1", "test_module2")
ipytest.run('--assert=plain', 'tests')

You can see how pytest works by looking at sys.path and sys.modules. In this example you will see that each pytest invocation adds ["./tests", "."] to the python path and the modules test_module1 and test_module2 are found in sys.modules.

However, there are a couple of issues here , that should be fixed in ipytest for better usability:

• #118
• #119
• #120
• restore the python path after executing pytest (otherwise the python path grows with each invocation of pytest) (probably rather an issue for pytest)

[chmp]

I added additional docs and an improved force_reload implementation in this PR. Do you think this solves the issue you brought up?

[chmp]

In particular, I would be interested, whether the following wording clarifies the underlying issue and how to address it:

Note that local test files are imported by pytest and fall under the same restriction as other modules. To test the most recent version of the tests, the corresponding modules need to be reloaded. The import behavior depends on both whether the tests are organized as packages (are __init__.py files present?) and the pytest configuration. For the default pytest configuration, ipytest can be configured with ipytest.autoconfig(force_reload="test_*"), assuming test modules prefixed with test_ not organized into packages, or ipytest.autoconfig(force_reload="tests"), assuming tests grouped in a tests package.

[Wizek]

That sounds like a quite elegant yet simple-for-now solution, (even if it sounds like it's 'only' 99% seamless) and a quite understandable explanation, thank you! The doc seems nice too.

I needed to solve the initial problem quick, so I ended up moving all of the previous cell contents into its own py file, and I enabled running pytest on save in an IDE-attached terminal, so it ended up working quite nice that way. It was more easy to split out groups of tests into their own py files afterward, and re-runs on file save was nice too. Maybe this is fine for now, since that original ipy cell grew almost 1k LOC and was starting to be unwieldy in more ways too.

So for now my issue is resolved, and I can report on my experience on using this solution next time I reach for ipytest, let's hope it's soon!

[chmp]

Thanks a lot for the feedback!