Compatibility with pytest-describe

[ROpdebee]

Describe the bug
It appears that flake8-aaa isn't compatible with the pytest-describe, most likely because the tests now don't begin with 'test_' or something similar.

Here's a test case:

"""pytest-describe incompatibility with flake8-aaa."""


def describe_list() -> None:
    """Plugin doesn't pick this up as tests."""

    def describe_append() -> None:

        def adds_to_end_of_list() -> None:
            lst = ['spam']
            lst.append('foo')
            lst.append('bar')
            assert lst == ['foo', 'bar']


def test_something() -> None:
    """Plugin is working here."""
    lst = [1, 2, 3]
    result = lst[2]
    # ^ AAA03 (+ AAA06 on this line)

    assert result == 2

The pytest-describe examples are adapted from its documentation. In short: adds_to_end_of_list is a test case, the describe_* functions group related test cases (although the prefix can be adjusted in the configuration of pytest).

Share debugging output
Please run python -m flake8_aaa on your test file and share the output for the test that is failing here. For more information see the Command line documentation

Note that this is incomplete output, and doesn't include the tests that aren't picked up by flake8-aaa, but I've included it for reference just to show that flake8-aaa works on traditional tests.

------+------------------------------------------------------------------------
16 DEF|def test_something() -> None:
17 ???|    """Plugin is working here."""
18 ARR|    lst = [1, 2, 3]
19 ACT|    result = lst[2]
           ^ AAA03 expected 1 blank line before Act block, found none
20 CMT|    # ^ AAA03 (+ AAA06 on this line)
           ^ AAA06 comment in Act block
21 BL |
22 ASS|    assert result == 2
------+------------------------------------------------------------------------
    2 | ERRORS
======+========================================================================
        FAILED with 2 ERRORS

If you are running via Flake8, please send Flake8's version signature - that's the output of flake8 --version:

3.8.4 (aaa: 0.11.0, flake-mutable: 1.2.0, flake8-annotations-complexity: 0.0.5, flake8-annotations-coverage: 0.0.4, flake8-bandit: 2.1.2, flake8-blind-except: 0.1.1, flake8-broken-line: 0.3.0, flake8-bugbear: 20.1.4,
flake8-comprehensions: 3.3.0, flake8-docstrings: 1.5.0, pydocstyle: 5.1.1, flake8-eradicate: 1.0.0, flake8-pie: 0.6.1, flake8-print: 3.1.4, flake8-pyi: 20.10.0, flake8-string-format: 0.3.0, flake8-tidy-imports: 4.1.0,
flake8-tuple: 0.4.1, flake8_builtins: 1.5.2, flake8_pep3101: 1.2.1, flake8_quotes: 3.2.0, import-order: 0.18.1, logging-format: 0.6.0, mccabe: 0.6.1, naming: 0.11.1, pycodestyle: 2.6.0, pyflakes: 2.2.0, rst-docstrings:
0.0.14, warn-symbols: 1.1.1) CPython 3.8.2 on Darwin

I can try with less plugins installed, but I doubt that it's a plugin conflict issue.

Please share the Python version and platform you are using:

• Python version (output of python --version): 3.8.2
• Platform (GNU Linux / Mac OSX / Windows): macOS

Expected behavior
It would be nice if flake8-aaa picks up those tests in the describe blocks to check as well.

For full compatibility, it probably also has to support customising the describe_ prefix, but I personally don't care much about that, since I don't use such customisation (yet). Maybe it also has to support that behaves_like decorator in their documentation, but to be honest, I'm not even sure I fully understand what's going on in that decorator, and I'm not sure how it would work together with the AAA style.

[jamescooke]

Hi @ROpdebee - thanks for opening this and taking the time to describe the new expected behaviour.

This does sound like an interesting feature - BUT - I would need more information on a few things for this to be implemented:

• A pointer on how to find describe_* functions. For example, should they be inside test files?

• What the layout of describe_* functions should be to pass tests with some examples. These "good" examples should be added to /examples/good probably in a new dir called something like describe.

• A list of errors that would be raised when checks do not pass. Each error will need to have an example in /examples/bad.

If the information about good format, new checks and error codes can be gathered and added to /docs in a branch, or even here in this issue thread, along with good and bad examples in /examples then this will make it possible to add this functionality.

I'm going to set this as needs more info

[ROpdebee]

Happy to oblige! It'll take me some time to collect/construct a bunch of examples, since I'm currently swamped with other work, but I'll update you once it's ready.

[jamescooke]

Closing this for now - feel free to reopen if and when you've got examples to share.