microsoft/playwright-pytest

pytest-playwright--parameterization， cases order Not what I expected

crazy-yuzhao opened this issue · 8 comments

crazy-yuzhao commented

Using pytest-playwright, after parameterization, the execution order of use cases is inconsistent with pytest itself.this is bug！！

crazy-yuzhao commented

import pytest
from playwright.sync_api import Page, expect

@pytest.mark.parametrize("x", ["aaa", "bbb"])
def test_a(x, page: Page):
page.goto("https://www.baidu.com")
page.locator("#kw").fill(x)
page.locator("#su").click()
print(page.title())

@pytest.mark.parametrize("y", ["mmm", "nnn"])
def test_b(y, page: Page):
page.goto("https://www.baidu.com")
page.locator("#kw").fill(y)
page.locator("#su").click()
print(page.title())

###########

collecting ... collected 4 items

aaa.py::test_a[chromium-aaa]
aaa.py::test_b[chromium-mmm]
aaa.py::test_a[chromium-bbb]
aaa.py::test_b[chromium-nnn]

crazy-yuzhao commented

import pytest

@pytest.mark.parametrize("x", ["aaa", "bbb"])
def test_a(x):
print(f'{x}')

@pytest.mark.parametrize("y", ["mmm", "nnn"])
def test_b(y):
print(f'{y}')

########
collecting ... collected 4 items

aaa.py::test_a[aaa] aaa
PASSED
aaa.py::test_a[bbb] bbb
PASSED
aaa.py::test_b[mmm] mmm
PASSED
aaa.py::test_b[nnn] nnn
PASSED

aspenboy commented

But shouldn't the tests be independent of each other?

crazy-yuzhao commented

But shouldn't the tests be independent of each other?但测试不应该相互独立吗？

It needs to be like the native execution results of pytest. All parameterized test cases need to be executed before other subsequent use cases are executed. You can see the comparison of the results after the two executions above.

siddiquiakram84 commented

Using pytest-playwright, after parameterization, the execution order of use cases is inconsistent with pytest itself.this is bug！！

If you're experiencing inconsistent execution order of test cases in pytest-playwright after parameterization, it's important to note that the order of test execution in pytest is not guaranteed to be consistent. pytest executes test functions in a random order by default to ensure that tests are independent of each other.

However, if the order of execution is critical for your test cases, you can take the following approaches:

Sort the Test Cases Explicitly:

Sort your test cases explicitly within the test file. You can use the pytest.mark.run marker to assign a specific order to your test cases.

import pytest

@pytest.mark.run(order=1)
def test_case1():
# Your test implementation

@pytest.mark.run(order=2)
def test_case2():
# Your test implementation

symonk commented

relates to #204

andersonhowever commented

issue resolved in pytest 8.0.0

pytest tests -v -s
platform win32 -- Python 3.10.7, pytest-7.4.4, pluggy-1.4.0 -- 
plugins: base-url-2.1.0, playwright-0.4.4
collected 6 items                                                                                                                                                                     

tests/test_bug.py::test_has_title[chromium-65] PASSED
tests/test_bug.py::test_get_started_link[chromium-65] PASSED
tests/test_bug.py::test_has_title[chromium-90] PASSED
tests/test_bug.py::test_get_started_link[chromium-90] PASSED
tests/test_bug.py::test_has_title[chromium-110] PASSED
tests/test_bug.py::test_get_started_link[chromium-110] PASSED

pytest tests -v -s
platform win32 -- Python 3.10.7, pytest-8.0.0, pluggy-1.4.0 --
plugins: base-url-2.1.0, playwright-0.4.4
collected 6 items                                                                                                                                                                    

tests/test_bug.py::test_has_title[chromium-65] PASSED
tests/test_bug.py::test_has_title[chromium-90] PASSED
tests/test_bug.py::test_has_title[chromium-110] PASSED
tests/test_bug.py::test_get_started_link[chromium-65] PASSED
tests/test_bug.py::test_get_started_link[chromium-90] PASSED
tests/test_bug.py::test_get_started_link[chromium-110] PASSED

import re

import pytest
from playwright.sync_api import Page, expect


@pytest.mark.parametrize("fov_value", ['65', '90', '110'])
def test_has_title(page: Page, fov_value):
    page.goto("https://playwright.dev/")
    expect(page).to_have_title(re.compile("Playwright"))


@pytest.mark.parametrize("fov_value", ['65', '90', '110'])
def test_get_started_link(page: Page, fov_value):
    page.goto("https://playwright.dev/")
    page.get_by_role("link", name="Get started").click()
    expect(page.get_by_role("heading", name="Installation")).to_be_visible()
  • 👍1
mxschmitt commented

Great discovery, thanks! Closing by that:

Pytest 7.4.4:

PASSED test_foo.py::test_a[chromium-aaa]
PASSED test_foo.py::test_b[chromium-mmm]
PASSED test_foo.py::test_a[chromium-bbb]
PASSED test_foo.py::test_b[chromium-nnn]

Pytest v8

PASSED test_foo.py::test_a[chromium-aaa]
PASSED test_foo.py::test_a[chromium-bbb]
PASSED test_foo.py::test_b[chromium-mmm]
PASSED test_foo.py::test_b[chromium-nnn]