/pyviztest

A Visual Testing Library in Python. Works with Playwright, Selenium. Can generate Allure Report.

Primary LanguagePythonMIT LicenseMIT

pyviztest

A Visual Test Automation Library in Python. Works with Playwright and Selenium. Integrable with Allure Report.

Project PyVizTest

Why Project pyviztest?

Forewords

In Python there was no robust and integrable Visual Test Automation library/module. Thankfully, Kumar Aaditya created a pytest fixture for Playwright using the power of pixelmatch library for Visual Testing, and Symon Storozhenko enhanced it to create pytest-playwright-visual. I really loved both, so I tried to realign the same concept by developing a library and added a few more functionality to make the use of it more flexible, and also compatible with Playwright and Selenium. All thanks go to both of them for being the harbinger of the visual testing libraries in python. I merely followed the path and improvized.

Features

So, previously created pytest-playwright-visual had the power to be summoned anywhere inside the tests which had it as a parameter, and to

    • Create Golden Snapshot (The reference image of the UI which needs to be compared with during test execution) of the AUT(Applicayion under test).
    • Update existing Golden Snapshot.
    • Take snapshot in runtime and compare it with the Golden Snapshot. If both do not match, it creates an image where the difference is highlighted.

In addition to these features, pyviztest is capable of doing a few more things!

    • Supports Windows, Linux and MacOS systems.
    • Supports Playwright and Selenium. You just need to provide the Page and WebDriver instances.
    • Supports custom snapshot directories.
    • Supports option to Create & Update snapshots(particular and all) without failing the tests.
    • Supports option to add multiple snapshot validation in a single test case, and if one of them fails it will still validate others.
    • Supports integration with Allure Report.
    • Supports auto-naming of the snapshots with respect to the test suite and test case name.
    • Provides api to return Golden, Actual and Difference Snapshots in bytes, to easily integrate with any custom report.
    • Supports comparing images with different in sizes without failing the test abruptly.
    • Supports comparing snapshots of multiple web elements together or the full page.

Upcoming Next:

    • Support for Appium.
    • Support for element specific identification.
    • A fixture along with the usual library.

How To Use It:

Installing

Simple and easy to install because it's just another python library! Install using below pip command:

    $ pip install pyviztest

Configuration

  1. Required: Import VisualTest from pyviztest.viztest
    from pyviztest.viztest import VisualTest
  1. Required: The object of VisualTest is the driving factor of the visual tests.
    visualtest = VisualTest(driverpage=driver)

Note You may directly create an object of VisualTest class for each test separately and proceed with that object, or you can create a pytest fixture for dependency injection to initialize the VisualTest instance from inside that class. Using a separate class/fixture to implement dependency injection is always recommended. The constructor of VisualTest class can take 5 arguments i.e. snapshot_path, [MANDATORY]driverpage: WebDriver/Page, updatesnapshot, savefailuresnapondisk & allurereport. The usage of them are elaborated in below code snippet comments.

    class VisualTest(
        snapshot_path: str = '', #The path where all the snapshots will reside. If not provided, it will consider the parent directory of the test class as the snapshot path by default.
        driverpage: Any = '', #You MUST to provide either the driver object (if you are using Selenium for your tests) or page fixture/object (if you are using Playwright)
        updatesnapshot: bool = False, # If True, it will update all the existing golden snapshots or to create new golden snapshots for the current session of Visualtest class.
        savefailuresnapondisk: bool = True, #If True, all the failure snapshots will be saved inside the failure directories under snapshot_path as image files in PNG format.
        allurereport: bool = False #If True, allure report will be generated with Golden snapshots and failure snapshots.
    )
  1. Required: Invoke the setpaths() function for each of the tests.
    visualtest.setpaths()

Note The setpaths function can take 2 optional arguments i.e. updatesnapshot and numberofclassesaftertestclass. The usage of them are elaborated in below code snippet comments.

    (method) def setpaths(
        updatesnapshot: bool = False, # Same as the updatesnapshot mentioned above. It provides the flexibility to update snapshots for a particular test case. If not given, it would take the default value during creation of VisualTest class object.
        numberofclassesaftertestclass: int = 0  # This parameter defines the number of classes between the test methods and where the visualtest_web() method is invoked.
    ) -> None

Warning numberofclassesaftertestclass parameter helps you to fetch the testcase name in order to generate the snapshot's name by default if you are not providing any specific name. By default the value of it is 0, that means it assumes that the visualtest_web() method is being invoked from inside the test mthod directly without any further abstraction. If you add N number of abstractions, then you should provide the value of the parameters as N. See point no. 4 for more info.

    visualtest.setpaths(numberofclassesaftertestclass=N)
  1. Now that the configuration is done, you have to invoke visualtest_web() method wherever the webpage snapshots need to be validated.
    assert vt.visualtest_web(stepname="validateloginpage")

Note This method takes a minimum of 7 arguments i.e. stepname, threshold, fail_fast, updatesnapshot, fullpage, snapshot_of_locators and exclude_locators. The usage of them are elaborated in below code snippet comments.

    (method) def visualtest_web(
        *,
        stepname: str = '', # Optional, but it is recommended to provide the value to distinguish snapshots
        threshold: float = 0.1, # Optional, sets the threshold for the comparing the snapshots 0 to 1
        fail_fast: bool = False, # Optional, if you'd like to fail the comparison even if 1 pixel doesn't match, make it True
        updatesnapshot: bool = False, # Same as the updatesnapshot mentioned above. It provides the flexibility to update snapshots for a particular test step. If not given, it would take the default value during creation of VisualTest class object or setpaths().
        fullpage: bool = True, # Optional, to take snapshot of the full page or not
        snapshot_of_locators: list = [], # Optional, you may send set of locators or webelements to take snapshots of only those elements instead of a full page snapshot
        exclude_locators: list = [] # Optional, you may send set of locators to mask them in the snapshots, so during the comparison they will be excluded.
    ) -> bool

Warning fullpage and exclude_locators are only applicable for Playwright tests. They do not work for Selenium tests. Note This visualtest_web() method returns a boolean value as per below table:

updatesnapshot(VisualTest()) updatesnapshot(setpaths()) updatesnapshot(visualtest_web()) Snapshots Match? visualtest_web()
True X X X True
False True X X True
False False True X True
False False False True True
False False False False False

Warning

Do not Change the value of numberofclassesaftertestclass if you don't understand the below concept well.

As mentioned above already, numberofclassesaftertestclass parameter helps you to fetch the testcase name in order to generate the snapshot's name by default if you are not providing any specific name. Let's look at an example:

       Project|
              |---src|
              |       |--testsuitename|
              |                       |---test_playwright.py|
              |                       |                     |---test_1()
              |                       |                     |---test_2()
              |                       |
              |                       |---test_selenium.py|
              |                                           |---test_1()
              |                                           |---test_2()
              |---VisualTestResults

In case of the folder structure depicted above, if the OS is Linux, snapshot_path is VisualTestResults, and the visualtest_web() method is invoked from the test_1() and test_2() methods, then the name of the Golden snapshot created for test_1() in test_playwright.py will be linux_test_1.png and it will reside inside VisualTestResults/Golden_Snapshots/test_playwright/test_1/linux_test_1.png. So after the execution if updatesnapshot = True, the folder structure will be like below:

       Project|
              |---src|
              |       |--testsuitename|
              |                       |---test_playwright.py|
              |                       |                     |---test_1()
              |                       |                     |---test_2()
              |                       |
              |                       |---test_selenium.py|
              |                                           |---test_1()
              |                                           |---test_2()
              |---VisualTestResults|
                                   |---Golden_Snapshots|
                                                       |---test_playwright|
                                                                          |---test_1|
                                                                          |         |---linux_test_1.png  
                                                                          |  
                                                                          |---test_2|
                                                                                    |---linux_test_2.png

Now, let's look at a different folder structure below:

       Project|
              |---src|
              |       |--testsuitename|
              |                       |---test_playwright.py|
              |                       |                     |---test_1()|
              |                       |                     |           |--validate_launchpage()|
              |                       |                     |                                   |--validate_loginbutton()
              |                       |                     |---test_2()|
              |                       |                                 |--validate_loginpage()|
              |                       |                                                         |--validate_homepage()
              |                       |
              |                       |---test_selenium.py|
              |                                           |---test_1()
              |                                           |---test_2()
              |---VisualTestResults

Here, the visualtest_web() method is invoked from validate_loginbutton() and validate_homepage() methods instead of the test_1() and test_2() methods. then the name of the Golden snapshot created for validate_loginbutton() will be linux_validate_loginbutton.png and it will reside inside VisualTestResults/Golden_Snapshots/validate_launchpage/validate_loginbutton/linux_validate_loginbutton.png. Now clearly, if we want the name of the test folders to be of the test method's name, then we should initialize numberofclassesaftertestclass = 2. (2 because there are 2 invoked methods between the test method test_1() and validate_loginbutton()).

    visualtest.setpaths(numberofclassesaftertestclass=2)

Example Code

Playwright:

    from playwright.sync_api import Page
    import pytest
    from pyviztest.viztest import VisualTest

    @pytest.fixture(scope="function", autouse=True)
    def before_each(page:Page):
        page.goto(url=url)
        yield

    def test_playwright_updatesnapshot(page:Page) -> None:
        # Create an object of VisualTest class
        # with parameters ->
        # @snapshotpath where all the Golden and other snapshots will be stored
        # @driverpage where you have to provide the page fixture of Playwright
        # @updatesnapshot a boolean value, make it True if you'd like to update all the snapshots
        # @savefailuresnapondisk a boolean value, make it True if you'd like to save all the snapshots as image files
        # @allurereport  boolean value, make it True if you'd like to generate allure report with the snapshots
        vt = VisualTest(snapshot_path=visualtestresults, driverpage=page, updatesnapshot=False, 
                        savefailuresnapondisk=False, allurereport=True)
        # setpaths() function must to be invoked before summoning any visualtest method
        # @updatesnapshot parameter will override the default updatesnapshot value set during 
        # VisualTest object creation
        vt.setpaths(updatesnapshot=True)
        page.wait_for_load_state(state='networkidle')
        page.wait_for_selector(id_header_productstore)
        assert page_title in page.url
        # For visual testing for any Web app you need to invoke visualtest_web() function
        assert vt.visualtest_web(stepname="launchpage")

Selenium:

    from playwright.sync_api import Page
    import pytest
    from pyviztest.viztest import VisualTest

    @pytest.fixture (scope="function", autouse=True)
    def browser():
        driver = webdriver.Chrome()
        driver.get(url=url)
        yield driver
        driver.quit()

    def test_selenium_updatesnapshot(browser) -> None:
        # Create an object of VisualTest class
        # with parameters ->
        # @snapshotpath where all the Golden and other snapshots will be stored
        # @driverpage where you have to provide the page fixture of Playwright
        # @updatesnapshot a boolean value, make it True if you'd like to update all the snapshots
        # @savefailuresnapondisk a boolean value, make it True if you'd like to save all the snapshots as image files
        # @allurereport  boolean value, make it True if you'd like to generate allure report with the snapshots
        vt = VisualTest(snapshot_path=visualtestresults, driverpage=browser, updatesnapshot=False, 
                        savefailuresnapondisk=False, allurereport=True)
        # setpaths() function must to be invoked before summoning any visualtest method
        # @updatesnapshot parameter will override the default updatesnapshot value set during 
        # VisualTest object creation
        vt.setpaths(updatesnapshot=True)
        headerelement = browser.find_element(By.XPATH, xpath_header_productstore)
        assert page_title in browser.current_url
        assert headerelement.is_displayed()
        # For visual testing for any Web app you need to invoke visualtest_web() function
        assert vt.visualtest_web(stepname="launchpage")

Execution

First Time:

When the Golden Snapshots (the reference snapshots with which the execution snapshots will be compared) are not existing, then you have to create them first. So make the value of updatesnapshot as True and call the pytest command in command line interface.

$ pytest

After the execution, you will see that the Golden Snaphots will be generated in your project directory where you have set your snapshot_path.

Create Golden Snapshots

Second Time onwards/For snapshot validation:

Make the value of updatesnapshot as False and call the pytest command in command line interface.

$ pytest

After the execution, if the value of savefailuresnapondisk is True, you will see that the Failure Snaphots will be generated in your project directory where you have set your snapshot_path. If no test case fails, no Failure Snapshot would be generated. If the value of savefailuresnapondisk is False, Snapshot files will not be generated.

Create Failure Snapshots

Now, if allurereport = True, then you can generate allure report but you have to run below command:

$ pytest --alluredir=allure-results

followed by

$ allure serve .\allure-results\

Then you will be able to see the generated allure reports with pass and failure reports with mismatch snapshots.

Allurereport Mismatch Snapshots Embedded in Allure Report

To Update Golden Snapshots:

If you want to update the Golden Snapshots of all the validations in a single VisualTest() session, then make the value of updatesnapshot as True while creating a VisualTest() object. You'd have the option to do the same for setpaths() and visualtest_web methods as well. Then call the pytest command in command line interface.

$ pytest

After the execution, the Golden Snapshots of the targeted test cases will be replaced with new snapshots.