simonw/datasette-lite

Add automated tests using Playwright

simonw opened this issue · 9 comments

simonw commented
Add automated tests using Playwright
simonw commented

I'm using pytest-playwright. Here's the first test I figured out which seems to do the right thing.

from playwright.sync_api import Page, expect


def test_initial_load(page: Page):
    page.goto("https://lite.datasette.io/")
    loading = page.locator("#loading-indicator")
    expect(loading).to_have_css("display", "block")
    # Give it up to 60s to finish loading
    expect(loading).to_have_css("display", "none", timeout=60 * 1000)

    # Should load faster the second time thanks to cache
    page.goto("https://lite.datasette.io/")
    expect(loading).to_have_css("display", "none", timeout=20 * 1000)
simonw commented

The expect() helper adds the magic where it keeps polling until the element becomes available. That's not necessary all the time though - this test does some direct assertions against elements on the page that it knows have loaded:

     assert [el.inner_text() for el in page.query_selector_all("h2")] == [
        "fixtures",
        "content",
    ]
simonw commented

Figured out how to use fixtures and run a test that navigates to the /fixtures page and submits the form to execute a custom query:

from playwright.sync_api import Browser, Page, expect
import pytest


@pytest.fixture(scope="module")
def dslite(browser: Browser) -> Page:
    page = browser.new_page()
    page.goto("https://lite.datasette.io/")
    loading = page.locator("#loading-indicator")
    expect(loading).to_have_css("display", "block")
    # Give it up to 60s to finish loading
    expect(loading).to_have_css("display", "none", timeout=60 * 1000)
    return page


def test_initial_load(dslite: Page):
    expect(dslite.locator("#loading-indicator")).to_have_css("display", "none")


def test_has_two_databases(dslite: Page):
    assert [el.inner_text() for el in dslite.query_selector_all("h2")] == [
        "fixtures",
        "content",
    ]


def test_navigate_to_database(dslite: Page):
    h2 = dslite.query_selector("h2")
    assert h2.inner_text() == "fixtures"
    h2.query_selector("a").click()
    expect(dslite).to_have_title("fixtures")
    dslite.query_selector("textarea#sql-editor").fill(
        "SELECT * FROM no_primary_key limit 1"
    )
    dslite.query_selector("input[type=submit]").click()
    expect(dslite).to_have_title("fixtures: SELECT * FROM no_primary_key limit 1")
    table = dslite.query_selector("table.rows-and-columns")
    table_html = "".join(table.inner_html().split())
    assert table_html == (
        '<thead><tr><thclass="col-content"scope="col">content</th>'
        '<thclass="col-a"scope="col">a</th><thclass="col-b"scope="col">b</th>'
        '<thclass="col-c"scope="col">c</th></tr></thead><tbody><tr>'
        '<tdclass="col-content">1</td><tdclass="col-a">a1</td>'
        '<tdclass="col-b">b1</td><tdclass="col-c">c1</td></tr></tbody>'
    )
simonw commented

Next challenge: serve localhost with the current state of the application, rather than running all of the tests directly against https://lite.datasette.io/

This is a bit tricky, because it means I need to spin up a localhost web server for the duration of the pytest run.

Some relevant code:

I'm inspired by the code that it uses to spin up that server though - it runs Popen(...) to start a python -m http.server --directory x process here: https://github.com/ppmdo/pytest-simplehttpserver/blob/a82ad31912121c074ff1a76c4628a1c42c32b41b/src/pytest_simplehttpserver/simplehttpserver.py

And then in https://github.com/ppmdo/pytest-simplehttpserver/blob/a82ad31912121c074ff1a76c4628a1c42c32b41b/src/pytest_simplehttpserver/pytest_plugin.py#L17-L28 it starts that process, yields it from a fixture, then later calls server_process.terminate() and server_process.wait().

I'm going to try spinning my own fixture that does effectively the same thing.

simonw commented

My version of that fixture:

from subprocess import Popen, PIPE
import pathlib
import pytest
import time
from http.client import HTTPConnection

root = pathlib.Path(__file__).parent.parent.absolute()


@pytest.fixture(scope="module")
def static_server():
    process = Popen(
        ["python", "-m", "http.server", "8123", "--directory", root], stdout=PIPE
    )
    retries = 5
    while retries > 0:
        conn = HTTPConnection("localhost:8123")
        try:
            conn.request("HEAD", "/")
            response = conn.getresponse()
            if response is not None:
                yield process
                break
        except ConnectionRefusedError:
            time.sleep(1)
            retries -= 1

    if not retries:
        raise RuntimeError("Failed to start http server")
    else:
        process.terminate()
        process.wait()
simonw commented

I've been running the tests with:

pytest --headed

So I can watch the browser as they run.

simonw commented

I can model the GitHub workflow on this one I wrote that exercises Datasette using pyodide and Playwright: https://github.com/simonw/datasette/blob/0.62a1/.github/workflows/test-pyodide.yml and https://github.com/simonw/datasette/blob/0.62a1/test-in-pyodide-with-shot-scraper.sh

simonw commented

Next challenge: serve localhost with the current state of the application, rather than running all of the tests directly against https://lite.datasette.io/

Hah, turns out I had a TIL about this that I'd forgotten about! https://til.simonwillison.net/pytest/subprocess-server

I added the static server recipe from above to that existing TIL.