ElSnoMan/pyleniumio

[GH Action] UI Tests sometimes fail with Selenoid

ElSnoMan opened this issue · 0 comments

Problem

When a test or pipeline passes sometimes and fails other times without any changes, we call that "flaky". Well, it seems our test-pylenium.yml pipeline is flaky 😢 . Here's an example:

https://github.com/ElSnoMan/pyleniumio/actions/runs/5336984960/jobs/9672430764

Potential Solutions

When our pipeline is "flaky", it seems the last 2-3 tests fail because of a timeout issue. The solution may be something as simple as configuring Selenoid to have a longer timeout. Look at their docs for some ideas:

https://github.com/marketplace/actions/start-selenoid-server?version=v2

Also, it would be nice to be able to run multiple tests as the same time to speed up the Test Run time. Right now we run all of our tests sequentially because the GitHub Action container isn't very powerful, but with some optimizations to Selenoid, we might be able to run 2-4 concurrently