Feature: Allow to increase `WEBDRIVER_TIMEOUT`?
olifre opened this issue · 3 comments
Currently, the WEBDRIVER_TIMEOUT seems hardcoded in dnstwist.
This causes flakiness with one of the pages I test, which sometimes has very complex javascript-rich content (sadly), and dnstwist errors out with:
fetching content from: http://uni-bonn.de/ > https://www.uni-bonn.de/en [110.4 KB]
rendering web page: http://uni-bonn.de/
Message: timeout: Timed out receiving message from renderer: 4.154
(Session info: headless chrome=116.0.5845.180)
Would it be possible to add an option to increase this timeout when required?
(I hope this is the correct timeout I am looking at)
Note: In case you want to reproduce, this was executed using the latest official docker container (phash flavour), on GitHub actions. The same test worked fine a day before, so it seems to me that this is related to site complexity and server performance on the site end, which could match hitting a timeout.
Although I wasn't able to reproduce this error with that particular domain name, I agree this might be "fixed" with setting a longer timeout in the web driver. I will figure out a way to make it configurable - probably with environment variables.
Thanks! In fact, I also can not reproduce with that domain "now", it seems to be time-dependent — I likely hit a period of high load when running my tests. So having this configurable (environment variables is fine of course) for "flaky servers" seems like a good "solution".