thp/urlwatch

Browser Job not working

Closed this issue · 5 comments

bokkabonga commented

Hi Everyone,

i created my first browser job after isntalling pyppeeeer. Sadly running the job causes some kind of error.
I couldn´t find any solution myself. So maybe someone here knows the problem and can make more sense of this error:/

`Exception while releasing resources for job:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 138, in test_filter
raise job_state.exception
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/handler.py", line 68, in enter
self.job.main_thread_enter()
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/jobs.py", line 377, in main_thread_enter
self.ctx = BrowserContext()
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 90, in init
BrowserContext._BROWSER_LOOP = BrowserLoop()
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 45, in init
self._browser = self._event_loop.run_until_complete(self._launch_browser())
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 51, in _launch_browser
browser = yield from pyppeteer.launch()
File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 307, in launch
return await Launcher(options, **kwargs).launch()
File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 168, in launch
self.browserWSEndpoint = get_ws_endpoint(self.url)
File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 227, in get_ws_endpoint
raise BrowserError('Browser closed unexpectedly:\n')
pyppeteer.errors.BrowserError: Browser closed unexpectedly:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/handler.py", line 78, in exit
self.job.main_thread_exit()
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/jobs.py", line 380, in main_thread_exit
self.ctx.close()
AttributeError: 'BrowserJob' object has no attribute 'ctx'
Traceback (most recent call last):
File "/usr/local/bin/urlwatch", line 11, in
load_entry_point('urlwatch==2.24', 'console_scripts', 'urlwatch')()
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/cli.py", line 112, in main
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 408, in run
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 210, in handle_actions
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/command.py", line 138, in test_filter
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/handler.py", line 68, in enter
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/jobs.py", line 377, in main_thread_enter
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 90, in init
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 45, in init
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/usr/local/lib/python3.6/site-packages/urlwatch-2.24-py3.6.egg/urlwatch/browser.py", line 51, in _launch_browser
File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 307, in launch
return await Launcher(options, **kwargs).launch()
File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 168, in launch
self.browserWSEndpoint = get_ws_endpoint(self.url)
File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 227, in get_ws_endpoint
raise BrowserError('Browser closed unexpectedly:\n')
pyppeteer.errors.BrowserError: Browser closed unexpectedly:`

thp commented

Seems like this is a pyppeteer error:

pyppeteer.errors.BrowserError: Browser closed unexpectedly:

Is there any other information printed somewhere?

Also, pyppeteer seems to be deprecated again, just to be replaced by playwright-python (we previously used a different library for browser jobs). Seems like maintaining a testing framework that hooks into a browser engine is something nobody wants to maintain?

In any case, can you check if running a simple Pyppeteer example works? (taken from the Pyppeteer README)

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://example.com')
    await page.screenshot({'path': 'example.png'})
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

If this doesn't work either, it might be due to your Pyppeteer installation, if this DOES work, we should investigate more what the issue could be.

bokkabonga commented

Sadly this didn´t work either:

Traceback (most recent call last):
File "./test.py", line 11, in
asyncio.get_event_loop().run_until_complete(main())
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "./test.py", line 5, in main
browser = await launch()
File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 307, in launch
return await Launcher(options, **kwargs).launch()
File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 168, in launch
self.browserWSEndpoint = get_ws_endpoint(self.url)
File "/usr/local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 227, in get_ws_endpoint
raise BrowserError('Browser closed unexpectedly:\n')
pyppeteer.errors.BrowserError: Browser closed unexpectedly:

Edit: Just noticed that pyppeteer was installed in Version 0.2.6 which is not the latest. I will see if I can update and if it changes anything

bokkabonga commented

Sadly 0.2.6 is the newest version i can install via pip. Installing via "pip install -U git+https://github.com/pyppeteer/pyppeteer@dev" sadly fails.

trevorshannon commented

I just did my first Browser job, and I got a similar error (Browser closed unexpectedly). Installing other dependencies and passing --no-sandbox to the pyppeteer launcher helped me resolve the issue. See:

Putting this here in case it helps anyone else!

bokkabonga commented

I just did my first Browser job, and I got a similar error (Browser closed unexpectedly). Installing other dependencies and passing --no-sandbox to the pyppeteer launcher helped me resolve the issue. See:

* https://stackoverflow.com/questions/61780476/pyppeteer-browser-closed-unexpectedly-in-aws-lambda/62305560#62305560

* https://stackoverflow.com/questions/57217924/pyppeteer-errors-browsererror-browser-closed-unexpectedly/71935536#71935536

Putting this here in case it helps anyone else!

The first link you provided fixed my error. In my case i had to install "libXScrnSaver.x86_64". Browser jobs now run without a problem. They don´t return what i wanted them to, but they work :D