RoguedBear/codezinger-due-date-scraper

program crashes somewhere when the browser is launched and chromium starts to consume very high cpu

Closed this issue · 6 comments

orphan chromium processes still remain active and consume cpu as if the browser isn't closed.

workaround for now:

  • restart the container
  • cron schedule to pkill all chromium after the bot runs

update: when this happens, the bot doesn't post anything on discord.
possible source of error could be these lines since:

browser = await launch(executablePath=chrome_path,
headless=True,
options={'args': ['--no-sandbox']})
page = await browser.newPage()

since they're not under try/except


relevant log files:
image
red is the last log and since when the chromium high cpu usage is started
green is when i manually restarted the container

processed running when cpu usage is high:
image

Last log message:
image

possible workarounds:

  • if opening chrome or that process takes more than 90seconds, kill it and restart
    (and/or set a cpu limit in compose)
  • add the --disable-gpu param
  • downgrade to python 3.9
  • install all puppeteer's dependencies

exact LOC where the freezing happens (confirmed via logs):

page = await browser.newPage()

Logs when the program freezes:

crond: USER root pid   9 cmd /entrypoint.sh
codezinger-due-date-scraper  Copyright (C) 2022  RoguedBear, Ya-s-h
    This program comes with ABSOLUTELY NO WARRANTY; see COPYING
    This is free software, and you are welcome to redistribute it
    under certain conditions; see COPYING
loaded short names
Starting browser...
browser started

Logs when the program runs normally

crond: USER root pid   9 cmd /entrypoint.sh
codezinger-due-date-scraper  Copyright (C) 2022  RoguedBear, Ya-s-h
    This program comes with ABSOLUTELY NO WARRANTY; see COPYING
    This is free software, and you are welcome to redistribute it
    under certain conditions; see COPYING
loaded short names
Starting browser...
browser started
logging in.
Logged in. allegedly
sorting
loading more
Processing stuff
[<pyppeteer.element_handle.ElementHandle object at 0x7f1d26000640>, <pyppeteer.element_handle.ElementHandle object at 0x7f1d26001030>]
2
Scraped 2 questions(50%)
awaiting browser close
Browser closed
Processing new events...
Sent 0 new webhook messages
Finished, exiting...

update:
unable to reproduce the problem when running current version and previous version repeatedly, but --disable-gpu does help in drastically reducing CPU usage.

graph:
image

running the program for 2 weeks, commit a64681c does seem to have fixed the issue