High memory consumption
juhacz opened this issue · 11 comments
High memory consumption There is something wrong with the class, if a program runs in a loop and downloads data non-stop, the program's memory consumption increases every second.
Sample program:
from selenium_driverless import webdriver
from selenium_driverless.types.by import By
import asyncio
async def main():
options = webdriver.ChromeOptions()
async with webdriver.Chrome(options=options) as driver:
while True:
await driver.get('https://pudelek.pl')
await driver.sleep(0.5)
source = await driver.page_source
await asyncio.sleep(3)
asyncio.run(main())
After several minutes of running the program, the memory usage of the python.exe process increases by several MB, and after several dozen minutes even up to 2 GB.
Python 3.10.6
Windows 11
Chrome 119.0.6045.124
Uh yeah there might actually be a memory leak indeed - probably even in the underlying CDP-socket library
@juhacz Thanks for raising this issue, I'll have a look at it.
To supplement the problem report, after about 1.5 hours of running the script and calling driver.get about 1,500 times, python.exe already occupies about 1.7 GB of RAM.
Then the TimeoutError occurs('page: "https://xxxxxxxxx" didn't load within timeout of 10')
chrome exited
A solution might be, for example, closing Chrome via driver.quit() every 100 iterations, but in my case it will not work, because often the first time I call driver.get() the page from which I am trying to download data displays the Captcha (datadome). After solving it, the program runs until it takes up a lot of memory and crashes.
@juhacz Solution probably will be to implement a time or date buffer, and not store all messages from//to chrome forever. Suspect that to be the case.
Unless Chrome as a subprocess takes up that much memory - but then I can't do a lot.
After downloading the page source, I parse the json from it and save it to a file. I don't store the values retrieved with each loop interaction, so the problem isn't on that side. The next time the loop runs, the variables are overwritten.
After downloading the page source, I parse the json from it and save it to a file. I don't store the values retrieved with each loop interaction, so the problem isn't on that side. The next time the loop runs, the variables are overwritten.
Yeah I meant more like - inside my library yk:)
on
- Win10
- Python 3.7.9
I can confirm this. Memory increases about 12.5 MB/minute
Google-Chrome process seems stable at 300MB - 500 MB
will be doing further testing..
I checked this version and unfortunately nothing had changed, after about 3 minutes of running the program, Python.exe was already using about 130MB of memory.
Python 3.10.6 x64
Windows 11 x64
Chrome 119.0.6045.160 x64
I checked this version and unfortunately nothing had changed, after about 3 minutes of running the program, Python.exe was already using about 130MB of memory.
Python 3.10.6 x64 Windows 11 x64 Chrome 119.0.6045.160 x64
@juhacz
Huh that's weird - can you check if the changes at e657c46 actually got applied in your env?
I'll check then Python 3.10 as well
I checked and I still had the old version of the class installed, I don't know why but I followed the command
pip install https://github.com/kaliiiiiiiii/CDP-Socket/archive/refs/heads/dev.zip
after executing the command:
pip install --upgrade --force-reinstall https://github.com/kaliiiiiiiiii/CDP-Socket/archive/refs/heads/dev.zip
the correct version has been installed. I checked the program and everything is OK, memory consumption is around 56MB.
Thank you for your great work, it helps me a lot!
I checked and I still had the old version of the class installed, I don't know why but I followed the command
pip install https://github.com/kaliiiiiiiii/CDP-Socket/archive/refs/heads/dev.zip
after executing the command:
pip install --upgrade --force-reinstall https://github.com/kaliiiiiiiiii/CDP-Socket/archive/refs/heads/dev.zip
the correct version has been installed. I checked the program and everything is OK, memory consumption is around 56MB. Thank you for your great work, it helps me a lot!
That's great to hear!
I noticed that opening and closing a tab creates further leaks. I'll leave this issue open for now & try to fix it as well.