dimitryzub/scrape-google-scholar-py

[Linux] Web-driver exits unexpectedly using CustomGoogleScholarOrganic()

Doloroy opened this issue · 5 comments

Doloroy commented

Traceback (most recent call last):
File "/mnt/c/Users/enarevbd/Documents/scraping_files/scrape-google-scholar-py/first_query.py", line 9, in
custom_parser_get_organic_results = CustomGoogleScholarOrganic().scrape_google_scholar_organic_results(
File "/mnt/c/Users/enarevbd/Documents/scraping_files/scrape-google-scholar-py/google_scholar_py/custom_backend/organic_search.py", line 122, in scrape_google_scholar_organic_results
driver = webdriver.Chrome(service=service, options=options)
File "/home/doloroy/.local/lib/python3.10/site-packages/selenium/webdriver/chrome/webdriver.py", line 80, in init
super().init(
File "/home/doloroy/.local/lib/python3.10/site-packages/selenium/webdriver/chromium/webdriver.py", line 101, in init
self.service.start()
File "/home/doloroy/.local/lib/python3.10/site-packages/selenium/webdriver/common/service.py", line 104, in start
self.assert_process_still_running()
File "/home/doloroy/.local/lib/python3.10/site-packages/selenium/webdriver/common/service.py", line 117, in assert_process_still_running
raise WebDriverException(f"Service {self.path} unexpectedly exited. Status code was: {return_code}")
selenium.common.exceptions.WebDriverException: Message: Service /home/doloroy/.wdm/drivers/chromedriver/linux64/112.0.5615/chromedriver unexpectedly exited. Status code was: 127

dimitryzub commented

@Doloroy thank you for opening this issue 👍

Could you mention:

  • your Python version.
  • does organic results extraction worked before or it's the first time you ran it?
  • do you have Chrome browser installed (the one that you use for browsing)?
Doloroy commented

Thank you @dimitryzub

  • the python version is 3.10.6
  • the organic result extraction did not work, and it is my first time running it.
  • Chrome is installed as the default browser, and I have also updated it.
  • 👍1
dimitryzub commented

@Doloroy thank you 👍

My assumption is that it is related to the Linux system. I've only tested and written this parser on Windows and haven't tested it on Linux.

I think that extra dependencies are required to run the chromedriver:

$ apt-get install -y libglib2.0-0 libnss3 libgconf-2-4 libfontconfig1

Ref:

Let me know if it helped, or didn't 👍

  • 👍1
Doloroy commented

Thanks! Errors have been resolved.

  • ❤️1
dimitryzub commented

@Doloroy Wonderful. Thank you for sharing this experience. I've updated README for Linux users 🙂

Let me know if you have any usability issues or something else.