/crawling

Primary LanguageJupyter Notebook

Crawling articles easier

Crawling research articles with Selenium, BeautifulSoup in Python.

Available websites

Chrome driver issues

  • Chromedriver usage with wd.Chrome(service=Service(ChromeDriverManager().install())) unavailable due to Chrome version (≥115)
  • The argument driver_dir must be provided.

How to use

  1. Select a crawling site (among the available lists above).
  2. Type keyword to keyword argument.
  3. Run {}_crawling function.

TODO