ADI-KOTKAR/Scrapy-Crawlers

Web Scraping with Python using Scrapy, Splash, Selenium.

Python

Scrapy-Crawlers

Web Scraping with Python using Scrapy, Splash and Selenium.

Installation

Cloning Repo:

git clone https://github.com/ADI-KOTKAR/Scrapy-Crawlers.git

Installing Requirements:

pip install requirements.txt

Resources

CSS : https://try.jsoup.org/
XPath : https://scrapinghub.github.io/xpath-playground/
IMDB : https://www.imdb.com/
Livecoin : https://www.livecoin.net/en
DuckDuckGo : https://duckduckgo.com/
worldometers : https://www.worldometers.info/world-population/population-by-country/
slickdeals : https://slickdeals.net/computer-deals/

Running Splash on Docker

docker run -it -p 8050:8050 --rm scrapinghub/splash