Xerrex/web-scraper

Vehicles web Scraper

• This Project scraps vehicles from Edmunds Cars
• It saves the Vehicles in Excel workbook.

Made with

• Python
• Scrapy - Docs
• Openpxyl - Docs

Table Of Contents

• Project Setup
• Running the Crawler

Project Setup

• Git clone or download and unzip file

  • To clone via terminal

  $ git clone https://github.com/Xerrex/web-scraper.git

• Open terminal in the Uncompressed Folder

  & cd web-scrapper

• Create Virtual Environment & Activate it.

  $ python3 -m venv venv
  $ source venv/bin/activate

  • for window use:

    $ py -m venv env
    $ .\env\Scripts\activate

• Install the dependancies

  pip install -r requirements.txt

  • for windows

  py -m pip install -r requirements.txt

Running the Crawler

• To Crawl:

  $ scrapy crawl vehicle_spider

• Crawl and Out put to json file

  $ scrapy crawl vehicle_spider -O data/vehicles.json

• Crawl with a radius

  $ scrapy crawl vehicle_spider -a radius=10