/django-scraping

This is a project where I have used django and scrapy together.

Primary LanguagePython

Django-Scraping

This is a project where I have integrated Django and scrapy together.I have used scrapy for crawling 2 websites namely Snapdeal.com and paytmmall.com. I have used 2 spiders for crawling the data and then django is used for displaying the results.

Prerequisites

  • Python3
  • Scrapy(1.7.3)
  • Scrapyd(1.2.1)
  • JSONField (Django)

Deployment

To deploy locally,

  • DB used is MYSQL and it should be configured and password for the database in picked via environment variable $DB_PASSWORD
  • python3 manage.py makemigrations
  • python3 manage.py migrate
  • ON a new tab in terminal change directory to /django-scraping/scraping_project/web_scraper and then run scrapyd (please note scrapyd should be ran on a new tab in terminal alongside django server)
  • python3 manage.py runserver