A student project for training web crawling and Test driven development.
It made with Django and Scrapy libraries.
- python3.4/3.5 (3.6 is not supported because scrapy-djangoitem do not support this specific version)
- python-virtualenv
- python-pip
- nodejs
- Clone the project.
- Go into the projet directory :
$ cd django-react-scrapy-sample/ - Create a Python3 Virtualenv :
$ virtualenv -p python3.5 venv - Activate the Virtualenv :
$ source venv/bin/activate - Install all python dependencies (this step can be long) :
$ pip install -r requirements.txt - Init the nodeenv :
$ nodeenv -p - Install all node dependencies :
$ npm install - Create the database :
$ python manage.py migrate - Load default shops fixtures :
$ python manage.py loaddata default_shops - Load sample items :
$ python manage.py loaddata sample_items - Create an admin :
$ python manage.py createsuperuser - Scrap sample items :
scrapy crawl shopscraper - Run a local server :
$ python manage.py runserver - Open 127.0.0.1:8000 in your favorite browser.