/adatapro-internship-2021

Site, which shows information about the current scraping status. Built with Django and Scrapy.

Primary LanguageHTML

Team Red


Site, which shows information about the current scraping status.


💻 About

Our project consists of a site that aims to help its users by providing them with up-to-date information on the availability of the PlayStation 5. The information is retrieved automatically through crawling. Once a day, registered users are sent an email with information about an available PS5, and users can set up the system to receive information only from the places they are interested in.

🎥 Interface

Our Documentation

⌨️ How it works

Our work can be split into 4 different parts:

  1. The Site (Our Front-end)
    • Home - gives basic information about the site.
    • Information - gives more information about the site and how to use the functions of the site.
    • Registration and login - to create an account and log in.
    • Account - to control user settings and functions.
  2. The Django Application (Our Back-end)
    • Our Back-end is entirely written on py Django.
    • It is centered around views, which are in different 📁 folders.
    • It can be started by the 📁 manage.py script.
    • You can find in the in 📁 API/ folder.
    • It is hosted on localhost on port 8000, although you can change it.
  3. 📊 The Database
    • We are using mariadb MariaDB because of its worldwide popularity and its simplicity.
    • We have a few of tables, but the main ones are the auth_user, accountview_profilepreferences and historyview_historyscraping.
      • The auth_user stores all ®️ registered accounts.
      • The accountview_profilepreferences stores all the profile preferences related to which sites you want to see.
      • The historyview_historyscraping stores all the information from the scrapers.
    • All of the tables are generated from py Django models.
  4. The Scraper

⛏️ Installation

Step 1: Clone the repository

git clone https://github.com/ABPozharliev19/adatapro-internship-2021.git
cd adatapro-internship-2021/

Step 2: Install dependencies

You can do this by either using pipenv:

pip3 install pipenv # Install pipenv with pip
pipenv shell # Activate the environment
pipenv install # Install all dependencies in the Pipfile.lock file

Or by using pip:

pip3 install -r requirements.txt # Install all dependencies in the requirements.

Step 3: Rename env.example to .env

Step 4: Change the information in the .env file

SECRET_KEY = 'Your django secret key'
DB_NAME = 'Your database name'
DB_USER = 'Your database user'
DB_PASSWORD = 'Your database password'
DB_HOST = 'Your host'
DB_PORT = 3306 # The port you want to use
EMAIL_HOST_USER = 'Your email username' # Should be gmail
EMAIL_HOST_PASSWORD = 'Your email password'

Step 5: Make the migrations and migrate the models

python manage.py makemigrations
python manage.py migrate

Step 6: Run the server

python manage.py runserver

Or specify the port for the server

python manage.py runserver PORT

Step 7: Go to localhost:PORT and use the site!

🧒 Our Team

Atanas Pozharliev - ABPozharliev19 - Back-End Developer

Kostadin Taligadzhiev - KNTaligadzhiev19 - Front-End Developer

Igor Simeonov - GrandMaster-42 - Scraping

Velin Kurtev - Velin1234 - Scraping

Diyan Nedelchev - dido322 - Scraping

Kaloyan Totev - KaloyanTotev339 - Grafana & Docker Setup