This application is a Django project which is a ad scraper.
- Clone the repository.
git clone https://github.com/Dylan-YD/scraper.git - Install Python on your machine.
- Navigate to the project folder.
cd scraper - Install requirements.
pip install -r requirements.txt - Run the server.
python manage.py runserver
- Go to the home page.
http://54.205.89.138:8001/ - Enter the query of the ads you want to scrape.
- Click the button to scrape the website.
- The progress will be shown on the page.
- After the scraping is done, the result will be shown on the page
http://54.205.89.138:8001/crawler/ads.
scraper
├── crawler
│ ├── admin.py
│ ├── apps.py
│ ├── __init__.py
│ ├── migrations
│ ├── models.py
│ ├── static
│ ├── templates
│ ├── tests.py
│ ├── urls.py
│ └── views.py
├── db.sqlite3
├── manage.py
├── README.md
├── requirements.txt
└── scraper
├── asgi.py
├── __init__.py
├── settings.py
├── urls.py
└── wsgi.py