Memescraper
Django web scraping app
Purpose
This app scraps memes from Reddit. By default you see newer memes, but you also can change it to random order.
Features
Web Scraping
Web scraping itself done with Selenium and Chromedriver. You may check it at posts/scrap.py . More about Selenium
Infinite scroll
Although Django supports pagination, we all know that memes are better served when scrolled endlessly. More about Endless Scroll
Background Task
New memes are scraped every minute. This task is scheduled it at posts/scrap.py as well, using Advanced Python Scheduler. More about APS
Deployment
App deployed at Heroku, you may check it here
Heroku specific
Make sure that following builpacks are present: Google Chrome buildpack, Chromedriver buildpack. More about Chrome and chromedriver on Heroku
Run locally
To run app locally, do next steps:
- Download files.
- In posts/scrap.py uncomment Local section and comment Heroku section, and edit chrome_driver_path to absolute path of chromedriver at your machine.
- In root folder (where manage.py located), run commands:
pip install requirements.txt
python manage.py runserver