There are two parts of this app:
- Session CRUD Deals with CRUD operation on session database
- Analytics Deals with analytics operation such as session by week, and average duration binned
- Create
virtualenv
and install requirements - Copy .env.sample file to .env
- Setup postgres database Configure .env
- Install redis server if not already
- Migrate using
python manage.py migrate
- Schedule tasks using
python manage.py schedule_analytics_tasks
- Run two services
python manage.py runserver
andpython manage.py qcluster
Tools used: django
, django-rest-framework
, postgres
(database), redis
(cache), django-q
(background tasks)
Postgres Count query is slow for large tables, so maintained separate table to store count to make CRUD operation fast
Pandas DataFrame vs SQL Query
- Loading whole database to DataFrame proved to be much slower than running groupby query.
- RAW sql was a bit faster than django ORM
Cache and Background Task Analytics is cached to achieve standard response time and background task is used to update the cache periodically
Checkout following files