ngchwanlii/aws-superbfresh

Superbfresh - Online Grocery Marketplace

Superbfresh - Online Grocery Marketplace

Description

• Superbfresh is an online grocery marketplace based on B2C model.

Technology Stack

• Language: Python3
• Framework: Django
• Asynchronous Task Queue: Celery
• Database: Amazon RDS (MySQL)
• In-memory data store & cache: Amazon ElastiCache (Redis)
• Cloud storage: Amazon S3
• Email service: Amazon SES
• Payment + Webhooks: Stripe
• Search Engine: Amazon ElasticSearch
• Virtual Server: Amazon EC2
• Web server gateway: uwsgi
• Web server proxy: nginx

Software Architecture Diagram

Implemented functionalities

• Backend function modules
  • User module
    • Register
    • Login
    • Logout
    • Email activation (celery + Amazon SES)
    • User center
    • Address
  • Goods module
    • Home page view (Generate static homepage through Celery, serve it from Static Content Web Server)
    • Goods detail view
    • Goods list view
    • Signal processing (When admin add/delete a product, it should also be indexed/removed from the ElasticSearch documents)
  • Cart module (Redis)
    • Get cart item
    • Add cart item
    • Remove cart item
    • Update cart item
  • Order module
    • Confirm order (in place_order view )
    • Commit order
    • Pay order (through Stripe)
    • Check order result & status
    • Comment ordered item
  • Webhooks module
    • Check order result & status (Stripe's webhooks)
  • Search module
    • Search item based on item's title, description & type (Amazon ElasticSearch)
    • Analyzer: html_strip
    • Tokenizer: letter
    • Filter: lowercase, stop, snowball
    • Char_filter: html_strip

General Setup

• Install all dependency / packages pip3 install -r requirements.txt
• See instruction below to setup each services
• Remember to set your own nginx & uwsgi configuration

Setup EC2

• Setup Amazon Ec2

Setup Database

• Setup mySQL with Amazon RDS

Setup Redis

• Setup Amazon ElastiCache Redis (Free Tier)
• Docs - django-redis
• Docs - redis-py
• Use cases:
  • For web optimization
    • Used to cache homepage html (or frequent visited html) so we can return these pages immediately to the client
  • As message broker for Celery, explained in Setup Celery section below.
• To check your redis's key-value store, use redis-cli -h <endpoint> -p <port>

Setup Celery

• Intro to Celery
• Used as a mechanism to distribute work across machines, for horizontal scaling purpose.
• Used for sending email asynchronously
  1. Submitting task to Redis (broker) endpoint as a task queue
  2. Celery worker in remote server will get a task from Redis and perform the task
• Start a worker in remote server, remember to clone your function modules to the remote server
• For development and testing
  • celery -A celery_tasks.tasks worker -l info
• For running celery worker in background
  • See Run celery in background
    • Ex: celery -A celery_tasks.tasks worker --detach

Setup mail service

• Setup Amazon SES
• NOTE:
  • Remember to request for increasing your SES Sending Limits and move out of Amazon SES sandbox to send email to others without pre-registered their email in SES console.

Setup TinyMCE editor

• Install django-tinymce

Storing images in cloud

• Setup Amazon S3
• Install django-storages

Setup Stripe for payment & webhooks

• Install Stripe Python Library

• Stripe API docs

• Setting Up Webhooks

• Setup ngrok

  • To make your local ec2 endpoint available for receiving events (in https scheme)
  • Fire up in background nohup ngrok http <port> &
    • Ex: nohup ngrok http 80 &
  • To get the https_public_url, use curl http://127.0.0.1:4040/api/tunnels
  • Pass this public_url as webhooks url with your api's url appended behind it.
    • Ex: <ngrok_url>/<api>/<path>

• To test Stripe payment

  • Credit card: 4242 4242 4242 4242
  • Expired Date: Set to future date
  • CSV: any number

Setup AWS ElasticSearch

• Intro to Amazon Elasticsearch Service

• Install elasticsearch-py

• Install elasticsearch-dsl-py

• In this application, you can search item through item's name, description and types.

  • Ex1 via item's name: grapes
  • Ex2 via item's description: sweet
  • Ex3 via item's types: fruit

• NOTE

  1. Expose TYPE=HTTPS, PROTOCOL=TCP, PORT=443 in your AWS Security Group (ex: es-sg)
  2. To index your datasets
     • Go to your python shell python manage.py shell
     • Import the bulk function from apps.search.documents import bulk_indexing
     • Start bulk indexing of specific document bulk_indexing()
     • Start bulk deleting of specific document bulk_deleting()

Setup Application Load Balancer (using Nginx on EC2)

• Setup nginx on EC2
• Serve static files in nginx, before that make sure to run python manage.py collectstatic in application server
• Reload nginx nginx -s reload
• Configure your nginx config depends on your software architecture diagram.

Setup uWSGI settings & configuration (web server gateway interface)

• Setup uWSGI
• Loading configuration files in background
  • uwsgi --ini <uwsgi.ini>
• Stopping the uWSGI server
  • uwsgi --stop <uwsgi.pid>

Collect staticfiles

• Django collect staticfile
• We can use a remote server that serve static files, or we could use CDN, like Amazon CloudFront

NOTE

• For integrating AWS services, please read the official docs provided above, and also make sure to configure your security groups (inbound / outbound) rules to allowed their access to your EC2 instances.

Miscellaneous

• Kill nohup process
• AWS VPC configuration
  • Another VPC configuration
  • NOTE:
    • To connect two EC2 instances, set both Route Table's destination (IPv4 CIDR) equal to the another EC2's IPv4 CIDR, and set Target as the newly established VPC Peer connection
    • Set application inbound rules
• Tutorial on ec2 deployment with nginx and uWSGI
• Check listening port
• The import-tags.sh file is used to import environment variables you set on your EC2 instances