/gadget-board

Showing latest data from mik4el's gadgets using Django (2.0.12), Angular 8 and Docker. Available at:

Primary LanguageJavaScriptApache License 2.0Apache-2.0

Introduction

This is a web app showing latest data from mik4el's gadgets using Django (2.0.12), Angular 8 and Docker. The app is currently available at https://m4bd.se.

The web app is separated into a backend web/gadget_board_backend using Django and a frontend web/gadget_board_frontend using Angular.

The full stack is running in Docker containers and consists of:

  1. "nginx_dev" or "nginx_prod": nginx for web server and serving static files.
  2. "web": Django served by gunicorn.
  3. "postgres": the postgres database.
  4. Persisting data volumes "postgres_data_dev" or "postgres_data" for the database.

This is also my way to learn what can be considered to be efficient development and deployment workflows in 2016. My goal's for the workflow is:

  • Closely mimick development environment and production environment
  • Quick feedback loop from code change to feature in development
  • Fast deployment to production of the entire stack as well as small updates
  • Entire stack is in repo
  • Easy to add services
  • Deployment agnostic for hosting providers

Suggested reading and inspiration for this repo: https://realpython.com/blog/python/django-development-with-docker-compose-and-machine/ (Good concept and introduction but some code is out-of-date and following instructions will not give a working setup, see blog comments for more info)

Requirements

  • Docker for mac or similar.
  • Node v10+.
  • Ng-cli installed globally.
  • VirtualBox

Setting up a development environment

This will start a new development environment and serve the web app on your machine. This requires the download of all depedencies which will take some time.

  1. docker-machine create -d virtualbox dev
  2. eval $(docker-machine env dev)
  3. Make your own .env-file with credentials e.g. cp .env_template .env
  4. docker-compose build
  5. docker-compose up -d
  6. docker-compose run --rm web python manage.py migrate
  7. docker-compose run --rm web python manage.py createsuperuser
  8. cd web/gadget_board_frontend
  9. ng build
  10. Open a browser at the ip from docker-machine ip dev

Development workflow Django

For normal development work, I suggest this workflow:

  1. Make change in Django related code, run manage.py commands using docker-compose run --rm web python manage.py startapp etc...
  2. Reload browser, run tests etc.

Special cases:

  • Make sure no old code is running if you are changing e.g. settings.py or urls.py by restarting the web container. Do this with docker-compose restart web.
  • Run manage.py commands using docker-compose run --rm, e.g. docker-compose run --rm web python manage.py makemigrations.
  • If you restart your computer etc, you may need to restart the machine running the containers, do so by:
    1. docker-machine start dev
    2. eval $(docker-machine env dev)(also when you open a new terminal)

Adding a dependency

When you add a dependency to web/requirements.txt you need to build a new container image and restart the container with this new image, this is done by:

  1. docker-compose build web
  2. docker-compose up -d
  3. Possibly run docker-compose run --rm web python manage.py collectstatic or other commands your dependency requires. NB: collectstatic needs to be run in the development workflow for static files to be saved in the static-dir, this is not possible to do on deployed containers.

Development workflow Angular

For normal development work, I suggest this workflow:

  1. ng build --watch
  2. Make change in source file in gadget_board_frontend
  3. Reload browser

Adding a dependency

See ng-cli documentation: https://github.com/angular/angular-cli#3rd-party-library-installation

First deployment

Now we need to set up the production environment to which you are deploying. By using Docker the production environment is very agnostic to what provider you choose. I like DigitalOcean for small projects that can grow but there are many options. Doing the first deployment requires you to migrate the database for the first time, also we use a different docker-compose file so you need to rebuild the container images.

  1. Get working SSL certs e.g. by following the steps below.
  2. Get an access token for your DigitalOcean account.
  3. docker-machine create -d digitalocean --digitalocean-access-token=<token> --digitalocean-region=fra1 production (use same region where your floating ip is for the domain you use in SSL)
  4. eval $(docker-machine env production)
  5. ng build --prod --output-path=../gadget_board_frontend_dist
  6. docker-compose -f production.yml build
  7. docker-compose -f production.yml up -d
  8. docker-compose -f production.yml run --rm web python manage.py migrate
  9. docker-compose -f production.yml run --rm web python manage.py createsuperuser

Setting up SSL

Since the app handles user data securing traffic using SSL is a requirement for production. To set it up on the production nginx container I followed these steps:

  1. Have a domain ready that you control. It is nice to point the domain towards a floating ip from e.g. DigitalOcean so you can change the production environment without needing to update your dns.
  2. Find a suitable SSL certificate authority (CA), I use positivessl from namecheap.com. For SSL certs with short expiration dates there are also free options but that is a hassle so I value buying an SSL cert. A better free option is https://letsencrypt.org/.
  3. Make a csr-file by running openssl req -newkey rsa:2048 -nodes -keyout example.com.key -out example.com.csr and go through the process of obtaining the cert. Save the key and csr file safely on your local machine.
  4. When you have got all the cert files back from your CA, prepare the cert for nginx by following e.g. https://www.namecheap.com/support/knowledgebase/article.aspx/9419/0/nginx. Store the files safely on your local machine.
  5. Download root certificate from your CA, e.g. https://www.namecheap.com/support/knowledgebase/article.aspx/9393/69/where-do-i-find-ssl-ca-bundle, save it as PositiveSSLBundle.cer
  6. Make a dhparam.pem file by running openssl dhparam 2048 and store safely.
  7. Copy over the files to a new directory sslin folder nginx. They will not be added to git.

Deploying

When you are deploying the next time we also need to rebuild the container that has changed files since the production environment does not mount your local machines files.

  1. ng build --prod --output-path=../gadget_board_frontend_dist
  2. docker-compose -f production.yml build
  3. docker-compose -f production.yml down
  4. docker-compose -f production.yml up -d
  • Or run the docker-compose commands together: docker-compose -f production.yml build && docker-compose -f production.yml down && docker-compose -f production.yml up -d NB: If in a new terminal remember eval $(docker-machine env production).

Testing

Testing for the backend is handled by the default Django test system, so running tests is easy, e.g:

  • docker-compose run --rm web python manage.py test

Testing for the frontend is done using angular-cli:

  • ng test

Backing up

A deployed environment can be backed up by your hosting provider, e.g. DigitalOcean. Since this is a very stateless deployment you can also make a scripted backup of your database and make it possible to easily restore the database from a backup. This will save some on hosting costs and make for a more self-contained and hosting provider agnostic solution.

Example command to run backup of postgres db on development machine:

  • docker-compose run --rm -e PGPASSWORD=postgres postgres pg_dump -U postgres -p 5432 -h postgres postgres > postgres_db_20160913_development.bak

To restore development machine:

  1. docker-compose up -d
  2. docker-compose run --rm -e PGPASSWORD=postgres postgres psql -U postgres -p 5432 -h postgres -F c < postgres_db_20160913_development.bak

Example command to create backup of postgres db on production machine:

  • docker-compose -f production.yml run --rm -e PGPASSWORD=postgres postgres pg_dump -U postgres -p 5432 -h postgres postgres > postgres_db_20190606_production.bak

To restore production machine:

  1. docker-compose -f production.yml up -d
  2. docker-compose -f production.yml run --rm -e PGPASSWORD=postgres postgres psql -U postgres -p 5432 -h postgres -F c < postgres_db_20180522_production.bak

The system has produced a lot of Gadget Data, in a year about 11M Gadget Data objects were stored. This becomes many GB easily so to make sure the system can run efficiently on a small VM it makes sense to purge Gadget Data from time to time. An easy and safe way to purge is to use the Django ORM, this is an example:

  1. docker-compose -f production.yml run --rm web python manage.py shell
  2. >>> from gadgets.models import GadgetData
  3. >>> GadgetData.objects.all().delete()

Todos

  1. Frontend update css to not look broken
  2. Frontend: Animation when data updated
  3. Refresh JWT tokens in background
  4. Sass or similar css build
  5. Tests for account service
  6. BUG: Redirect after new account didn’t work
  7. Create new password functionality
  8. Fullscreen logic in routes rather than in swimthermocomponent
  9. More info link for gadget detail
  10. Next bus gadget data
  11. Next bus gadget visualization
  12. Show historic data for gadgets
  13. Django Channels for e.g. Flight radar
  14. Fancy map visualization for Flight radar
  15. My own component class that animates, has standard css and so on.