Divers Web Site

Website of Saxy Divers Ultimate Leipzig. Created with Django Web Development Framework.

Features

add database entries with CSV Files (Events like recurrent trainings, birthdays, whatever, user lists, articles, ...), very easy.
history of articles and public infos are automatically created and can be reverted if mistakes happen. Also abuse can thus be tracked
send emails
event manager

install locally for development

Prerequisites:
- install Python use python 3.6.9 (this is works for sure) https://www.python.org/downloads/release/python-369/ Make sure to tick the box "Add to Path" or similar. This is important for your OS to find the python executable!!!
- On linux you may need to install pip sudo apt install python3-pip
- install git if necessary)
- Text editor of your choice (Visual Studio Code, Pycharm, Notepad, Vim, ...) (https://code.visualstudio.com/).
- sudo apt install postgresql libpq-dev
clone repository into new directory git clone git@github.com:flo-schu/diverssite.git
ideally with use of command line (cmd, terminal, etc.) navigate inside the directory.
create a virtual environment (https://docs.python.org/3/library/venv.html#creating-virtual-environments) inside the project folder (diverssite), in your console type: python -m venv venv if there are errors, read the error messages and proceed as suggested (The specific command may vary depending on your OS (Windows, Linux, Mac)) this will create a virtual environment called 'venv', which can be understood as putting a blank table in your office where you will add all the equipment that you need for the project
activate the environment (instructions on the same page as above): Windows: venv\Scripts\activate Linux/Mac: source venv\bin\activate To extend the previous analogy. This is like seating yourself at your newly placed desk.
install requirements (get all the tools for the project, luckily there is a list). Pip is the store where you can get all the tools update pip: python -m pip install --upgrade pip pip install -r requirements.txt or install pip install .
create a '.env' file. This files contains all custom settings, which are imported when the app is launched. A different version of this is used in production (when the site is online), which then contains only absolutely secret information. The info in he file used locally are not sensitive and therefore can be used without problems. This is why an example file is already included: rename the file '.env.example' to '.env' (this can be done normally in the explorer or with the command line) cp .env.example .env
The website is now set up. We need to apply some modifications and provide fixed datasets to the database for the website to be up and running. ````python manage.py migrateorpython3 manage.py migrate``` Migrations are important for existing databases. The use of 'python3' or 'python' depends on the OS you are using. Use the command that is working for the following commands as well.
And install fixed data in the database, which the site requires to run: python manage.py loaddata fixtures.json
That's it. We can test our server. Execute the following and then open your browser and go to site: 127.0.0.1:8000 python manage.py runserver
Create an admin user: python manage.py createsuperuser
For further infor refer to: https://docs.djangoproject.com/en/3.1/ and https://developer.mozilla.org/en-US/docs/Learn/Server-side/Django/Tutorial_local_library_website

Code Formatting and Conventions

The code is formatted using black. The imports are sorted using isort. Follow the respective installation instructions. Then run:

black .
isort .

in the repository root directory.

Hosting the website on a server

installation of the basic website

obtain a server and install OS ideally rent a server from an IaaS platform (e.g. https://www.strato.de/server/linux-vserver/) If necessary install a current OS on the server (instructions should be available on the hosting website)
set up server follow the instructions on https://www.digitalocean.com/community/tutorials/ initial-server-setup-with-ubuntu-18-04 If you are using a newer OS, there should be a more recent version, but most concepts should be the same

there are possibly some other apps which should be installed.
```
sudo apt-get nano  # install text editor for terminal
sudo apt-get install git  # install version control
chsh -s /bin/bash  # to change from dash to bash, in case this is not enabled by default somehow
```
register a user on the server which is responsible for ONLY the project see: https://djangodeployment.readthedocs.io/en/latest/03-users-and-directories.html
set up django app on server follow instructions on https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-18-04 until activating the virtual environment

if source myvenv/bin/activate command does not work use . myvenv/bin/activate

you should now have an activated environment shown by (myvenv) $ before the code prompt. Install some packages now
```
pip install gunicorn psycopg2-binary
```

clone git repository

git clone https://github.com/flo-schu/diverssite.git
pip install -r requirements.txt

set up environmental variables

touch .env
nano .env

paste these variables and replace the values with real stuff. Make sure

export server_ip=REAL_SERVER_IP
export engine=django.db.backends.postgresql_psycopg  # this turns on postgresql engine
export host=localhost
export db=REAL_DATABASE
export usr=REAL_DATABASE_USER
export pwd=REAL_PASSWORD
export SECRET_KEY=REAL_SECRET_KEY
export DJANGO_DEBUG=False
export SSL_REDIRECT=True
export CSRF_COOKIE_SECURE="False"
export SESSION_COOKIE_SECURE="False"

set -a; source ~/sites/diverssite/.env; set +a # try out if .env works

set up Django app on server

test whether server is running and apply migrations, install fixtures

python manage.py runserver  # only temporary to debug error messages, break afterwards
python manage.py migrate
python manage.py loaddata fixtures.json  # install fixtures of necessary hardcoded model objects
python manage.py createsuperuser

at this point the website should be running without obvious errors

sudo ufw allow 8000  # adds port 8000 to firewall
python manage.py runserver 0.0.0.0:8000  # test if server runs site

follow instructions on how to set up gunicorn and nginx. https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-18-04

to get started with nginx, check this guide. It is really helpful to understand what nginx does https://nginx.org/en/docs/beginners_guide.html

to reset the server after changes:
```
sudo systemctl daemon-reload && sudo systemctl restart gunicorn && sudo systemctl restart nginx
```
set up SSL certificate

follow the instructions on https://certbot.eff.org/lets-encrypt/ubuntubionic-nginx (set up for Ubuntu 18.04 and Nginx)
Set up the SMTP Server with Postfix and Dovecot

What is needed for the whole thing to work is an smtp server with an AUTH function. For this in turn I needed an DNS record for the mailserver, but actually everything is described in the next two tutorials. Those are brilliant. Work through them step by step. I added below where I deviated from the instructions https://www.linuxbabe.com/mail-server/setup-basic-postfix-mail-sever-ubuntu https://www.linuxbabe.com/mail-server/secure-email-server-ubuntu-postfix-dovecot

Warning: Here monit is used to automatically restart postfix. Monit caused problems in starting postfix. Prefer to not install it
- set hostname: This may actually be not required because later on we fix the hostname in postfix config sudo hostnamectl set-hostname mail.saxy-divers.de
- follow the rest of the tutorial
- open the aliases file sudo nano /etc/aliases
- add the line to aliases file, so that error mails are sent to an external mail in case the server breaks down root: your@mail.de
- updating certificate to email instead of creating a new one: certbot --expand -d saxy-divers.de,mail.saxy-divers.de
- dont use the auth_username_format = %n option. I think it will be simpler to just use usernames.
- enabling monit at the end of part 1 tutorial caused problems. Disabling it fixed postifx shutting down repeatedly
To check problems of postifx and dovecot inspect the log
- check mailbox: nano /var/mail/florian
- check log: nano /var/log/mail.log
- then add the environmental variables to the settings file and add them to the .env file on the server like under point 3. email_usr and email_pw müssen gesetzt sein. Dafür muss auf dem SMTP Server ein benutzer existieren. Dies sollte aber schon im Tutorial geschehen sein.
```
export email_tls=True
export email_default_from=ultimail@saxy-divers.de
export email_host=mail.saxy-divers.de
export email_usr=XXXXXX
export email_pw=XXXXXX
export email_port=587
```

Development

Migrations

always only migrate the app you have been working on e.g. python3 manage.py makemigrations wiki

for changes that affect a database retrospectively, use of django-extensions runscript is recommended

python3 manage.py runscript save_models

Maintenance

when you do changes on the server, first put the server into maintenance mode in the .env file set MAINTENANCE_MODE=on then sudo systemctl restart gunicorn reload the website. It should now display a message that the site is currently unavailable.

server

server restart

sudo hostnamectl set-hostname mail.saxy-divers.de (not sure if this is really necessary)
sudo systemctl restart gunicorn nginx postfix dovecot

after changes to the django app have been made

NEVER push settings from the develop branch
push if needed changes from main branch
ssh onto server ssh username@server
navigate into repository cd xyz
activate virtual environment: source divers_venv/bin/activate
pull changes from git repository.
migrate changes if necessary: python3 manage.py migrate
restart gunicorn: sudo systemctl restart gunicorn
check if site works. If server error occurs, it could be because changes rely on model values which have not been set. This can be directly tackled in the admin view --> https://mysite.de/admin
restart postfix: sudo systemctl restart postfix
restart dovecot: sudo systemctl restart dovecot
to monitor the status of gunicorn, nginx, dovecot, postfix replace restart with status
to create a new mailaccount: sudo useradd -m username sudo passwd username

certificates

NEVER delete CERTIFICATES!!! IT IS A PAIN IN THE ASS. IF YOU NEED TO https://eff-certbot.readthedocs.io/en/stable/using.html#safely-deleting-certificates

add subdomains

it is important to include the name of the certificate first (domina name) and then ALL other domains see which domains are included with certbot certificates certbot --expand -d existing.com -d example.com -d newdomain.com

renew certificates

certbot renew --pre-hook "service nginx stop" --post-hook "service nginx start"

trouble with nginx

if there are startup issues with nginx, sudo pkill -f nginx & wait $! can help. Followed by sudo service nginx start

modifying automated renewal

https://eff-certbot.readthedocs.io/en/stable/using.html#modifying-the-renewal-configuration-of-existing-certificates

Backup

see https://django-dbbackup.readthedocs.io/en/master/index.html

creating backups is done with:

python3 manage.py dbbackup
python3 manage.py mediabackup

for restoring backups see the documentation

upgrade system

first open a port on ufw wtih the rule 1022/tcp for ssh access on a different port and add it in
do it very slowly according to the instructions and make sure the port is added in sudo nano /etc/ssh/sshd_config as Port 1022

in case the upgrade messes with the previously used python version follow the to ansper in https://stackoverflow.com/questions/61541281/python-3-7-venv-broken-after-upgrade-to-ubuntu-20-04

there will be several updates to existing configuration files necessary it is recommended to start the services one by one and inspect error messages: sudo systemctl start SERVICE to check errormessages exchange start with status

gunicorn (potential python migration issues)
nginx (occupied port: https://stackoverflow.com/questions/42303401/nginx-will-not-start-address-already-in-use check used ports sudo netstat -tulpn)
postfix
dovecot (configuration files: /etc/dovecot/conf.d/10-ssl.conf /etc/dovecot/dovecot.conf)

disable unneeded services

sudo systemctl disable apache2.service --> blocks port 80

Known Issues and Fixes

admin site is not rendered correctly - Sidebar issue

sitebar is triggered out of unknown overflow reasons. Fix: Disable sidebar https://stackoverflow.com/questions/64016816/django-admin-sidebar-bug

Roadmap

Wünsche

Mailbenachrichtigung bei neuem turnier
kategorien anlegen