ihr-django

Internet Health Report API

This is the implementation for the IHR API: https://ihr.iijlab.net/ihr/en-us/api

📝 Table of Contents

basic installation for all
- If you wish to use your machine
- using docker
  - using docker with local database
  - using docker with image database
Add test data to the database

basic installation for all

Required packages for Ubuntu:

sudo apt install apache2 python3 python3-pip postgresql postgresql-contrib

Install virtualenv and django:

pip3 install virtualenv
virtualenv ihr

Install django:

cd ihr
. bin/activate
pip install django

Create a new django project:

django-admin startproject internetHealthReport

Copy IHR's django application :

cd internetHealthReport
git clone git@github.com:InternetHealthReport/ihr-django.git ihr

Then copy settings.py, urls.py, wsgi.py, Dockerfile,docker compose to the correct place:

cp ihr/config/*.py internetHealthReport/
cp ihr/config/Dockerfile .
cp ihr/config/docker-compose.yml .

You may have to adjust some variables in settings.py to match your database, smtp account, recapcha credentials.

If you wish to use your machine

install dependencies

pip install -r ihr/requirements.txt

make sure that the host of database in the settings is localhost

    cd internetHealthReport
    nano settings.py

scroll down to the DATABASES section and make sure that the host is localhost

Create the database and django user (change password as needed):

sudo su postgres
psql
postgres=# CREATE DATABASE ihr;
CREATE DATABASE
postgres=# CREATE USER django WITH PASSWORD '123password456';
CREATE ROLE
postgres=# ALTER ROLE django SET client_encoding TO 'utf8';
ALTER ROLE
postgres=# ALTER ROLE django SET timezone TO 'UTC';
ALTER ROLE
postgres=# GRANT ALL PRIVILEGES ON DATABASE ihr TO django;
GRANT
postgres=#\q
exit

Remove migration files and create tables: (TODO move production migration files to a different repository)

find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc"  -delete
./manage.py makemigrations
manage.py migrate

Start django:

./manage.py runserver

if you want to use docker

install docker

sudo apt install docker

make sure that the host of database in the settings is db

    cd internetHealthReport
    nano settings.py

scroll down to the DATABASES section and make sure that the host is db

make sure you are in the internetHealthReport directory

if you want to use docker with local postgres

Get your local ip address

hostname -I

you will get something like that

xxx.xxx.x.xx

copy the first IP address and paste it in the docker compose file in extra hosts section

    extra_hosts:
      - "database:xxx.xxx.x.xx"

allow your postgres to accept connections from outside

sudo nano /etc/postgresql/**/main/postgresql.conf

change the following line

#listen_addresses = 'localhost'

listen_addresses = '*'

allow your postgres to accept connections from outside

sudo nano /etc/postgresql/**/main/pg_hba.conf

add the following line

host    all             all            172.xx.0.00/16            md5

xx may vary depending on postgres version but in newer versions it is 20 else it could be 17

start the docker container

docker compose up

and of course you need to have a postgres database running on your machine

if you want to use docker with remote postgres

uncomment the postgres image and volume database in docker compose

    #   db:
    #     image: kartoza/postgis:9.6-2.4
    #     volumes:
    #       - postgres_data:/var/lib/postgresql/data/
    #     environment:
    #       - POSTGRES_USER=django
    #       - POSTGRES_PASSWORD=123password456
    #       - POSTGRES_DB=ihr

    # volumes:
    #   postgres_data:

make sure in settings in DATABASES the host is db not database

and then start the container

    docker compose up

If this is the first time to run the container, you need to apply the migration files

docker ps

you will find something like that

    CONTAINER ID   IMAGE                                        COMMAND                  CREATED          STATUS          PORTS                    NAMES
    1c1c1c1c1c1c   internethealthreport-django-app         "python manage.py ru…"       20 seconds ago   Up 19 seconds

copy the container id and run the following command

docker exec -it 1c1c1c1c1c1c /bin/bash

you will be inside the container

python manage.py migrate

congratulations, you have a running django server

Go to http://127.0.0.1:8000/hegemony/ to check if it is working.

Add test data to the database

In the production database some of the ids are changed to BIGINT. We should locally apply these changes before importing data:

psql -U django -d ihr -c "ALTER TABLE ihr_hegemony ALTER COLUMN id SET DATA TYPE bigint"
psql -U django -d ihr -c "ALTER TABLE ihr_hegemony_prefix ALTER COLUMN id SET DATA TYPE bigint"
psql -U django -d ihr -c "ALTER TABLE ihr_hegemony_country ALTER COLUMN id SET DATA TYPE bigint"
psql -U django -d ihr -c "ALTER TABLE ihr_atlas_delay ALTER COLUMN id SET DATA TYPE bigint"

Download a database snapshot and load it:

wget https://ihr-archive.iijlab.net/ihr-dev/psql-snapshot/2022-03-10/2022-03-10_psql_snapshot.sql.lz4 
lz4 2022-03-10_psql_snapshot.sql.lz4 
psql -U django ihr < 2022-03-10_psql_snapshot.sql

Running the application

Activate the python environment and lunch django server:

cd ihr 
. bin/activate
internetHealthReport/manage.py runserver