/MyAEGEE

A docker environment to bootstrap the whole membership system

Primary LanguageShellApache License 2.0Apache-2.0

AEGEE-Europe's Online Membership System

MyAEGEE

Description

The repository for the "Online Membership System" (OMS), an open-source project of the student/youth association AEGEE-Europe.

It makes use of docker, and docker-compose.

Read more about the project

Installation

Explanation of the installation are here. Explanation of why we're doing it this way is at the bottom.

A note on terminology:

  • The computer you run Virtualbox on is the HOST.
  • The created VM which runs Docker is the GUEST.

Pre-requisites: installations required

Install Virtualbox first, then Vagrant. Even if you have a linux box, this is very recommended. If you decide to not do it, sigh... but don't come to cry to us.

If you decide you know better than us, install docker and docker-compose on your Windows/Linux/Mac machine, instead of Virtualbox and Vagrant. (Make sure you install the correct versions: tested with Docker CE 19.03.1 and docker-compose 1.24.1)

Note: if you use Vagrant, Docker will be already automatically on the virtual machine.

Memory requirements for the VM bootstrapped with Vagrant: 2GB (i.e. you need a machine with at least 3GB physical RAM)

Pre-requisites: URL mapping

To be able to use advanced features, you are encouraged to edit (a script will do it for you, read below) the /etc/hosts file on the HOST machine (on windows: C:\Windows\system32\drivers\etc\hosts) to add the entry:

Vagrant case: 192.168.168.168 appserver.test my.appserver.test traefik.appserver.test portainer.appserver.test pgadmin.appserver.test

Pure docker case: 127.0.0.1 appserver.test my.appserver.test traefik.appserver.test portainer.appserver.test pgadmin.appserver.test

In the linux case, it is handled by start.sh. More on the launching of this script later. As a helper in the windows case, you have the script "run_as_win_administrator.bat" (not very advanced). You have to right-click it and click "run as administrator". It will tell you the line to copy (on another terminal that will open) and open the file you need to edit in notepad. Paste the content at the last line of the file, save, and exit.

Now you can install the system!

Install the web application (linux):

On the HOST

git clone --recursive https://github.com/AEGEE/MyAEGEE.git
cd MyAEGEE
./start.sh

You will have to wait for up to 20'. A message appears when the bootstrap completes, and you can check if it works in the ways described in the usage section.

See below for explanation of start.sh

Install the web application (non-linux):

On the HOST

git clone --recursive https://github.com/AEGEE/MyAEGEE.git
cd MyAEGEE
vagrant up

You will have to wait for up to 20'. A message appears when the bootstrap completes, and you can check if it works in the ways described in the usage section.

Configuration file

Everything related to the behaviour of the app is defined in the top-most .env file. Most important parameters are:

ENABLED_SERVICES: telling which parts of the system are enabled

MYAEGEE_ENV: telling in which mode the system is run

<servicename>_SUBDOMAIN: telling how to access a specific service

See below for more info.

Usage

Accessing it

After launching the system, you have two ways to check everything is working:

  1. on the HOST
# on the _HOST_ you run the following

vagrant ssh

# ...which connects you to the _GUEST_, where docker is.
# So now you can run:

docker ps

Let's run the commands and see that they should yield an output like

username@computername:~/Documents/aegee/MyAEGEE$ vagrant ssh
Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-111-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

 System information disabled due to load higher than 1.0

 * Kubernetes 1.19 is out! Get it in one command with:

     sudo snap install microk8s --channel=1.19 --classic

   https://microk8s.io/ has docs and details.

97 packages can be updated.
70 updates are security updates.


This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento
Last login: Fri Sep 18 14:13:43 2020 from 10.0.2.2

appserver[/vagrant]$ [master !?]

appserver[/vagrant]$ [master !?] docker ps
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS                 PORTS                                      NAMES
cbe689ea9ee2        aegee/discounts:dev                    "docker-entrypoint.s…"   5 hours ago         Up 5 hours (healthy)   8084/tcp                                   myaegee_discounts_1
ed1c20a4e65a        aegee/statutory:dev                    "docker-entrypoint.s…"   5 hours ago         Up 5 hours (healthy)   8084/tcp                                   myaegee_statutory_1
f60476fa4a6c        aegee/events:dev                       "docker-entrypoint.s…"   5 hours ago         Up 5 hours (healthy)   8084/tcp                                   myaegee_events_1
06add303acd0        aegee/core:dev                         "docker-entrypoint.s…"   5 hours ago         Up 5 hours (healthy)   8084/tcp                                   myaegee_core_1
72a7cb2d4100        postgres:10                            "docker-entrypoint.s…"   5 hours ago         Up 5 hours             5432/tcp                                   myaegee_postgres-statutory_1
316ced849d6f        postgres:10                            "docker-entrypoint.s…"   5 hours ago         Up 5 hours             5432/tcp                                   myaegee_postgres-core_1
f3110b11d9c6        postgres:10                            "docker-entrypoint.s…"   5 hours ago         Up 5 hours             5432/tcp                                   myaegee_postgres-discounts_1
9985f6cd3042        postgres:10                            "docker-entrypoint.s…"   5 hours ago         Up 5 hours             5432/tcp                                   myaegee_postgres-events_1
623356e7ef0d        dpage/pgadmin4:4.23                    "/entrypoint.sh"         6 hours ago         Up 6 hours             80/tcp, 443/tcp, 5050/tcp                  myaegee_pgadmin_1
0728da470558        erikdubbelboer/phpredisadmin:v1.11.4   "tini -- php -S 0.0.…"   6 hours ago         Up 6 hours             80/tcp                                     myaegee_redisadmin_1
8263f843df15        portainer/portainer:1.22.1             "/portainer -H unix:…"   6 hours ago         Up 6 hours             9000/tcp                                   myaegee_portainer_1
d056ce2c92e3        swaggerapi/swagger-ui:v3.28.0          "/docker-entrypoint.…"   6 hours ago         Up 6 hours             80/tcp, 8080/tcp                           myaegee_swagger_1
4ee964dede44        traefik:v1.7.4-alpine                  "/entrypoint.sh --we…"   6 hours ago         Up 6 hours (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   myaegee_traefik_1
30963e2faf63        aegee/portal:dev                       "docker-php-entrypoi…"   6 hours ago         Up 6 hours             80/tcp                                     myaegee_portal_1
49485e19dc87        aegee/frontend:dev                     "nginx"                  6 hours ago         Up 6 hours             80/tcp, 8083/tcp                           myaegee_frontend_1
e81bcc38bf76        aegee/nginx-static:latest              "nginx -g 'daemon of…"   6 hours ago         Up 6 hours (healthy)   80/tcp                                     myaegee_statutory-static_1
dc2ea7b33c97        aegee/nginx-static:latest              "nginx -g 'daemon of…"   6 hours ago         Up 6 hours (healthy)   80/tcp                                     myaegee_events-static_1

what do we see here? That we can connect to vagrant (we are inside the virtual machine with vagrant ssh), and that a bunch of containers are up (we ask the docker containers with docker ps). We'll see later that we should use some commands (make start) on the GUEST, so knowing how to access the virtual machine is important.

HOORAY! YOUR SYSTEM IS UP!

  1. you can navigate to it in your HOST web browser.

For accessing it, the three most important URLs (NB there is no https for development):

Service URL Description
The app (MyAEGEE) http://my.appserver.test What you're here for
Traefik http://traefik.appserver.test Quick test to see if everything works well
Portainer http://portainer.appserver.test Visual docker manager

Do you connect to any of these three URLs without troubles? HOORAY! YOUR SYSTEM IS UP!

See right below for the URL of extra services.

See at the bottom for the default credentials of MyAEGEE's fresh install.

Subdomains registered on traefik

read "subdomain.appserver.test"; e.g. you put in your HOST browser http://traefik.appserver.test

Subdomain What Container
my MyAEGEE frontend
portainer Easier container mgmt (development only) (under login) portainer
traefik Traefik's dashboard (under login) traefik
pgadmin Administration of databases (development only) (under login) pgadmin
www Website wordpress
wiki AEGEE's Wiki, the backbone of knowledge mediawiki

You can customise these subdomains by editing the .env file as mentioned above, and relaunching the script (see below about Makefile).

FIXME: For more detailed usage guides see this usage tips page. For container-specific usage guides see the container's repository.

Easy script to manipulate the installation

(At the end of this section there will be links to detailed explanations, don't panic!)

There is a file called Makefile that gives some easy shortcut to do stuff. This must be launched on the GUEST.

On first run of vagrant, the bootstrap target will be invoked (you don't need to do it). If you are stubborn and decide to not use Vagrant, you still don't have to invoke it (it is invoked by start.sh)

The general flow is that once you edit the .env file, make start should be run (on the GUEST) to update the running configuration.

You can invoke the easy scripting in the following way (this shell command must be run in the same folder of the Makefile):

On the GUEST

Command What
make bootstrap (init, build, start) in this order. (Run only the first time by vagrant/start.sh)
make init Initialise the system (most likely you don't need to launch this)
make build Build the containers registered in the .env file
make start Run the containers registered in the .env file
make monitor If you didn't enable kibana, then you may want to have a look at the logs through this
make live-refresh Updates the containers to the new version (if any) and restarts them
make stop/restart/hard-restart Just don't use them on the server, EVER
make bump Only for development: updates the submodules

Guest? Host? wtf? read the under the hood section, and the difference between start.sh and makefile.

Reading the logs

For now, if one wants to follow some specific logs, they have to invoke helper.sh manually e.g.

./helper.sh --monitor container1 container2...containerN

Likewise, for now if one wants to execute a command on a container they have to invoke helper.sh manually e.g.

./helper.sh --execute containername command

What's next?

It's your time to shine! Your system is set up, now it's up to you to create a new service, to improve an existing one, or to play with integrations leveraging existing services (e.g. wordpress, magento...). See the moving parts section that explain how to configure the system after you add something new.

For better development experience: the files in your HOST folder MyAEGEE are mounted inside the GUEST (location /vagrant), which means that if you create a file in the GUEST /vagrant folder you will find it in your HOST <your cloned dir>/MyAEGEE, and vice versa if you edit a file in your repository it is reflected in the VM. This way you can use the IDE you use normally on the system you use normally, without the need of getting accustomed to some new tool!

Make sure however to use an extension in your IDE called editorconfig! It will avoid pains especially if your HOST is a windows machine.

Contribute

You can read more about contributing on our confluence.

Issue tracker

We use JIRA as our preferred issue tracker.

Licence

Apache License 2.0, see LICENSE.txt for more information.

Under the hood

Virtualbox is a utility that lets people creating virtual machines (VM) on your computer. The created VM is the GUEST. The computer you run Virtualbox on is the HOST

Vagrant is used as a tool to define VMs characteristics, that will be then run through Virtualbox - in other words, it is used so we can write a manifesto that defines the characteristics of a VM, and the VM generated has always the same characteristics. It is useful in this case to model the development VM just as if it was the server on which we will run the application.

Make is a tool that, among other things, chains commands together. So, for instance, you write in the Makefile that a runs a specific long command, b a different long command, and you can call the commands with make a or make b. You can also write a command c which is a chain of a followed by b. We use it to set a 'flow' of operations that should be followed (e.g. as explained, make bootstrap chains 3 operations, and one such operation is used very often i.e. build and/or start). This is used on the GUEST.

start.sh runs either vagrant up or make bootstrap (according to how you want to run your system in local) so one has to literally only launch one command and it's set to be working, after the startup time of around 10-20 minutes (according to internet connection speed). This is used on the HOST.

start.sh and Makefile

start.sh

On the HOST, i.e. the machine that runs the virtual machine, you use start.sh which can either:

  • Start the vm
    • Use ./start.sh for normal development cycle: app runs in development mode
    • Use ./start.sh --fast for sysops/integration development cycle: app runs in production mode so you can concentrate on developing integration to the app, not the app itself
  • Reset the settings to recreate the virtual machine (./start.sh --reset). This is in case you experimented so hard that you made something exploooode. Doing so, you will lose the users and other content you created on your local instance of MyAEGEE, but this will not remove your source code.

If you are a know-it-all who doesn't want to use Vagrant, use ./start.sh --no-vagrant (but again, if you're in trouble you will only get superficial support from our side)

Makefile

On the GUEST, i.e. the virtual machine that runs docker, you use make which uses the Makefile (explained above).

Individual containers

For prerequisites and installation of individual containers, see their docker(/-compose) files, located in the (service)/docker folder in their respective repository.

For more detailed info, we hoped to have a better knowledge base here, it's not great right now but it's a something ¯\_(ツ)_/¯

Moving parts

.env

The file contains variables where e.g. you define the base url (aegee.test) and where will various app be reachable (e.g. my. for my.aegee.test to reach the frontend).

List of defined variables:

  • base url
  • subdomain urls
  • activated services
  • runtime environment
  • some default passwords
  • SMTP user/pass/server for mailer
  • Sendgrid user/pass
  • Superadmin credentials (for other services to read; it does not set them in the system)
  • folder locations

so for instance...

Example 1: you would use this file if you had a problem with 1 microservice and wanted to remove it from the setup. Note: the removal of the ms would not stop a container if there was one running already, so make sure you cleanup (not mandatory, just avoids headaches in case of troubleshooting and "oh, I forgot about this). How would you do that? Example: discounts

  • Edit .env file
  • Remove the service from the array ENABLED_SERVICES
  • docker stop myaegee_discounts_1 && docker rm myaegee_discounts_1 (on the GUEST)

Example 2: you would use this file if you wrote a new microservice and wanted to add it from the setup. How would you do that? Example: your service is into a folder called boombastic (at the same level of the folders core,events etc)

  • Edit .env file
  • Add the service at the end of the array ENABLED_SERVICES=<whatever is here already>:boombastic. Remember: all the services are separated by colon (:)
  • make start (on the GUEST)

docker-compose.yml

In the docker-compose files there are the definitions of where an app should be reached.

Docker-compose will use the variables defined above, and put them under the labels section of a container (if a container needs it). The labels section is parsed by traefik to route all the HTTP calls to the correct containers. In other words, this is where the values contained in the .env file are used to specify that the app replies on 'my.appserver.test' instead of e.g. 'magic.appserver.test'.

/etc/hosts

This file which is located on the HOST machine is used to add new subdomains for the services: e.g. add pgadmin to be able to go to pgadmin.appserver.test and use a visual tool for the databases. The important part is that this name of subdomain matches the one defined in the file .env

Troubleshooting / other

  • Remember to check what is running on your GUEST via docker ps (or portainer). Enter your GUEST with vagrant ssh (from your HOST).

  • If you are stuck and something doesn't seem to work, make sure you don't have a mismatch between /etc/hosts, the URL you type in the browser, and the address that the system expects. See file current-config.yml for that: it is a file which is generated every time make start is launched. It contains the description of the desired state of the app. NB: /etc/hosts is the only file mentioned in this README that only exists in the HOST and is not mounted (as described above).

  • You can add dev-tools to the array of services and troubleshoot docker using portainer (mentioned above). Make sure that you have added also the URL to the hostfile (/etc/hosts) and that it matches the variable defined in .env

  • As written in the 'moving parts' section: if you remove services from array ENABLED_SERVICES, they are not stopped. This is not a problem in general, just don't be surprised if when you start (having the configuration ENABLED_SERVICES=core:frontend:events) and stop later (with the configuration ENABLED_SERVICES=events) docker writes a message WARNING: found orphan containers.

  • make start calls a script helper.sh that will look into all folders described in ENABLED_SERVICES for a docker-compose.yml file. Then it creates in-memory a giant docker-compose configuration which is what will be run. This configuration is also outputted to a file (current-config.yml) for ease of troubleshoot.

  • Running the system in development mode means that every docker-compose.yml has extra-settings in a docker-compose.dev.yml file (in the same folder). Some modules are not thought for development mode, if you find an error about a missing docker-compose.dev.yml just create an empty one where the system wants it.

How to run unit/integration tests when you start the system

?? @serge1peshcoff

Default credentials

From the core readme: there are test users. All users have 5ecr3t5ecr3t as password.

  • admin@example.com - admin
  • board@example.com - board member of antenna
  • member@example.com - regular member of antenna
  • not-confirmed@example.com - member who is not confirmed
  • password-reset@example.com - member who requested a password reset
  • suspended@example.com - a suspended member

You can use 5ecr3t for a password reset token (for a member with email password-reset@example.com) and 5ecr3t for a mail confirmation (for a member with email not-confirmed@example.com).

NOTICE if you use the --fast mode of start.sh, then core does NOT provision in production mode, so you will be fast in bootstrapping but without users to play with. This should not be an issue for you if you are interested in only the infrastructure work; if you however need a user, you can register yourself.

## How to reset the database and recreate

A couple of options

  1. As mentioned above, you can enable dev-tools for pgadmin. From there you can delete the db
  • Notice unfortunately you have to configure pgadmin when you first login: specify the host, username, password.
    • Username and password of the database is NOT the same username and password of pgadmin. Find everything in the files mentioned in 'moving parts', or current-config.yml for ease.
    • You put the name of the service as hostname (docker internally resolves stuff with its internal DNS). In other words, every container is reachable at the host named like its service (e.g. frontend, core, postgres-core, etc). The 'service' is named in the docker-compose.yml under the key services:. Again: the title of the container is also the hostname of the container
  1. Use portainer to delete the service and db, then make start to fix everything.
  • The service must be deleted/restarted because it runs the migrations and therefore fills the DB with the important data. If you delete only the db, the service will expect the db to be filled with data, causing errors