TravisCI build:
| Name | Version | Dockerfile | Build |
|---|---|---|---|
| nginx | 1.13.x | Dockerfile |  |
| mssql | 2017 | Dockerfile |  |
| php53 | 5.3 | Dockerfile |  |
| php53-debug | 5.3 + xdebug 2.0.5 | Dockerfile |  |
| php54 | 5.4 | Dockerfile |  |
| php54-debug | 5.4 + xdebug 2.4.1 | Dockerfile |  |
| php55 | 5.5 | Dockerfile |  |
| php55-debug | 5.5 + xdebug 2.5.5 | Dockerfile |  |
| php56 | 5.6 | Dockerfile |  |
| php56-debug | 5.6 + xdebug 2.5.5 | Dockerfile |  |
| php70 | 7.0 | Dockerfile |  |
| php70-debug | 7.0 + xdebug 2.7.2 | Dockerfile |  |
| php71 | 7.1 | Dockerfile |  |
| php71-debug | 7.1 + xdebug 2.9.6 | Dockerfile |  |
| php72 | 7.2 | Dockerfile |  |
| php72-debug | 7.2 + xdebug 2.9.6 | Dockerfile |  |
| php73 | 7.3 | Dockerfile |  |
| php73-debug | 7.3 + xdebug 2.9.6 | Dockerfile |  |
| php74 | 7.4 | Dockerfile |  |
| php74-debug | 7.4 + xdebug 2.9.6 | Dockerfile |  |
This project aims to provide an easy way to start developing for Totara by providing a Docker setup.
This setup was created and tested intensively on a Mac OS and Linux. It works on Windows as well (best with WSL2).
Although this project started as a development environment for Totara Learn it can be used for any other PHP project.
- NGINX as a webserver
- PHP 5.3, 5.4, 5.5, 5.6, 7.0, 7.1, 7.2, 7.3, 7.4 to test for different versions
- PostgreSQL (9.3, 9.6, 10, 11, 12), MariaDB (10.2, 10.4 and 10.5) and MySQL (5.7.x and 8), and Microsoft SQL Server 2017 support
- A PHPUnit and Behat setup to run tests (including Selenium)
- A mailcatcher instance to inspect mails
- Redis for caching and/or session handling
- XHProf for profiling
- XDebug installed, ready for debugging with your favorite IDE
- Totara source code: https://help.totaralearning.com/display/DEV/Getting+the+code
- Docker: https://www.docker.com (for Docker on Mac OS please read the Performance section)
- Docker-compose: https://docs.docker.com/compose/install (included in Docker for Mac/Windows)
- Mutagen v0.10.0+: http://mutagen.io/ (optional, for more speed on Mac, not needed for Linux)
- At least 3.25GB of RAM for MSSQL
Please check out the contribute page for more information on how you can help us.
- Clone the Totara source code (see requirements)
- Clone this project
- Install mutagen (optionally, recommended for Mac OS)
- Copy the file .env.dist to .env and change at least the path to your local Totara source folder (LOCAL_SRC)
- Make sure you have all the hosts in your /etc/hosts file to be able to access them via the browser
Example:
127.0.0.1 localhost totara54 totara54.debug totara54.behat totara55 totara55.debug totara55.behat totara55 totara55.debug totara56.behat totara70 totara70.debug totara70.behat totara71 totara71.debug totara71.behat totara72 totara72.debug totara72.behat totara73 totara73.debug totara73.behat totara74 totara74.debug totara74.behatIf you are already using the docker setup, but you want to make sure you get the latest changes and features:
- check out the newest release upgrade notes
- make sure you pull the latest code from this repository
- and use the
tpullscript in the bin/ folder to pull the latest images tupany already running containers to apply changes
tpull [all] # updates all images already present locally by pulling the latest changes from docker hub
tpull nginx # to update a specific image use the last part of the repository name, for example nginx resolves to docker-dev-nginx
tpull php73Alternatively to pulling the pre-built images you can also rebuild themselves by using tbuild [container], for example tbuild php-7.3. Please note that rebuilding the images can take some time.
To speed up performance you can use a sync tool called mutagen.
This is especially relevant for Mac OS and Windows as the performance of mounted volumes on those platforms is really bad. If you are using Linux you can skip this as performance there is pretty good, almost native.
See chapter mutagen for instructions on how to set it up.
It is recommended to specify the containers you really need. The minimum you probably need is the db and the php container of your choice, the nginx container is started automatically alongside the php container.
The scripts for the following commands are located in the bin/ folder of this project. Either run the commands directly, like bin/tup, or add the bin folder to your PATH to not bother about your current folder.
tup pgsql php-7.3If you need additional containers at a later point just run tup with the container you need:
tup php-5.6
tup mariadb
tup selenium-hubThis starts a lot of containers so consider running only those you really need.
tup# this just stops the containers, equivalent to docker-compose stop
tstop
# this shuts all containers (and pauses an existing mutagen session) down, equivalent to docker-compose down
tdownThis project comes with a few bash scripts to simplify usage across platforms. The scripts are located in the bin/ folder. Ideally you add the bin folder to your PATH environment variable so you can run the commands from anywhere.
tbash [container] # log into a container via bash, i.e. php-7.2
tbuild [container] # build (all) container(s)
tdb [options] # run common actions for your databases
tdocker # shortcut to general docker-compose ... command
tdown # shutdown all containers
tgrunt [options] # run grunt in container, supports running in subfolders
tnpm [options] # run npm in container, supports running in subfolders
tpull # pull latest images (only those which you already have locally)
trestart [container] # restart (all) container(s)
tscale [container] [number] # scale up the number of containers, i.e. `tscale selenium-chrome 6`
tstats # show docker stats including container names
tstop [container] # stop (all) container(s)
tunit [container] [folder] [init] # run or init unit tests in given container for given version
tup [containers] # start (all) container(s)
tzsh [php container] # log into a php container via oh my zsh, i.e. php-7.2It is recommended to check out each Totara Learn version in a different subfolder below the folder LOCAL_SRC defined in .env. This enables you to access different versions without having to switch branches all the time.
Make sure you have configured Totara and created the databases you need. You can connect to the databases from your host using the tdb command, or any tools you prefer (host = localhost, use default ports).
The tdb command allows you to easily interact with any of the 4 supported DBMSes in a simple and consistent way. The script allows you to create, drop, backup and restore any database without having to remember the specific commands for each dbms. To get started, simply define your database $CFG variables in your config.php, then run tdb from your totara site directory (not in a container)
Alternatively, you can manually configure your databases via the following credentials and commands.
| DB | Host | User | Password |
|---|---|---|---|
| PostresSQL 12 (latest) | pgsql | postgres | |
| PostresSQL 11 | pgsql11 | postgres | |
| PostresSQL 10 | pgsql10 | postgres | |
| PostresSQL 9.6 | pgsql96 | postgres | |
| PostresSQL 9.3 | pgsql93 | postgres | |
| Mysql 8 | mysql8 | root | root |
| Mysql 5.7 | mysql | root | root |
| MariaDB 10.5 | mariadb | root | root |
| MariaDB 10.4 | mariadb104 | root | root |
| MariaDB 10.2 | mariadb102 | root | root |
| Mssql | mssql | SA | Totara.Mssql1 |
To use the command line clients provided by the containers you can use the following commands:
# PostgreSQL
tdocker exec pgsql psql -U postgres
# MySQL / MariaDB
tdocker exec mysql mysql -u root -p"root"
tdocker exec mariadb mysql -u root -p"root"
# Microsoft SQL Server
tdocker exec php-7.1 /opt/mssql-tools/bin/sqlcmd -S mssql -U SA -P "Totara.Mssql1"Create a database for each Totara version you would like to develop on.
Example commands:
# PostgreSQL
CREATE DATABASE totara_13;
# MariaDB/MySQL 5.7
CREATE DATABASE totara_13 DEFAULT CHARACTER SET utf8 COLLATE utf8_bin;
# MySQL 8
CREATE DATABASE totara_13 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_cs;
# MSSQL
CREATE DATABASE totara_13 COLLATE Latin1_General_CS_AS ALTER DATABASE totara_13 SET ANSI_NULLS ON ALTER DATABASE totara_13 SET QUOTED_IDENTIFIER ON ALTER DATABASE totara_13 SET READ_COMMITTED_SNAPSHOT ON;
GOThe nginx container automatically creates a bunch of data folders ready to be used.
All data directories have to be created within the /var/www/totara/data directory to be part of the data volume and be persistent even after shutting down docker.
/var/www/totara/data/ver[versionnumber].[database]
/var/www/totara/data/ver[versionnumber].[database].phpunit
/var/www/totara/data/ver[versionnumber].[database].behat
# example
/var/www/totara/data/ver13.pgsql
/var/www/totara/data/ver13.pgsql.phpunit
/var/www/totara/data/ver13.pgsql.behatversionnumber = 22, 24, 25, 26, 27, 29, 9, 10, 11, 12, 13, 14 database = pgsql, mysql, mssql
To create a custom data directory just log into the nginx container (tbash nginx) and then create your custom folder inside /var/www/totara/data.
You may need to create custom directories for multiple installations. If you do this, you will also need to set the correct permissions on custom directories that you create.
tbash php-7.3
cd /var/www/totara/data
mkdir your_custom_folder
chown www-data:www-data your_custom_folder
chmod g+s your_custom_folder
# Now you can use /var/www/totara/data/your_custom_folder as your dataroot
This is an example for the t13 branch with the 3 different databases and the correct data directories. Please note: You will need additional configuration parameters for PHPUnit and Behat. Please refer to Totara docs and have a look at config-dist.php/config-example.php for examples.
//$CFG->dbtype = 'mysqli';
//$CFG->dbhost = 'mysql'; // eg 'localhost' or 'db.isp.com' or IP
//$CFG->dbuser = 'root'; // your database username
//$CFG->dbpass = 'root'; // your database password
//$CFG->dataroot = '/var/www/totara/data/ver13.mysql';
//$CFG->phpunit_dataroot = '/var/www/totara/data/ver13.mysql.phpunit';
//$CFG->dbtype = 'sqlsrv';
//$CFG->dbhost = 'mssql'; // eg 'localhost' or 'db.isp.com' or IP
//$CFG->dbuser = 'SA'; // your database username
//$CFG->dbpass = 'Totara.Mssql1'; // your database password
//$CFG->dataroot = '/var/www/totara/data/ver13.mssql';
//$CFG->phpunit_dataroot = '/var/www/totara/data/ver13.mssql.phpunit';
$CFG->dbtype = 'pgsql'; // 'pgsql', 'mariadb', 'mysqli', 'mssql', 'sqlsrv'
$CFG->dbhost = 'pgsql'; // eg 'localhost' or 'db.isp.com' or IP
$CFG->dbuser = 'postgres'; // your database username
$CFG->dbpass = ''; // your database password
$CFG->dataroot = '/var/www/totara/data/ver13.pgsql';
$CFG->phpunit_dataroot = '/var/www/totara/data/ver13.pgsql.phpunit';
$CFG->dblibrary = 'native'; // 'native' only at the moment
$CFG->dbname = 'totara_13'; // database name, eg moodle
$CFG->prefix = 'mdl_'; // prefix to use for all table names
$CFG->dboptions = array(
'dbpersist' => false, // should persistent database connections be
// used? set to 'false' for the most stable
// setting, 'true' can improve performance
// sometimes
'dbsocket' => false, // should connection via UNIX socket be used?
// if you set it to 'true' or custom path
// here set dbhost to 'localhost',
// (please note mysql is always using socket
// if dbhost is 'localhost' - if you need
// local port connection use '127.0.0.1')
'dbport' => '', // the TCP port number to use when connecting
// to the server. keep empty string for the
// default port
);
// This detects and sets the wwwroot dynamically so you don't have to manually change it
if (!empty($_SERVER['SERVER_NAME']) && !empty($_SERVER['REQUEST_SCHEME'])) {
$dir = __DIR__;
$parts = explode('/', $dir);
$base_path = '/' . array_pop($parts);
if (file_exists(__DIR__.'/server/version.php')) {
$base_path .= '/server';
}
$parts = explode('.', $_SERVER['SERVER_NAME']);
$wwwroot = $_SERVER['REQUEST_SCHEME'] . '://';
if (count($parts) > 1) {
// debug or behat
if ($parts[1] == 'behat') {
$wwwroot .= $parts[0];
} else {
$wwwroot .= $_SERVER['SERVER_NAME'];
}
} else {
$wwwroot .= $parts[0];
}
$wwwroot .= $base_path;
$CFG->wwwroot = $wwwroot;
} else {
// Fallback for CLI
$CFG->wwwroot = "http://totara73/perform/server";
}
// Recommended behat configuration options
$CFG->behat_config = array(
'default' => array(
'extensions' => array(
'Behat\MinkExtension' => array(
'browser_name' => 'chrome',
'default_session' => 'selenium2',
'selenium2' => array(
'browser' => 'chrome',
'wd_host' => 'http://selenium-hub:4444/wd/hub',
'capabilities' => array(
"browser" => "chrome",
"browserName" => "chrome",
"version" => '',
'platform' => 'LINUX'
)
)
)
)
),
);
// Point this to your subfolder if you are using any
$CFG->behat_wwwroot = 'http://totara73.behat/perform/server';
//$CFG->behat_dataroot = '/var/www/totara/data/ver13.mysql.behat';
//$CFG->behat_dataroot = '/var/www/totara/data/ver13.mssql.behat';
$CFG->behat_dataroot = '/var/www/totara/data/ver13.pgsql.behat';
$CFG->behat_prefix = 'bht_';
// For parallel runs modify the following, use different dbs or different prefixes
$CFG->behat_parallel_run = [
[
'dbname' => 'totara_13_behat1',
'behat_prefix' => 'bht_',
],
[
'dbname' => 'totara_13_behat2',
'behat_prefix' => 'bht_',
],
// ...
];
// Uncomment the following to enable screenshots being taken for failed steps
// $CFG->behat_faildump_path = __DIR__ . '/screenshots/';
// Useful options for development
// Force a debugging mode regardless the settings in the site administration
@error_reporting(E_ALL | E_STRICT);
@ini_set('display_errors', '1');
$CFG->debug = (E_ALL | E_STRICT);
$CFG->debugdisplay = 1;
// If you want performance information being displayed
$CFG->perfdebug = 15;
// Prevent caching
$CFG->langstringcache = false;
$CFG->cachejs = false; // NOT FOR PRODUCTION SERVERS!
// Only for t13
$CFG->tuidesignermode = true;
$CFG->cache_graphql_schema = false;
$CFG->forced_plugin_settings['totara_tui'] = [
'cache_js' => false,
'cache_scss' => true,
'development_mode' => true
];Make sure your config file contains the PHPUnit configuration needed and the database is ready.
Log into one of the PHP containers:
tbash php-5.6
tbash php-7.2
# or if you need xdebug support
tbash php-5.6-debug
tbash php-7.2-debugIf your project is in a subfolder:
cd subfolderIf needed initiate the PHPUnit setup:
php admin/tool/phpunit/cli/init.php
# for t13 use
php test/phpunit/phpunit.php initStart the testsuite:
vendor/bin/phpunit
# for t13 use
php test/phpunit/phpunit.php runMake sure your config file contains the Behat configuration needed and the database is ready.
If you want to run selenium tests make sure the selenium container is started:
tup selenium-huband you have a behat_local.yml file in your root code folder that provides the correct configuration (e.g. points to the selenium-hub host and the base_url is correct).
Log into one of the PHP containers:
tbash php-5.6
tbash php-7.2
# or if you need xdebug support
tbash php-5.6-debug
tbash php-7.2-debugIf your project is in a subfolder:
cd subfolderIf needed initiate the behat tests
# use --parallel=x if needed
php admin/tool/behat/cli/init.php
# for t13 use
php test/behat/behat.php initRun behat with:
# for t11
vendor/bin/behat
# for others use the command prompted after init, for example:
vendor/bin/behat --config /var/www/totara/data/ver9.pgsql.behat/behatrun/behat/behat.yml
# for t13 use
php test/behat/behat.php run
# for parallel run
php test/behat/behat.php behat_runBy default, prebuilt images from Docker Hub will be used. If you want to modify any of the containers to your needs then you can rebuild them locally with the following command:
tbuild
# or for individual images
tbuild php-7.2You can run the cron manually by logging into a php container and from your source root run:
php admin/cli/cron.php
# for t13
php server/admin/cli/cron.php
You can also use the cron containers to run the cron automatically using crontab. Just create your own crontab files within the cron.d folder and start a cron container like:
# in the foreground
tdocker up php-7.2-cron
# in the background
tup php-7.2-cron
# access the logs anytime with
tdocker logs -f php-7.2-cron
# stop a daemonized cron container
tstop php-7.2-cronThe setup comes with mailcatcher support. Just add the following to your config and all mails will be sent to it:
$CFG->smtphosts = 'mailcatcher:25';Open http://localhost:8080 to open the mailcatcher GUI.
If needed, modify the local port in the docker-compose.yml file.
If you want to use npm you can use tnpm like this:
# if your project lives in a subfolder then run the command from inside that folder
tnpm install
tnpm run tui-dev
tnpm run tui-watchIf you want to use grunt you can use tgrunt like this:
# if your project lives in a subfolder then run the command from inside that folder
# for t13 make sure you run this from within the server directory and run `tnpm init` there first
tgrunt
tgrunt gherkinlint
tgrunt cssOr you can just directly log in to the container directly run node/grunt commands:
tdocker run nodejs bash
# go to your source directory and
npm install
./node_modules/.bin/gruntThis should work on all platforms but is especially relevant for Mac OS and Windows as the performance of mounted volumes on those platforms is really bad. If you are using Linux you can skip this as performance there is pretty good, almost native.
Mutagen is a two-way-sync tool with focus on performance. Read more about it here: https://mutagen.io.
It runs in the background and keeps syncing your files onto a mounted volume inside your docker containers. It's pretty performant and the delay is minimal even if you change a lot of files at once.
To use mutagen first install it. On Mac OS you can use homebrew for that or alternatively download the appropriate release file from https://github.com/havoc-io/mutagen/releases
brew install havoc-io/mutagen/mutagenMake sure you have at least version 0.10.0 runnning.
To have mutagen automatically start up with your machine
mutagen daemon registerThen start the daemon. This is a background process without the sync does not work. If you have registered the daemon with the command above you won't need to do this every time.
mutagen daemon startTo activate the use of mutagen copy the file .use-mutagen.dist to .use-mutagen.
cp .use-mutagen.dist .use-mutagenIf you then use the commands tup and tdown as described in the following chapters the correct sync session is automatically created for you.
To find out if your sync is working you can use the following command:
mutagen sync listwhich shows something like:
⇒ mutagen sync list
--------------------------------------------------------------------------------
Name: totara
Identifier: ddc06807-2554-47f4-ba8a-230f37c5e577
Labels: None
Alpha:
URL: /your/local/path/to/totara/src
Connection state: Connected
Beta:
URL: docker://totara_sync/var/www/totara/src
DOCKER_HOST=
DOCKER_TLS_VERIFY=
DOCKER_CERT_PATH=
Connection state: Connected
Status: Watching for changes
--------------------------------------------------------------------------------You can use the session id or any part of the paths to monitor the session, for example:
mutagen sync monitor totaraYou can customise the docker compose configurations simply adding your own .yml or .yaml compose files into the custom folder. Any containers or other options you have will automatically override any existing default container options.
The shell folder lets you add custom aliases and functions to your php containers. Any file with the .sh extension will be sourced into your php container whenever you bash/sh into it. This is useful for when you need to run complex commands often during development, such as initialising tests. To get started, simply copy aliases.sh.dist to aliases.sh and define your aliases.
Oh My Zsh is an extention for the standard zsh shell. You can use it with the php containers instead of bash by using the tzsh command.
It is better than the basic bash shell as it brings colour support, autocompletion, autosuggestions and more.
To begin using it, you will need to install custom fonts and configure your terminal emulator to use them.
Oh My Zsh is highly configurable - see shell/.zshrc for the current configuration and check out the ohmyzsh and powerline10k docs for more ideas on what is possible.