Effortless data collection and curation
- Gather
Gather is an ODK-compatible data collection tool. It is built on top of the Aether framework. If you want to try it out, the easiest way is to follow the instructions on the Gather microsite.
git clone git@github.com:eHealthAfrica/gather.git
cd gather
./scripts/prepare-containers.sh
IMPORTANT NOTE: the docker compose files are intended to be used exclusively for local development. Never deploy these to publicly accessible servers.
# gather
127.0.0.1 gather.local
# aether
127.0.0.1 aether.local
Note: Make sure you have openssl installed in your system.
./scripts/generate-credentials.sh > .env
This instruction is included in the ./scripts/prepare-containers.sh
script.
Most of the environment variables are set to default values. This is the short list of the most common ones with non default values. For more info take a look at the files docker-compose-base.yml and /scripts/generate-credentials.sh.
See also Django settings.
See also Aether Django SDK environment variables.
-
Gather specific:
INSTANCE_NAME
:Gather 3
identifies the current instance among others.
-
Data export:
EXPORT_MAX_ROWS_SIZE
: between0
and1048575
indicates the maximum number of rows to include in the export file. The limit is an Excel 2007 restriction.
-
uWSGI specific:
CUSTOM_UWSGI_ENV_FILE
Path to a file of environment variables to use with uWSGI.CUSTOM_UWSGI_SERVE_STATIC
Indicates if uWSGI also serves the static content. Isfalse
if unset or set to empty string, anything else is consideredtrue
.- Any
UWSGI_A_B_C
Translates into thea-b-c
uswgi option.
See more in https://uwsgi-docs.readthedocs.io/
-
Django specific:
ADMIN_USERNAME
:admin
the setup script will create a superuser with this username. There is no default value.ADMIN_PASSWORD
:secresecret
the setup script will create the superuser with this password. There is no default value.DB_NAME
:gather
Postgres database name.WEB_SERVER_PORT
:8105
Web server port.
-
Aether specific:
-
EXTERNAL_APPS
:aether-kernel,aether-odk
Comma separated list with the available modules. To avoid confusion, the values will match the container name prependingaether-
,kernel
,odk
. -
Aether Kernel:
AETHER_KERNEL_TOKEN
:aether_kernel_admin_user_auth_token
Token to connect to Aether Kernel Server.AETHER_KERNEL_URL
:http://kernel:8100
Aether Kernel Server url.
-
Aether ODK:
AETHER_ODK_TOKEN
:aether_odk_admin_user_auth_token
Token to connect to Aether ODK Server.AETHER_ODK_URL
:http://odk:8102
Aether ODK Server url.
-
Aether consumers:
CONSUMERS_CONFIG_FILE
: Path to default consumers configurations. Edit./app/conf/consumers.json
to include new consumers.AUTO_CONFIG_CONSUMERS
: Flag used to indicate if consumers be automatically configure on the creation of a survey. mandatoryCONSUMERS_CONFIG_FILE
must be provided.
Note To include sensitive properties to
CONSUMERS_CONFIG_FILE
such as credentials, add an environment variable in the format{CONSUMER}_{RESOURCE}_{VARIABLE}
. For instance to add akibana
username and password to theES
consumer, the environment variable will be:ES_KIBANA_USERNAME
, andES_KIBANA_PASSWORD
. For the elasticsearch instance.ES_ELASTICSEARCH_USERNAME
, andES_ELASTICSEARCH_PASSWORD
-
docker compose up
This will start:
- gather on
http://gather.local/
. - aether-kernel on
http://aether.local/kernel/
. - aether-odk on
http://aether.local/odk/
. - aether-ui on
http://aether.local/
.
All the created superusers have username ${ADMIN_USERNAME}
and
password ${ADMIN_PASSWORD}
in each container.
The communication with the Aether servers is done via token authentication.
In gather
there are tokens per user to connect to other servers.
This means that every time a logged in user tries to visit any page that requires
to fetch data from any of the other apps, aether-kernel
, aether-odk
,
the system will verify that the user token for that app is valid or will request
a new one using the global token, AETHER_<<APP>>_TOKEN
;
the user token is going to be used for all requests and will allow the system to
track better the user actions.
Warning: The global AETHER_<<APP>>_TOKEN
needs to belong to an admin user,
because only those users can create new users with tokens in the Aether apps.
All development should be tested within the container, but developed in the host folder. Read the docker-compose-base.yml file to see how it's mounted.
If you want to develop with your local Aether images use the file docker-compose-local.yml and change the container paths to your current paths.
Build local aether and gather containers
./scripts/prepare-containers-local.sh
Start local aether and gather containers
docker compose -f docker-compose-local.yml up
The code style is tested:
- In python with flake8.
Defined in the file
app/setup.cfg
. - In javascript with standard.
- In styles with sass-lint.
Defined in the file
app/gather/assets/conf/sass-lint.yml
.
# Python files
docker compose run --rm --no-deps gather test_lint
# Javascript files
docker compose run --rm --no-deps gather-assets eval npm run test-lint-js
# CSS files
docker compose run --rm --no-deps gather-assets eval npm run test-lint-sass
There are a couple of naming/coding conventions followed by the Python modules and the React Components:
-
Names are self-explanatory like
export_project
,RefreshingSpinner
,ProjectList
,constants
and so on. -
Case conventions:
- Javascript specific:
- component names use title case (
TitleCase
) - utility file names use kebab case (
kebab-case
) - method and variable names use camel case (
camelCase
)
- component names use title case (
- Python specific:
- class names use title case (
TitleCase
) - file, method and variable names use snake case (
snake_case
)
- class names use title case (
- Javascript specific:
-
Javascript specific:
- Meaningful suffixes:
Container
indicates that the component will fetch data from the server.List
indicates that the data is a list and is displayed as a table or list.Form
indicates that a form will be displayed.
- The file name will match the default Component name defined inside, it might be the case that auxiliary components are also defined within the same file.
- App "agnostic" components are kept in folder
app/gather/assets/apps/components
- App "agnostic" methods are kept in folder
app/gather/assets/apps/utils
- Meaningful suffixes:
Comments are warmly welcome!!!
Each commit message consists of a header, a body and an optional footer. The header has a special format that includes a type, a scope, and a subject:
<type>(<scope>): <subject>
<BLANK LINE>
<body>
<BLANK LINE>
<footer>
- build: Changes that affect the build system or external dependencies
- ci: Changes to CI configuration files and scripts
- docs: Documentation only changes
- feat: A new feature
- fix: A bug fix
- perf: A code change that improves performance
- refactor: A code change that neither fixes a bug nor adds a feature
- style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
- test: Adding missing tests or correcting existing tests
The candidates for scope depends on the project and the technologies being used. It is fairly up to the developer to select a scope.
The subject contains a succinct description of the change:
- use the imperative, present tense: “change” not “changed” nor “changes”
- don’t capitalize the first letter
- no dot/period (.) at the end
Footer is used for citing issues that this commit closes (if any).
To learn more about the Aether release process, refer to the release management page on the wiki.
Frontend assets include JS, CSS, and fonts. They are all handled by webpack.
Frontend assets are mounted on the pages via the django-webpack-loader.
-
There is a file with all the apps list:
app/gather/assets/conf/webpack.apps.js
. -
There are three webpack configuration files:
app/gather/assets/conf/webpack.common.js
-- contains the common features to build the webpack files.app/gather/assets/conf/webpack.server.js
-- starts the server in port3005
with Hot Module Replacement (HMR).app/gather/assets/conf/webpack.prod.js
-- compiles the files to be used in the Django app.
-
The
start_dev
entry point starts a webpack development server (port3005
), that watches assets, rebuilds and does hot reloading of JS Components.docker compose up gather-assets
-
The
build
entry point compiles the files to be used in the Django app. The resultant files are kept in theapp/gather/assets/bundles
folder.docker compose run --rm gather-assets build
-
The CSS build is separate, and can contain both
.sass
and.css
files. They spit out a webpack build calledstyles.css
. -
Each page has their own JS entry point (needs to be defined in
webpack.apps.js
). On top of that, they load a common chunk, containingbootstrap
,popper.js
and other stuff that thewebpack common chunk
plugin finds is shared between the apps.
The list of the main containers:
Container | Description |
---|---|
db | PostgreSQL database |
gather | Gather app |
gather-assets | Gather assets module |
kernel | Aether Kernel app |
odk | Aether ODK Collect Adapter app (imports data from ODK Collect) |
ui | Aether Kernel UI (only needed for advanced mapping functionality) |
All the containers definition for development can be found in the docker-compose-base.yml file.
The pattern to run a command is always
docker compose run --rm [--no-deps] <container-name> <entrypoint-command> <...args>
If there is no interaction with any other container then include the option --no-deps
.
See more in docker compose run.
The app/entrypoint.sh script offers a range of commands to start services or run commands. The full list of commands can be seen in the script file.
The following are some examples:
Action | Command |
---|---|
List predefined commands | docker compose run --rm --no-deps gather help |
Run tests | docker compose run --rm gather test |
Run code style tests | docker compose run --rm --no-deps gather test_lint |
Run python tests | docker compose run --rm gather test_coverage |
Create a shell inside the container | docker compose run --rm --no-deps gather bash |
Execute shell command inside the container | docker compose run --rm gather eval <command> |
Run django manage.py | docker compose run --rm gather manage help |
Create a python shell | docker compose run --rm gather manage shell |
Create a postgresql shell | docker compose run --rm gather manage dbshell |
Show ORM migrations | docker compose run --rm gather manage showmigrations |
Create pending ORM migration files | docker compose run --rm gather manage makemigrations |
Apply pending ORM migrations | docker compose run --rm gather manage migrate |
Check outdated python libraries | docker compose run --rm --no-deps gather eval pip list --outdated |
Update outdated python libraries | docker compose run --rm --no-deps gather pip_freeze |
Start django development server | docker compose run --rm gather start_dev |
Start uwsgi server | docker compose run --rm gather start |
The app/gather/assets/conf/entrypoint.sh script offers a range of commands to start services or run commands. The full list of commands can be seen in the script file.
The following are some examples:
Action | Command |
---|---|
List predefined commands | docker compose run --rm gather-assets help |
Run tests | docker compose run --rm gather-assets test |
Run code style tests | docker compose run --rm gather-assets test_lint |
Run JS tests | docker compose run --rm gather-assets test_js |
Create a shell inside the container | docker compose run --rm gather-assets bash |
Execute shell command inside the container | docker compose run --rm gather-assets eval <command> |
Check outdated node libraries | docker compose run --rm gather-assets eval npm outdated |
Build assets used in the Django app | docker compose run --rm gather-assets build |
Start webpack server with HMR | docker compose run --rm gather-assets start_dev |
The Python code is tested using coverage.
The CSS style is analyzed by Sass Lint.
The Javascript style is analyzed by Standard JS.
The Javascript code is tested using Jest and Enzyme.
- Python test files are kept in the folder
tests
of each module and the name istest_my_module.py
. - Javascript test files are kept in the same folder and the name
is
MyComponent.spec.jsx
ormy-utility.spec.jsx
.
This will stop ALL running containers and execute gather
tests.
./scripts/test.sh
or
docker compose run --rm gather test
docker compose run --rm gather-assets test
or
docker compose run --rm gather test_lint
docker compose run --rm gather test_coverage
docker compose run --rm gather-assets test_lint
docker compose run --rm gather-assets test_js
# more detailed
docker compose run --rm gather-assets eval npm run test-lint-sass
docker compose run --rm gather-assets eval npm run test-lint-js
# in case you need to check `console.log` messages
docker compose run --rm gather-assets eval npm run test-js-verbose
docker compose run --rm --no-deps gather eval pip list --outdated
docker compose run --rm --no-deps gather-assets eval npm outdated
docker compose run --rm --no-deps gather pip_freeze