/aggregate

ODK Aggregate is a Java server that stores, analyzes, and presents survey data collected using ODK Collect. Contribute and make the world a better place! ✨🗄✨

Primary LanguageJavaOtherNOASSERTION

⚠️ ODK Aggregate is no longer being updated. Please use ODK Central instead. ⚠️

Overview

Platform License Build status Slack

ODK Aggregate provides a ready-to-deploy server and database to:

  • provide blank forms to ODK Collect (or other OpenRosa clients)
  • accept submissions (finalized forms) from ODK Collect and manage collected data
  • visualize the collected data using maps and simple graphs
  • export submissions in CSV, KML, and JSON format
  • publish submissions to external systems like Google Spreadsheets

ODK Aggregate can be deployed on an Apache Tomcat server, or any servlet 2.5-compatible (or higher) web container, backed with a PostgreSQL or a MySQL database server.

Getting the code

  1. Fork the Aggregate project (why and how to fork)

  2. Install Git LFS

  3. Clone your fork of the project locally. At the command line:

    git clone https://github.com/YOUR-GITHUB-USERNAME/aggregate

Setting up the database

Aggregate supports a variety of database engines, but we strongly recommend PostgreSQL. If you wish to use MySQL, see the database configurations guide.

PostgreSQL with Docker

  1. Install Docker and Docker Compose

  2. Start the development server with ./gradlew postgresqlComposeUp

    Check that the port number 5432 is not used by any other service in your computer. You can change this editing the ports section of the db/postgresql/docker-compose.yml configuration file. Be sure to check the documentation: Compose file version 3 reference - Ports section.

  3. Stop the server with ./gradlew postgresqlComposeDown

For more information see here for Docker and [here for Docker Compose]((docs/build-and-run-with-docker-compose.md).

Local PostgreSQL server

  1. Download and install PostgreSQL 9 or later

    • If you are a macOS user, we recommend Postgres.app
    • If you are a Windows user, we recommend BigSQL

  2. In a command.line terminal, run the following commands to set up a database for Aggregate:

    (Linux and macOS)

    sudo su - postgres -c "psql -c \"CREATE ROLE aggregate WITH LOGIN PASSWORD 'aggregate'\""
sudo su - postgres -c "psql -c \"CREATE DATABASE aggregate WITH OWNER aggregate\""
sudo su - postgres -c "psql -c \"GRANT ALL PRIVILEGES ON DATABASE aggregate TO aggregate\""
sudo su - postgres -c "psql -c \"CREATE SCHEMA aggregate\" aggregate"
sudo su - postgres -c "psql -c \"ALTER SCHEMA aggregate OWNER TO aggregate\" aggregate"
sudo su - postgres -c "psql -c \"GRANT ALL PRIVILEGES ON SCHEMA aggregate TO aggregate\" aggregate"

    (Windows)

    psql.exe -c "CREATE ROLE aggregate WITH LOGIN PASSWORD 'aggregate'"
psql.exe -c "CREATE DATABASE aggregate WITH OWNER aggregate"
psql.exe -c "GRANT ALL PRIVILEGES ON DATABASE aggregate TO aggregate"
psql.exe -c "CREATE SCHEMA aggregate" aggregate
psql.exe -c "ALTER SCHEMA aggregate OWNER TO aggregate" aggregate
psql.exe -c "GRANT ALL PRIVILEGES ON SCHEMA aggregate TO aggregate" aggregate

Building and Running the project

  • Copy the jdbc.properties.example, odk-settings.xml.example, and security.properties.example files at /src/main/resources to the same location, removing the .example extension.

    If you have followed the database configuration steps above, you don't need to make any change in these files. Otherwise, head to the Aggregate configuration guide and make the required changes for your environment.

    If you are running the project in Docker, see here for the next steps.

  • Start a local development Aggregate server with ./gradlew appRunWar

    Gradle will compile the project and start the server, which can take some time.

    Eventually, you will see a "Press any key to stop the server" message. At this point, you can browse http://localhost:8080 to use Aggregate.

  • Stop the server pressing any key in the terminal where you started the server

If you have more than one Java version installed in your computer, you can ensure that Java 8 will be used when running Gradle tasks from the command-line by adding -Porg.gradle.java.home={PATH_TO_JAVA8_HOME} to the task.

Connect from an external device

By default, Gretty will launch a server using a localhost address which will not be accessible by external devices (e.g., ODK Collect in an emulator, ODK Briefcase on another computer). To set a non-localhost address, edit the following files:

  • In src/main/resources/security.properties, change security.server.hostname to the address
  • In build.gradle, inside the gretty block, change host to the same address

Setting up your development environment

These instructions are for IntelliJ IDEA Community edition, which is the (free) Java IDE we use for all the ODK toolsuite, but you don't really need any specific IDE to work with this codebase. Any Java IDE will support any of the steps we will be describing.

Import

  • On the welcome screen, click Import Project, navigate to your aggregate folder, and select the build.gradle file.

    Make sure you check Use auto-import option in the Import Project from Gradle dialog.

    Ignore any message about any detected GWT, Spring or web facets.

  • Make sure you set Java 8 as the project's selected SDK

Run

  1. Show the Gradle tool window by selecting the menu option at View > Tool Windows > Gradle

    You will see a new panel on the right side with all the Gradle task groups

  2. Double click the appRunWar Gradle task under the gretty task group

    A new Run bottom panel will pop up.

    Gradle will compile the project and start the server, which can take some time.

    Eventually, you will see a "Press any key to stop the server" message. At this point, you can browse http://localhost:8080 to use Aggregate.

You can stop the server by pressing any key in the Run panel.

Debug

  1. In the Run menu, select Edit Configurations...

  2. Press the + button to add a Remote configuration

    • Name: Debug Aggregate (or whatever you'd like)
    • Host: localhost
    • Port: 5005
    • Search sources using module's classpath: aggregate

  3. Press OK

  4. Run Aggregate with the appRunWarDebug task (double click it on the Gradle panel at the right side)

  5. Run the Debug Aggregate run configuration you've created (use the debug button, not the play button, which should be disabled)

Eventually, the compilation will finish and the server will be ready for you to browse http://localhost:8080

To stop the debugging session, press any key in the Run bottom panel. This will close your debug process in the Debug bottom panel as well.

Extended topics

There is a /docs directory in the repo with more documentation files that expand on certain topics:

Contributing

Any and all contributions to the project are welcome. ODK Aggregate is used across the world primarily by organizations with a social purpose so you can have real impact!

If you're ready to contribute code, see the contribution guide.

The best way to help us test is to build from source! We are currently focusing on stabilizing the build process.

Troubleshooting

  • We enabled Git LFS on the Aggregate codebase and reduced the repo size from 700 MB to 34 MB. No code was changed, but if you cloned before December 11th, 2017, you'll need to reclone the project.

  • If you get an Invalid Gradle JDK configuration found error importing the code, you might not have set the JAVA_HOME environment variable. Try these solutions.

  • If you are having problems with hung Tomcat/Jetty processes, try running the appStop Gradle task to stop running all instances.

  • If you're using Chrome and are seeing blank pages or refreshing not working, connect to Aggregate with the dev tools window open. Then in the Network tab, check Disable cache.