Cumulus Framework

📖 Documentation

Documentation for the latest released version.
Documentation for the unreleased work.

More Information

For more information about this project of more about NASA's Earth Observing System Data and Information System (EOSDIS) and its cloud work, please contact Katie Baynes or visit us at https://earthdata.nasa.gov.

🔨 Development

The Cumulus core repo is a monorepo managed by Lerna. Lerna is responsible for installing the dependencies of the packages and tasks that belong in this repo. In general, Cumulus's npm packages can be found in the packages directory, and workflow tasks can be found in the tasks directory.

To help cut down on the time and disk space required to install the dependencies of the packages in this monorepo, all devDependencies are defined in the top-level package.json. The Node module resolution algorithm allows all of the packages and tasks to find their dev dependencies in that top-level node_modules directory.

TL;DR - If you need to add a devDependency to a package, add it to the top-level package.json file, not the package.json associated with an individual package.

Installation

This is for installation for Cumulus development. See the Cumulus deployment instructions for instructions on deploying the released Cumulus packages.

Prerequisites

NVM and node version 12.18
AWS CLI
BASH
Docker (only required for testing)
docker-compose (only required for testing pip install docker-compose)
Python 3.6+

Install the correct node version:

nvm install
nvm use

Install Lerna

We use Lerna to manage multiple Cumulus packages in the same repo. You need to install lerna as a global module first:

$ npm install -g lerna

Install Local Dependencies

We use npm for local package management

$ npm install
$ npm run bootstrap

Building All packages:

$ npm run build

Build and watch packages:

$ npm run watch

Running the Cumulus APIs locally

Start the API:

$ npm run serve

Or start the distribution API:

$ npm run serve-dist

See the API package documentation for more options.

📝 Tests

Unit Tests

LocalStack

LocalStack provides local versions of most AWS services for testing.

The LocalStack repository has installation instructions.

Localstack is included in the docker-compose file. You only need to run the docker-compose command in the next section in order to use it with your tests.

Docker containers

Turn on the docker containers first:

$ npm run start-unit-test-stack

Stop localstack/unit test services:

$ npm run stop-unit-test-stack

Run database migrations

$ npm run db:local:migrate

Run tests

Run the test commands next

    $ export LOCAL_ES_HOST=127.0.0.1
    $ export LOCALSTACK_HOST=127.0.0.1
    $ npm test

Integration Tests

For more information please read this.

Running tests via VS Code debugger

Copy the .vscode.example directory to .vscode to create your debugger launch configuration. Refer to the VS Code documentation on how to use the debugger.

🔦 Code Coverage and Quality