kiara

A pipeline management and execution engine for the DHARPA project.

kiara user documentation: https://dharpa.org/kiara_documentation
Code: https://github.com/DHARPA-Project/kiara
Development documentation for this repo: https://dharpa.org/kiara

Description

Kiara is the data orchestration engine developed by the DHARPA project. It uses a modular approach to let users re-use tried and tested data orchestration pipelines, as well as create new ones from existing building blocks. It also helps you manage your research data, and augment it with automatically-, semi-automatically-, and manually- created metadata. Most of this is not yet implemented.

Development

Requirements

Python (version >=3.6 -- some make targets only work for Python >=3.7, but kiara itself should work on 3.6)
pip, virtualenv
git
make
direnv (optional)

Prepare development environment

git clone https://github.com/DHARPA-Project/kiara.git
cd kiara
python3 -m venv .venv
source .venv/bin/activate
make init

If you use direnv, you can alternatively do:

git clone https://github.com/DHARPA-Project/kiara.git
cd kiara
cp .envrc.disabled .envrc
direnv allow
make init

Note: you might want to adjust the Python version in .envrc (should not be necessary in most cases though)

`make` targets

init: init development project (install project & dev dependencies into virtualenv, as well as pre-commit git hook)
update-modules: update default kiara modules package from git
flake: run flake8 tests
mypy: run mypy tests
test: run unit tests
docs: create static documentation pages (under build/site)
serve-docs: serve documentation pages (incl. auto-reload) for getting direct feedback when working on documentation
clean: clean build directories

For details (and other, minor targets), check the Makefile.

Running tests

> make test
# or
> make coverage

Copyright & license

This project is MPL v2.0 licensed, for the license text please check the LICENSE file in this repository.

makkus/kiara