/scystream

Open-source data science pipeline tooling

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

The scystream project is an open-source data-science pipeline toolkit containing all necessary tools to create and carry our data-science workflows. With an easy to use frontend, you can schedule and deploy custom workflows containing different data processing tasks.

Architecture

.assets/arch.png

quickstart

Its recommended to use docker and docker-compose

Docker

To setup all services just run the following command in the root directory

docker compose up -d

Development

You can find the development READMEs in the according directories