tpt

Orchestrate your pipelines using tpt. Deploy them for scheduling, catchup, retries, and live monitoring.

Features

write your pipeline YAML or Rust code and let tpt handle execution order, parallelism, timeouts, and retries
create multiple dynamic tasks from upstream results or control flow using branching tasks
easy testing
- test both YAML and Rust pipelines locally
- rust's compile-time checks ensure code safety and prevent common bugs

Installation
Usage
Documentation
Examples
Deployment
Advanced
License

Installation

cargo install thepipelinetool

Usage

Usage: tpt [pipeline] <COMMAND>

Commands:
  describe  Describe pipeline tasks or edges
  check     Check for circular depencencies
  graph     Displays graph
  tree      Displays tree
  run       Run complete pipeline or function by name
  help      Print this message or the help of the given subcommand(s)

Arguments:
  [pipeline]

Options:
  -h, --help     Print help
  -V, --version  Print version

Examples

Find more examples here

Deployment

The pipeline files must be placed inside PIPELINES_DIR for both the server and workers to access. Visit the template project for the docker-compose.yml example

Advanced

Get started by cloning the template project

git clone https://github.com/thepipelinetool/thepipelinetool_template

Or create a new project and add thepipelinetool_core dependency

mkdir your_pipeline_name
cd your_pipeline_name
cargo init --bin
cargo add thepipelinetool_core

Add the following to src/main.rs

use thepipelinetool_core::{prelude::*, tpt};

#[tpt::main]
fn main() {
    // define your tasks here
}

Find advanced usage here

License

AGPLv3

MGlolenstine/thepipelinetool

tpt