apalis

Simple, extensible multithreaded background job and messages processing library for Rust

Features

Simple and predictable job handling model.
Jobs handlers with a macro free API.
Take full advantage of the tower ecosystem of middleware, services, and utilities.
Runtime agnostic - Use tokio, smol etc.
Optional Web interface to help you manage your jobs.

apalis job processing is powered by tower::Service which means you have access to the tower middleware.

apalis has support for:

Source	Crate	Example
Cron Jobs
Redis
Sqlite
Postgres
MySQL
Amqp
From Scratch

Getting Started

To get started, just add to Cargo.toml

[dependencies]
apalis = { version = "0.5", features = ["redis"] } # Backends available: postgres, sqlite, mysql, amqp

Usage

use apalis::prelude::*;
use apalis::redis::RedisStorage;
use serde::{Deserialize, Serialize};
use anyhow::Result;

#[derive(Debug, Deserialize, Serialize)]
struct Email {
    to: String,
}

impl Job for Email {
    const NAME: &'static str = "apalis::Email";
}

/// A function that will be converted into a service.
async fn send_email(job: Email, data: Data<usize>) -> Result<(), Error> {
  /// execute job
  Ok(())
}

#[tokio::main]
async fn main() -> Result<()> {
    std::env::set_var("RUST_LOG", "debug");
    env_logger::init();
    let redis_url = std::env::var("REDIS_URL").expect("Missing env variable REDIS_URL");
    let storage = RedisStorage::new(redis).await?;
    Monitor::new()
        .register_with_count(2, {
            WorkerBuilder::new(format!("email-worker"))
                .data(0usize)
                .with_storage(storage)
                .build_fn(send_email)
        })
        .run()
        .await
}

Then

//This can be in another part of the program or another application eg a http server
async fn produce_route_jobs(storage: &RedisStorage<Email>) -> Result<()> {
    let mut storage = storage.clone();
    storage
        .push(Email {
            to: "test@example.com".to_string(),
        })
        .await?;
}

Feature flags

tracing (enabled by default) — Support Tracing 👀
redis — Include redis storage
postgres — Include Postgres storage
sqlite — Include SQlite storage
mysql — Include MySql storage
cron — Include cron job processing
sentry — Support for Sentry exception and performance monitoring
prometheus — Support Prometheus metrics
retry — Support direct retrying jobs
timeout — Support timeouts on jobs
limit — 💪 Limit the amount of jobs
filter — Support filtering jobs based on a predicate

Storage Comparison

Since we provide a few storage solutions, here is a table comparing them:

Feature	Redis	Sqlite	Postgres	Sled	Mysql	Mongo	Cron
Scheduled jobs	✓	✓	✓	x	✓	x	✓
Retry jobs	✓	✓	✓	x	✓	x	✓
Persistence	✓	✓	✓	x	✓	x	BYO
Rerun Dead jobs	✓	✓	✓	x	✓	x	x

How apalis works

Here is a basic example of how the core parts integrate

sequenceDiagram
    participant App
    participant Worker
    participant Backend

    App->>+Backend: Add job to queue
    Backend-->>+Worker: Job data
    Worker->>+Backend: Update job status to 'Running'
    Worker->>+App: Started job
    loop job execution
        Worker-->>-App: Report job progress
    end
    Worker->>+Backend: Update job status to 'completed'

External examples

Shuttle: Using apalis-cron with shuttle.rs
Actix-Web: Using apalis-redis with actix-web

Projects using apalis

Ryot: A self hosted platform for tracking various facets of your life - media, fitness etc.
Summarizer: Podcast summarizer
Universal Inbox: Universal Inbox is a solution that centralizes all your notifications and tasks in one place to create a unique inbox.

Resources

Background job processing with rust using actix and redis

Web UI

If you are running apalis Board, you can easily manage your jobs. See a working rest API example here

Thanks to

tower - Tower is a library of modular and reusable components for building robust networking clients and servers.
redis-rs - Redis library for rust
sqlx - The Rust SQL Toolkit

Roadmap

v 0.5

v 0.4

v 0.3

v 0.2

Redis Example
Actix Web Example

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

Njuguna Mureithi - Initial work - Njuguna Mureithi

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE.md file for details

sandyz1000/apalis