scaleway/dagster-scaleway

Dagster Scaleway (WIP)

This repository contains a Dagster integration for Scaleway.

It allows you to run Dagster pipelines on Scaleway Serverless Jobs.

Basic usage

Build a docker image containing your Dagster code and push it to the Scaleway Registry (or any other registry of your choice):

dagster project scaffold --name my-dagster-project
cd my-dagster-project

cat <<EOF > Dockerfile
FROM python:3.12-slim-bookworm
WORKDIR /app
COPY . .
RUN pip install pendulum==2.0.3
RUN pip install .
# Install the Dagster Scaleway module. You can also specify it in your "setup.py" file
RUN pip install dagster_scaleway
EOF

Build and push the image:

docker build -t rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest .
docker push rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest

Then, configure the dagster.yaml file to use this image:

run_launcher:
  module: dagster_scaleway
  class: ScalewayServerlessJobRunLauncher
  config:
    docker_image: rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest

Run Dagster locally:

pip install -e ".[dev]" "dagster-scaleway"
dagster dev

Your Dagster ops will be run as Scaleway Serverless Jobs! 🎉

See the Dagster documentation for more information on how to get started with Dagster.

Examples

See the examples folder for examples of how to use this integration.