compascafe/mage-ai

🧙 Mage is an open-source data pipeline tool for transforming and integrating data.

🧙 Mage

Mage is an open-source data pipeline tool for transforming and integrating data.

Here is a sample data pipeline defined across 3 files:

# data_loaders/load_data_from_file.py
@data_loader
def load_csv_from_file():
    return pd.read_csv('default_repo/titanic.csv')

# transformers/select_columns.py
@transformer
def select_columns_from_df(df, *args):
    return df[['Age', 'Fare', 'Survived']]

# data_exporters/export_to_file.py
@data_exporter
def export_titanic_data_to_disk(df) -> None:
    df.to_csv('default_repo/titanic_transformed.csv')

What the data pipeline looks like in the UI:

New? We recommend reading about blocks and learning from a hands-on tutorial.

Table of contents

1. Quick start
2. Demo
3. Tutorials
4. Features
5. Documentation
6. Core design principles
7. Core abstractions

🏃‍♀️ Quick start

Install Mage:

Using Docker

Create a new project and launch tool (change demo_project to any other name if you want):

docker run -it -p 6789:6789 -v $(pwd):/home/src \
  mageai/mageai mage start demo_project

_{Want to use Spark or other integrations? Read more about integrations.}

Using pip or conda

1. Install Mage

pip install mage-ai

or

conda install -c conda-forge mage-ai

_{For additional packages (e.g. spark, postgres, etc), please see Installing extra packages.}

_{If you run into errors, please see Install errors.}

2. Create new project and launch tool (change demo_project to any other name if you want):

mage start demo_project

Open tool in browser

Open http://localhost:6789 in your browser and build a pipeline.

🎮 Demo

Live demo

Try a hosted version of the tool here: http://demo.mage.ai.

WARNING

The live demo is public, please don’t save anything sensitive.

Demo video (2 min)

_{Click the image to play video}

👩‍🏫 Tutorials

• Train model on Titanic dataset
• Load data from API, transform it, and export it to PostgreSQL
• Integrate Mage into an existing Airflow project

🔮 Features

• Data pipeline management
• Notebook for building data pipelines

_{Read more here.}

📚 Documentation

Read more here.

🏔️ Core design principles

Every user experience and technical design decision adheres to these principles.

1. Easy developer experience
2. Engineering best practices built-in
3. Data is a first-class citizen
4. Scaling made simple

_{Read more here.}

🛸 Core abstractions

These are the fundamental concepts that Mage uses to operate.

• Project
• Pipeline
• Block
• Data product
• Trigger
• Run

_{Read more here.}

🙋‍♀️ Contributing

Check out the 🎁 contributing guide to get started by setting up your development environment and exploring the code base.

🤔 Frequently Asked Questions (FAQs)

Check out our FAQ page to find answers to some of our most asked questions.

🧙 Community

We love the community of Magers (/ˈmājər/); a group of mages who help each other realize their full potential!

To live chat with the Mage team and community:

For real-time news, fun memes, data engineering topics, and more, join us on:

• Twitter
• LinkedIn
• GitHub
• Slack

🪪 License

See the LICENSE file for licensing information.