airflow-docker

There are 151 repositories under airflow-docker topic.

  • coder2j/airflow-docker

    Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j)

    Language:Python31622255

  • Wittline/uber-expenses-tracking

    The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.

    Language:Jupyter Notebook1226336

  • data-burst/airflow-git-sync

    Sync DAG changes from Git to Airflow

    6411310

  • anastasiia-p/airflow-ml

    Airflow Pipeline for Machine Learning

    Language:Python38108

  • sergio11/lyric_wave_architecture

    🎵 LyricWave – AI Music Composer (Proof of Concept) 🎶 A personal project exploring automatic generation of unique MP4 songs. LyricWave blends lyrics with AI-generated melodies and synthetic vocals to experiment with new forms of musical expression. A creative testbed to push your ideas into sound. 🚀🎧

    Language:Python35306

  • anilkulkarni87/airflow-docker

    This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.

    Language:Python343311
  • orangutan-stem

    mikestack15/orangutan-stem

    An open-source project dedicated to constructing robust data pipelines and scalable software infrastructure. We leverage industry-standard tools favored by developers to enhance efficiency and reliability. Uniquely, these pipelines are field-tested on farms across Sumatra, Indonesia, ensuring real-world applicability and resilience.

    Language:Python34403

  • DatacollectorVN/Airflow-Tutorial

    My self-learning about Apache Airflow

    Language:Python331015
  • docker-airflow

    abhioncbr/docker-airflow

    Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/

    Language:Python323165

  • mikeroyal/Apache-Airflow-Guide

    Apache Airflow Guide

    Language:Python283014

  • judeleonard/Prescriber-ETL-data-pipeline

    An End-to-End ETL data pipeline that leverages pyspark parallel processing to process about 25 million rows of data coming from a SaaS application using Apache Airflow as an orchestration tool and various data warehouse technologies and finally using Apache Superset to connect to DWH for generating BI dashboards for weekly reports

    Language:Python25103

  • rahul765/Machine-Learning-Pipelines

    From data gathering to model deployment. Complete ML pipeline using Docker, Airflow and Python.

    Language:Python13104

  • jksinghpro/docker-airflow

    Docker for airflow with mysql as backend

    Language:Shell12019

  • okzapradhana/etl-flatfile-airflow

    Building Data Warehouse on BigQuery which takes flat file as the data sources with Airflow as the Orchestrator

    Language:Python12208

  • resdevd/docker-compose-airflow

    Quick Docker Compose deployment for Apache Airflow version 2.x. Ideal for local development

    Language:Shell12317

  • ericdaat/data-stack

    A starting point for a data stack using Python, Apache Airflow and Metabase.

    Language:Python10205

  • knowsuchagency/docker-compose-airflow

    An airflow deployment configuration with sane defaults

    Language:Python10202

  • vjgpt/twitter-pipeline

    In this project, you will be building a Twitter Scheduler using Apache Airflow on Docker.

    Language:Python10107

  • pacuna/airflow-docker

    Run Apache Airflow using Docker containers

    Language:Python8306

  • AhmetFurkanDEMIR/airflow-spark-kafka-example

    Airflow, Spark and Kafka example

    Language:Dockerfile620

  • Aiscalate/aiscalator

    Tools to streamline Jupyter Notebook Prototypes into robust Data Products

    Language:Python5202

  • Booss3my/Indeed_scraper

    Automated Indeed Job Offer Scraper: Airflow Orchestrated and Scheduled, Data Loaded into PostgreSQL Database

    Language:Python5110

  • Data-Bishop/Team5-BuildItAll-Data-Platform

    This repository contains the codebase for the BuildItAll Big Data Processing Platform, a case study project designed to manage large daily data for a hypothetical Belgian client.

    Language:HCL5

  • prasadanilmore/Flask-Airflow-ETL

    This documentation provides an overview of the tasks completed in this project. The project comprises three key tasks: ETL (Extract, Transform, Load), API development, and Data Orchestration using Airflow. Each task is detailed below along with explanations of design choices and considerations.

    Language:Python5101

  • rushabh2390/SharesETL

    Shares ETL to scrape historical data of one shares and import in metabase

    Language:Python5200

  • DunnBC22/Data_Engineering_Projects

    This repository includes data engineering projects using Apache Airflow. I hope to add more projects using different technologies soon!

    Language:TSQL4110

  • gustschaefer/Twitter-Batch-ETL

    ETL que extraí diariamente os trending topics do Twitter em diversos países, realiza transformações com PySpark e envia os dados para o Amazon S3 utilizando Apache Airflow como orquestrador.

    Language:Python4100

  • saianupkumarp/airflow-docker

    Airflow docker boiler plate with LocalExecutor, CeleryExecutor. #TODO Kubernetes

    Language:Python4102

  • sreekesh93/airflow_beginners

    to start airflow in local with basic setup

    Language:Python4120

  • abhishekbhakat/Turbine

    Simple python script for easy local airflow deployment with docker. Packed with additional components. Will be adding more going forward.

    Language:Python3110

  • Chinaskidev/ETL-Clima-ElSalvador

    MLOps, haciendo un ETL sencillo usando Docker y Airflow y Google Cloud

    Language:Python3101

  • erik1110/invoice

    iThome 15th-ironman (2023) - Creating an Invoice System with Python (Check Your Own Invoices)

    Language:Python3101

  • kaushil24/llm-finetune-pipeline

    Airflow pipeline to finetune LLM on Kubernetes

    Language:Python3101

  • mpavanetti/airflow

    This set of code and instructions has the porpouse to instanciate a compiled environment with set of docker images like airflow webserver, airflow scheduler, postgresql, pyspark, Data Pipeline consuming data from weather api , processing with pyspark and storing in postgresql

    Language:PHP3101

  • Pirate-Emperor/BigData-Pipeline

    BigData Pipeline is a local testing environment for experimenting with various storage solutions (RDB, HDFS), query engines (Trino), schedulers (Airflow), and ETL/ELT tools (DBT). It supports MySQL, Hadoop, Hive, Kudu, and more.

    Language:Dockerfile3100

  • tarmkalavan/selenium-airflow

    Using Airflow and Selenium to automate PM2.5 forecasting, from feature scraping (metereological data from earth.nullschool.net) to reporting.

    Language:Python3101