cloud-composer

sungchun12/airflow-toolkit

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) :desktop_computer: >> [ :rocket:, :ship: ]

doitintl/doit-composer-airflow-training

Getting started with Apache Airflow on Cloud Composer

janaom/gcp-data-engineering-etl-with-composer-dataflow

This project leverages GCS, Composer, Dataflow, BigQuery, and Looker on Google Cloud Platform (GCP) to build a robust data engineering solution for processing, storing, and reporting daily transaction data in the online food delivery industry.

GoogleCloudPlatform/cloud-composer-mssql-dataflow-bigquery

This repository contains an example of how to leverage Cloud Composer and Cloud Dataflow to move data from a Microsoft SQL Server to BigQuery. The diagrams below demonstrate the workflow pipeline.

google/spindle-dv360

QA dashboard for DV360 advertisers

TristanBilot/airflow-rbac-roles-cli

A tool to create Airflow RBAC roles with dag-level permissions from cli.

Niangmohamed/GCP-Professional-Data-Engineer-Learning-Path

I got Google Cloud Certified. I have what it takes to leverage Google Cloud technology. Here my certification: https://www.credential.net/ee1bd2d6-fdb0-4037-8a8d-9afae3d79c86.

cristian-rincon/action-composer-sync

This is a simple action that helps you to fetch your Apache Airflow DAGs to Google Cloud Composer

tosun-si/teams-league-airflow-elt

The goal of this article is showing a real world use case for ELT batch pipeline, with Cloud Storage, BigQuery, Apache Airflow and Cloud Composer : The Extract part is managed in Cloud Storage The Load part is managed from Cloud Storage to BigQuery The Transform part is managed by a BigQuery SQL query Everything is orchestrated by Airflow

zaimy/trinity

A tool to synchronize workflows (DAGs) between Codebase, Cloud Storage and Airflow metadata.

jasmeetsb/airflow_ml_ops

Sample Airflow ML Pipelines

kentarokamiyajp/dbt-bigquery-cloud-composer-tutorial

DBT execution from Cloud Composer. BigQuery is used for main DWH and Compute Engine is built for dbt execution.

wintermi/cloud-composer-trigger

Examples on how to Trigger a Google Cloud Composer DAG

Franklin-Muhuni/Toronto-Shelter-ETL-Pipeline

In this project, I built an end-to-end data pipeline that processes and analyzes daily occupancy and capacity data from Toronto's shelter and overnight service programs.

Giray18/Data-Engineering-on-GCP

A repo containing auto-triggered Airflow ETL activities for datasets located on GCP storage that flattens and creates analytical views on Big Query

supakunz/Book-Revenue-Pipeline-GCP

Deploying ETL pipeline on cloud, analyzing on BigQuery, dashboarding on Looker Studio

supakunz/Uber-Pipeline-GCP

Deploying ETL pipeline on cloud, analyzing on BigQuery, dashboarding on PowerBI

hilsdsg3/Google_Cloud_Platform_Composer

To provide an introduction and demos in Google Cloud Platform Airflow Composer

mchestnut91/Google-Cloud-Composer-Practice

yuya-hanzawa/bq-cloud-composer

Practice GCP Cloud composer

addi-kamal/twitter-etl-airflow

Airflow Pipline to extract data from twitter API then transform it and finally load it to google cloud storage bucket.

mahesh-c-pathak/gcs_pipeline

Building a fully automated data Pipeline with Google Cloud Services

Niangmohamed/Cloud-Composer-Advanced-Lab

Cloud Composer: Copying BigQuery tables across different locations.

ricardolsmendes/ci-cd-for-data-processing-workflow

This repository contains source code based on a guide on how to use Cloud Build and Cloud Composer to create a CI/CD pipeline for building, testing, and deploying a data processing workflow

sofiasawczenko/ETL_employee_info_pipeline

This repository contains code and configuration files for an Extract, Transform, Load (ETL) project using Google Cloud Data Fusion for data extraction, Apache Airflow/Composer for orchestration, and Google BigQuery for data loading.