apache-beam
There are 283 repositories under apache-beam topic.
tensorflow/tfx
TFX is an end-to-end platform for deploying production ML pipelines
GoogleCloudPlatform/DataflowTemplates
Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
nielsbasjes/yauaa
Yet Another UserAgent Analyzer
GoogleCloudPlatform/flink-on-k8s-operator
[DEPRECATED] Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.
blockchain-etl/bitcoin-etl
ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ
google/weather-tools
Tools to make weather data accessible and useful.
spotify/flink-on-k8s-operator
Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.
google/fhir-data-pipes
A collection of tools for extracting FHIR resources and analytics services on top of that data.
google/tensorflow-recorder
TFRecorder makes it easy to create TensorFlow records (TFRecords) from Pandas DataFrames and CSVs files containing images or structured data.
ngrunwald/datasplash
Clojure API for a more dynamic Google Dataflow
mohaseeb/beam-nuggets
Collection of transforms for the Apache beam python SDK.
blockchain-etl/blockchain-etl-streaming
Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes
mercari/pipeline
Tool to define Apache Beam pipeline in YAML or JSON
tosun-si/asgarde
Asgarde allows simplifying error handling with Apache Beam Java, with less code, more concise and expressive code.
Fematich/mlengine-boilerplate
Repository to quickly get you started with new Machine Learning projects on Google Cloud Platform. More info(slides):
yu-iskw/bigquery-to-datastore
Export a whole BigQuery table to Google Datastore with Apache Beam/Google Dataflow
luisbelloch/data_processing_course
Some class materials for a data processing course using PySpark
xmlking/micro-apps
Microservices in Post-Kubernetes Era. A polyglot monorepo
blockchain-etl/blockchain-etl-architecture
Blockchain ETL Architecture
doitintl/banias
Opinionated serverless event analytics pipeline
tosun-si/pasgarde
Asgarde allows simplifying error handling with Apache Beam Python, with less code, more concise and expressive code.
asaharland/beam-pipeline-examples
Apache Beam examples for running on Google Cloud Dataflow.
janaom/gcp-data-engineering-etl-with-composer-dataflow
This project leverages GCS, Composer, Dataflow, BigQuery, and Looker on Google Cloud Platform (GCP) to build a robust data engineering solution for processing, storing, and reporting daily transaction data in the online food delivery industry.
NucleusEngineering/hack-your-pipe
Efficient streaming data ingestion, transformation & activation
sayakpaul/count-tokens-hf-datasets
This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.
google-parfait/dataset_grouper
Libraries for efficient and scalable group-structured dataset pipelines.
mozilla-services/foxsec-pipeline
Log analysis pipeline utilizing Apache Beam
google/consent-based-conversion-adjustments
Code to statistically up-weight conversion values of consenting customers to feed up to 100% of the factual conversion values back into Google Ads.
mercari/DataflowTemplates
Convenient Dataflow pipelines for transforming data between cloud data sources
Ganeshsivakumar/langchain-beam
Integrates LLMs as PTransform in Apache Beam pipelines using LangChain
cassiobolba/Data-Engineering
Projects and studies regarding Data Engineering Area
datastacktv/apache-beam-explained
Source code for the YouTube video, Apache Beam Explained in 12 Minutes
delftdata/stateflow
Prototype which extracts stateful dataflows by analysing Python code.
esakik/beam-mysql-connector
An Apache Beam I/O connector for seamless integration with MySQL database 🔗 https://beam.apache.org/documentation/io/connectors/#other-io-connectors-for-apache-beam
O2-Czech-Republic/proxima-platform
The Proxima platform.