/rust-lambda-datafusion

Primary LanguageRustMIT LicenseMIT

Introduction

lambda-datafusion is a Rust project that implements an AWS Lambda function in Rust.

The lambda function listens for new Parquet files in an S3 bucket and executes SQL queries on the files using Apache DataFusion. It is a PoC emulating data processing pipelines; where raw data land in the S3 bucker raw-{ENV} and the processed data land in the S3 bucket bronze-{ENV}. Where ENV is the environment, e.g. dev, test, etc. Additionally, some metrics are being sent to CloudWatch.

DO NOT USE ANYWHERE; JUST FOR EDUCATIONAL PURPOSES
❌ Horrible code quality
❌ Barely any error handling
❌ Very likely not be Rust-idiomatic

Prerequisites

  • Rust
  • Cargo Lambda
  • Apache DataFusion
  • Localstack
  • Terraform & terraform-local

Building

  make build

Deploy

  cd terraform
  tflocal apply -auto-approve -var-file=test.variables.tfvars