/behavior-personalization-pipeline

Build batch processing pipelines to do behavior analytics.

Primary LanguagePowerShell

Behavior Personalization Pipeline with AWS

Description

Build batch processing pipelines to do behavior analytics with AWS.

Technology

Apache Airflow, AWS (S3, EMR, IAM, Redsfhit), Docker, and PySpark

Prerequisets

  1. Docker
  2. PostgreSQL
  3. AWS Account
  4. AWS CLI Configured

Usage

This code run in Windows 10 (Powershell 5.1) while the reference [1] run in linux/macOS (bash shell).

To setup the infrastructure needed, run infra_setup.ps1 with bucket_name as an argument.

Example : infra_setup.ps1 name-of-bucket

Make python environment using venv and install requirement packages

    # Make
    python -m venv bp-env
    
    # Activate
    .\bp-env\Scripts\Activate.ps1
    
    # Install
    pip install -r requirements.txt

To destroy infrastructure, run infra_teardown.ps1 with bucket_name as an argument.

Example : infra_teardown.ps1 name-of-bucket

References

[1] Data Engineering Project for Beginners - Batch edition

[2] Getting started with Amazon Redshift Spectrum

[3] Changing PowerShell's default output encoding to UTF-8

[4] authorize-security-group-ingress

[5] S3DistCp (s3-dist-cp)

[6] StepConfig