/aws_lambda_s3_python

This repo meant to be retrieved data from API using lambda funciton then stored data on S3 bucket

Primary LanguageHCLApache License 2.0Apache-2.0

aws_lambda_s3_python

Overview

The code will ingest data from Random API into AWS s3 Bucket on 5 minutes interval. The code was ran by Lambda Function that triggered by CloudWatch Events and can see the log on Cloud Watch monitor

Pre-requisite

Installed AWS CLI

$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
$ unzip awscliv2.zip
$ sudo ./aws/install
$ aws configure
$ pip install boto3

Install Terraform

$ sudo apt-get update && sudo apt-get install -y gnupg software-properties-common

Install boto3

$ pip install boto3

Usage

To run this example you need to execute:

$ terraform init

# without variables
$ terraform plan
$ terraform apply

# with variables
$ terraform plan --var-file="[INPUT_HERE]"
$ terraform apply --var-file="[INPUT_HERE]"
$ terraform output -state="terraform.tfstate"
$ terraform refresh --var-file="[INPUT_HERE]" -lock=false

Assumption

  1. The IAM user that deployed the infrastructure has sufficient permission to apply and make change to the system
  2. The data stored is in json format that will be stored as-is, which will be used for transformation in the downstream process
  3. User has sufficient to create role, iam policy to archieve similar things
  4. This code is not for production purpose, therefore there will be reluctant on certain aspect, such as code format, security concerns, storage
  5. Data store in s3 is unstructure raw data, therefore will be needed further cleaning and transformation to be usable