Customer Data Platform
- segment
- hubspot
- aws glue
- aws lambda
- A configured aws cli
- Java 8 / Maven
- Node 14
cd glue-etl
mvn clean install
# Manually copy target/cdp-x.x-shaded.jar => s3://aws-glue-jars-301027959319-us-east-1
aws s3 cp ./target/cdp-1.0-SNAPSHOT-shaded.jar s3://aws-glue-jars-301027959319-us-east-1/
# Manually upload scala source code file to s3://aws-glue-scripts-301027959319-us-east-1
aws s3 cp ./src/main/scala/com/circle/data/jobs/InvestorActionProcessor.scala s3://aws-glue-assets-301027959319-us-east-1/scripts/
# deploy the lambda to import investor basic information
cd lamba/basic
npm install
./release-dev.sh
- Read s3://aws-glue-segment-dev-301027959319-us-east-1/** (SegmentLambdaExecute-DEV)
- Write s3://aws-glue-segment-dev-301027959319-us-east-1
- s3://aws-glue-segment-dev-301027959319-us-east-1
- s3://aws-glue-jars-301027959319-us-east-1
- s3://aws-glue-scripts-301027959319-us-east-1
- Founders: Glue job runs 10pm nightly
- Companies: Glue job runs at 11pm nightly
- cleanup IAM
- dev vs prod setup
- ci/cd
- audit/data integ