This solution is modified from an existing AWS Pattern ( https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-an-aws-glue-job-with-an-aws-codepipeline-ci-cd-pipeline.html ). It adds the logic for updating Glue job and approval step in Codepipeline. These changes make it closer to real case.
The following description is copied from above link:
This pattern demonstrates how you can integrate Amazon Web Services (AWS) CodeCommit and AWS CodePipeline with AWS Glue, and use AWS Lambda to launch jobs as soon as a developer pushes their changes to a remote AWS CodeCommit repository.
When a developer submits a change to an extract, transform, and load (ETL) repository and pushes the changes to AWS CodeCommit, a new pipeline is invoked. The pipeline initiates a Lambda function that launches an AWS Glue job with these changes. The AWS Glue job performs the ETL task.
This solution is helpful in the situation where businesses, developers, and data engineers want to launch jobs as soon as changes are committed and pushed to the target repositories. It helps achieve a higher level of automation and reproducibility, therefore avoiding errors during the job launch and lifecycle.
- AWS Account with Admin Right
- git
- Python3 Envionment
To simplify deployment, we can add the following permissions to Lambda execution role:
- AWSCodePipeline_FullAccess
- AmazonS3FullAccess
- AWSGlueConsoleFullAccess
- AWSCodeCommitReadOnly
- CloudWatchLogsFullAccess
- Trust relationships for lambda.amazonaws.com
Note: For production environment, please only allocate the least permissions.
For exmaple, the role name is: CodePipeline_For_Glue_Lambda_Role.
To simplify deployment, we can add the following permissions to Glue job execution role:
- AWSGlueServiceRole
- Trust relationships for glue.amazonaws.com
For exmaple, the role name is: CodePipeline_For_Glue_Execution_Role
For example, the repository name is glue-etl. Use 'Connection steps' guide to set up git credentials.
Create a SNS topic to recieve Approval request. Subscribe your email to this topic.
Create a Lambda Function with the following:
- Name: lambda_pipepline_update_glue
- Role: CodePipeline_For_Glue_Lambda_Role
- Code: Copy the content from lambda_pipepline_update_glue.py to lambda_function.py
- Timeout: 5 min
- Environment variables:
Create a Lambda Function with the following:
- Name: lambda_pipepline_run_glue
- Role: CodePipeline_For_Glue_Lambda_Role
- Code: Copy the content from lambda_pipepline_run_glue.py to lambda_function.py
- Timeout: 5 min
-
Create a pipeline
-
Add source stage
-
Add build stage
- Skip build stage
-
Add deploy stage
-
Edit this pipeline
- Edit Deploy stage
- Delete existing S3 Deploy
- Add action group:
- Add stage after Deploy
- Stage name: Execute
- Add action group
- Add action group:
-
The final piplie is like
-
Clone the empty repository from CodeCommit
git clone https://<endpoint of codecommit>/v1/repos/glue-etl
cd glue-etl
copy the folder glue-etl-01 in sampel to here
The folder name and script name is identical to Glue job name
Modify glue-etl-01.json with your information.
-
Push code to CodeCommit to create a Glue job. The commit message needs to use format of
create <glue job name> xxx
.git checkout -b main
git stage .
git commit -m "create glue-etl-01 initial job"
git push --set-upstream origin main
-
The commit triggers the Codepipeline
-
The pipeline goes to Deploy stage to create Glue job
-
Once Deploy is done, it goes to Execute stage to wait for Approval to continue.
-
You will get an email for notification of approval request. Click Review
-
If it is approved, pipeline continues to run the Glue job.
-
Soon the job is run.
-
Update Glue code, then git commit an update. The commit message needs to use format of
update <glue job name> xxx
.git stage .
git commit -m "update glue-etl-01 add more features"
git push
-
The pipeline starts to Source, Deploy and Execute stages as above.
The following logs can be examinzed for trouleshooting:
- Lambda logs in CloudWatch logs.
- Glue logs in CloudWatch logs.
Next plan is to simplify commit message to remove the need of create/update <glue job name>
. It will retrieve the code change to determine the related glue job.