/aws-cicd

Materials for AWS CICD workshop

Primary LanguagePythonApache License 2.0Apache-2.0

aws-cicd

Materials for AWS CICD workshop

Content

Slides

Here are the slides I used in the workshop

  • The CICD journey of SPN Infra., Coretech, Trendmico on AWS
  • Building CI/CD Pipelines for Serverless Applications - SRV302 - re:Invent 2017
  • My thoughts for - Building CI/CD Pipelines for Serverless Applications sharing

Prerequisites

Back to top

  1. An AWS (Amazon Web Service) account W/ administrator permission, if you have none, to get one !

There are few things need your notices and actions in your AWS account

  • All the hands-on activities will be in region us-east-1, codes are not tested in other regions.
  • And you need to make sure you can create bucket and put files on S3... due to you can not use it before account activated successfully.
  • Create a SSH key (e.g. cicd) in us-east-1 region under your AWS account.
  1. The AWS expenses will occur in the hands-ons, it should cost less than USD $1.0 for each hands-on...
  2. A public GitHub account, if you have none, to get one !

There are two more things need your actions in your GitHub account

Hands-on

Back to top

I am trying to leverage as most managed services on AWS as I can, so I picked S3, CodePipline, CodeBuild and CloudFormation as a foundation for our hands-on.

Frankly to say, there are variety of technology combinations doing CI/CD on every environment (In this case, is AWS), so my choices here may not be the most suitable for your needs, it simply plays as a quick start to bring you the overall concepts and a PoC to home. And I hope, it can help you an idea to tailor your own one.

What really concerning about for your CICD infra. ?

According to my experiences, whatever technology stacks you are using for building your CICD infra., I think there will be three aspects you should need to consider about are...

  • What CI server you are using to support your workflow ?

    • In this case is CodePipline + CodeBuild
    • Others may consider to use Jenkins or others CI SaaS services

  • What tools you are using to support your CD (end-to-end from your running services to monitoring) ?

    • In this case is CloudFormation
    • Others may write a bunch of scripts to do their CD

  • How to pass your configuration parameters (Non-secret and secret) to your CD ?

    • In this case are CodePipeline + SSM parameter store
    • Others may use environment variables setting supported by CI server

CI/CD with EC2 on AWS

Back to top

0. Overview

We use AWS CodePipeline to pull commits from GitHub, build, test and deploy a standalone VPC and a very simple API service running within it, in AWS region us-east-1.

The components are

  • A standalone VPC
    • public subnet * 2, private subnet * 2, NAT gateway * 2, S3 VPC endpoint * 1
  • An EC2 instance as bsation node
    • You can login to this VPC via this bastion node
  • Application load balancer
  • Two EC2 instances running a very simple web app behind the ALB
  • CloudWatch Alarm
  • SNS Topic for alarm mail
  • CloudWatch dashboard

1. Launch AWS CodePipeline

Back to top

Click following icon to provision AWS CodePipline via AWS CloudFormation (cf-ec2-cp.yaml) in your AWS account.

  1. Fill up the parameters whom are empty or you want to replace
  2. Click next, and next again
  3. Check all confirmation questions for access IAM resources
  4. Click create ChangeSet button
  5. Click execute and will bring you to CloudFormation console automatically.

You can see the progress of provisioning of CodePipline on AWS CloudFormation console. After provisioning status gets to CREATE_COMPLETED, pls go to CodePipline console to find your first CodePipline project.

Pls remember to Confirm subscription of two mails sent from AWS SNS topics in your mailbox, which is the email address you had written and passed to CloudFormation.

2. AWS CodePipline should automatically trigger a build at first place

Back to top

You can see the CodePipline project starts to trigger a build automatically in few minutes. Due to the CodePipeline will pull repo at first time.

After the service provisioning gets to CREATE_COMPLETED, you can get the endpoint FQDN at the row ALBDNSName at Output tab in AWS CloudFormation console. let's veirfy your build whether successfully !

curl http://<ALBDNSName>
Welcome to my home

The very simple API impl. is api/main.py you can take a look if interested in.

3. Take a look at every component in the overall CodePipeline

Back to top

Take a look on CodePipline console and corresponding docs as follows to clarify our questions.

3.1 What is Stage, action and transition ?

3.2 What types of actions ?

The action types we are using are...

Category Action
Source GitHub
Build AWS CodeBuild
Deploy CloudFormation
Approval Manual approval

You can see more details in cf-ec2-cp.yaml

3.3 Why not use Jenkins, CodeDeploy, or a or b or c ?

Let's say it again, this hands-on simply plays as a quick start to bring you the overall concepts and a PoC to home. And I hope, it can help you an idea to tailor your own one.

But I still can say that, CodePipeline using Jenkins need to launch a long running EC2 instance behind the scene, it is too costly for a PoC. And CodeDeploy can support variety of deployment methods (Lambda, ECS, etc), but in the end, I still need to use CloudFormation to provision monitoring related resources based on end-to-end point of view. And also, CloudFormation is more portable running in diffrent CI servers (Maybe :P).

4. Take a look at our RESTful API service

Back to top

The very simple API impl. is api/main.py you can take a look if interested in.

4.1 healthcheck API

curl http://<ALBDNSName>/healthcheck
Hello World!

This API is used by ALB and auto-scaling group, auto-scaling group will increase/decrease based on the responses of this API from every EC2 instance.

The configuration order will be ALB -> Target group -> Auto-scaling group -> Auto-scaling group configuration -> EC2. You can get start on following doc.

4.2 What will happen if we want to update/deploy our new codes to existing service ?

In this hands-on we are using Blue/Green deployment (create 2 new EC2s -> all healthchek passed -> delete 2 old EC2s), the details in following doc

Take a look on cd/cf-ec2.yaml for more details.

4.3 To use AWS::CloudFormation::Init instead of OpsWorks service

OpsWorks service has a lot of pitfalls and performance issues according my experiences, I recommend to use AWS::CloudFormation::Init instead.

Actually, AWS::CloudFormation::Init is based on cloud-init project.

4.4 sleep API

curl http://<ALBDNSName>/sleep/20
sleep for 20 secs

This API is used to create a response latency (by seconds) of our service, to trigger the backend alarm.

In this case, the alarm will send notification to a SNS topic and then to the topic subscriber, which is, an email address you provided at first place.

The SNS Topic can send to various types of subscriber, for example, to the Slack and PagerDuty.

All these basic stuffs can be created by AWS CloudFormation

But actually, there are still more complicated usecases the CloudFormation can not fulfill them in a out-of-box manner, in these cases, we can consider to use AWS::CloudFormation::CustomResource to deal them.

AWS::CloudFormation::CustomResource is actually a Lambda function extension plugged in CloudFormation service, which can bring a flexible way to provision and configure our resources in CloudFormation

4.5 secret API

curl http://<ALBDNSName>/secret
{"Name": "mysecret03", "Value": "mysecretonaws03"}

This API is trying to demo how we pass the secret data (e.g. database password, etc) in CI/CD. In this case, we encrypted and stored the secret data in SSM parameter store at first place. And try to load and decrypt the secret data in API code.

How to use SSM parameter store

How to encrypt the data with AWS::CloudFormation::CustomResource and AWS::KMS::Key

How to decrypt the data in code

CI/CD with API gateway + Lambda functions on AWS

Back to top

0. Overview

We use AWS CodePipeline to pull commits from GitHub, build, test and deploy an API gateway and multiple Lambda functions behind it, in AWS region us-east-1.

The components are

  • An API gateway
    • Multiple Lambda functions exposing our very simple API services
  • A custom metric generated from a Lambda function logs
  • CloudWatch Alarm
  • SNS Topic for alarm mail
  • CloudWatch dashboard

1. Launch AWS CodePipeline

Back to top

Click following icon to provision AWS CodePipline via AWS CloudFormation (cf-ec2-cp.yaml) in your AWS account.

Pls go to hands-on#1-1. Launch AWS CodePipeline for reference

2. AWS CodePipline should automatically trigger a build at first place

Back to top

You can see the CodePipline project starts to trigger a build automatically in few minutes. Due to the CodePipeline will pull repo at first time.

After the service provisioning gets to CREATE_COMPLETED, you can get the endpoint FQDN at the row APIGatewayURL at Output tab in AWS CloudFormation console. let's veirfy your build whether successfully !

curl <APIGatewayURL>
Welcome to my home

The very simple API impl. is still based on api/main.py, but every API is actually constructed by Lambda functions(lambdas/), you can take a look if interested in.

3. Take a look at every component in the overall CodePipeline

Back to top

You can see more details in cf-sl-cp.yaml

Pls go to hands-on#1-3. Take a look at every component in the overall CodePipeline for reference.

4. Take a look at our RESTful API service

Back to top

The very simple API impl. is still based on api/main.py, but every API is actually constructed by Lambda functions(lambdas/), you can take a look if interested in.

We are using CloudFormation W/ serverless application model(SAM) to deploy our overall resources including API gateway, Lambda functions, and others.

We basically don't need to care about the operations and scaling related issues for underlying resources due to AWS serverless architecture is taking about them for us. But the world is not perfect, there are still some issues we need to care about...for example, cold start issue, etc. Pls refer to slides,p#5 for details

4.1 healthcheck API

curl <APIGatewayURL>/healthcheck
Hello World!

This API originally used by ALB + EC2 architecture, it is useless for serverless architecture.

4.2 What will happen if we want to update/deploy our new codes to existing service ?

We are using SAM to deploy/update our API gateway and Lambda function resources, but there is actually a drawback for SAM model...which is...it replaces our underlying API gateway deployment and/or Lambda functions in place (depends on what resource you changed). It will impact our API availability in a very short of time, so you need to consider this issue whether break your SLA. Pls refer to slides,p#4 for more details.

Take a look on cd/cf-sl.yaml for more details.

4.3 sleep API

Pls go to hands-on#1-4.4 sleep API for reference.

I also enhanced a new testcase for this API, which is.

curl <APIGatewayURL>/sleep/hello
{"message": "Internal server error"}

This error will trigger an alarm and also will shown in dashboard. I created a custom metric from CloudWatch Log Group storing the log messages generated from Sleep Lambda function, and then create an alarm and dashboard widget based on this custom metric. Pls refer to cd/cf-sl.yaml#L245 for more details.

4.4 secret API

Pls go to hands-on#1-4.5 secret API for reference.

Delete hands-on resources

Back to top

  1. Approve ApproveForDeletion action in CodePipeline

Approve this action will keep to delete CloudFormation stacks

  • *-ec2 and *-ec2-vpc for hands-on#1
  • *-sl for hands-on#2
  1. Delete S3 bucket for CodePipeline artifacts

Go to S3 console and delete bucket: *-cicd-test

  1. Delete CodePipeline CloudFormation stack

Go to CloudFormation console and delete CodePipeline stack (The stack name was given by you at first place).

Tips

Back to top

Sync your forked git repo

git remote add upstream https://github.com/takeshimiao/aws-cicd.git
git fetch upstream
git checkout master
git merge upstream/master
# you may need to fix the merge conflicts if any
git push origin master

Reference: https://help.github.com/articles/syncing-a-fork/