A collection of awesome things related to DevOps and AWS
-
Follow The Twelve-Factor App principles. See also Applying the Twelve-Factor App Methodology to Serverless Applications
-
Create a release branch and apply hotfixes first to master and cherrypick to release branch. This is a Trunk-Based Development model, which Amazon strongly encourages internally - see “before we begin” in Implementing GitFlow Using AWS CodePipeline, AWS CodeCommit, AWS CodeBuild, and AWS CodeDeploy. Some exceptions are when a fix was already introduced to master branch somewhere, so in this case it’s faster to temporarily open the release branch for direct pull request (the “temporarily” can be enforced through JIRA and a CI system for example)
-
Consider Branch by Abstraction for major changes that take time and continue using Feature Toggles for deciding when to release
-
Test upstream dependencies based on their production at the least. A shift left approach would be to test upstream dependencies based on their staging environment (would incur additional costs for each service to support this)
-
Separate Continuous Delivery and [Continuous] Deployment processes - implement as 2 pipelines. This will enable a decision point as to which regions to deploy instead of all or nothing
- Continuous Delivery pipeline will build the artifacts, run unit/integration tests (to keep development cycle fast with short feedback loop for developers if something fails at this stage), run system tests, capacity tests, performance tests and promote to staging for manual tests (e.g. manual QA and UAT)
- [Continuous] Deployment pipeline will deploy to specified regions or all at once in rolling manner (to reduce blast radius of failed change), depending on the need. A central database is needed to track deployments and keep versions across regions in sync (e.g. do not deploy newer version in region B, while region A has an older one - upgrade region A first). Deployment might be triggered manually or automatically following the Continuous Delivery pipeline
-
ThoughtWorks Technology Radar - Techniques. Specifically, "Four key metrics": The thorough State of DevOps reports have focused on data-driven and statistical analysis of high-performing organizations. The result of this multiyear research, published in Accelerate, demonstrates a direct link between organizational performance and software delivery performance. The researchers have determined that only four key metrics differentiate between low, medium and high performers: lead time, deployment frequency, mean time to restore (MTTR) and change fail percentage. Indeed, we've found that these four key metrics are a simple and yet powerful tool to help leaders and teams focus on measuring and improving what matters. A good place to start is to instrument the build pipelines so you can capture the four key metrics and make the software delivery value stream visible
-
Use multiple accounts for different Software Development Life Cycle (SDLC) stages to reduce blast radius of configuration changes, security breaches, account or regional quotas (a.k.a. limits) and more. See Establishing your best practice AWS environment for more motivation and details
- Towards Operational Excellence blog post series:
- AWS re:Invent 2019: [REPEAT 1] Amazon’s approach to failing successfully (DOP208-R1) (video, slides)
- AWS re:Invent 2018: [REPEAT 1] Releasing Mission-Critical Software at Amazon (DEV209-R1) (video, slides)
- AWS re:Invent 2019: [REPEAT 1] Amazon's approach to high-availability deployment (DOP404-R1) (video, slides)
- Failing successfully: The AWS approach to resilient design
- Fireside Chat: DevOps at Amazon with Ken Exner, GM of AWS Developer Tools - AWS Online Tech Talks
- AWS Toolkit for Visual Studio Code Adds New CDK Explorer in Preview
- Announcing Cloud Debugging (beta) for Debugging Your Applications Running in the Cloud with JetBrains IDEs
- CDK Pipelines: Continuous delivery for AWS CDK applications
- Continuous Delivery for CDK Apps (AWS CDK RFC)
- Building a cross-account continuous delivery pipeline for database migrations
- Automating cross-account actions with an AWS CDK credential plugin
- Serverless CI/CD for the Enterprise on AWS
- Best practices for CI/CD using AWS Fargate and Amazon ECS (video, slides)
- Amazon CI/CD Practices for Software Development Teams (video, slides)
- AWS re:Invent 2015: DevOps at Amazon: A Look at Our Tools and Processes (DVO202)
- Best Practices for CI/CD with AWS Lambda and Amazon API Gateway
- Ensuring rollback safety during deployments
- Building and testing polyglot applications using AWS CodeBuild
- Deploying GitOps with Weave Flux and Amazon EKS
- Include CloudFormation templates in the CDK
- Validating AWS CodeCommit Pull Requests with AWS CodeBuild and AWS Lambda
- Running end-to-end Cypress tests for your fullstack CI/CD deployment with Amplify Console
- Test Reports with AWS CodeBuild
- New – Building a Continuous Integration Workflow with Step Functions and AWS CodeBuild
- Using AWS Step Functions State Machines to Handle Workflow-Driven AWS CodePipeline Actions
- AWS X-Ray (see also Integrating AWS X-Ray with Other AWS Services)
- AWS X-Ray Now Supports Amazon API Gateway and New Sampling Rules API (not a methodology, but an important FYI for observability)
- Visualize and Monitor Highly Distributed Applications with Amazon CloudWatch ServiceLens
- Debugging with Amazon CloudWatch Synthetics and AWS X-Ray
- Amazon CloudWatch Now Includes Contributor Insights - in Preview
- Container monitoring for Amazon ECS, EKS, and Kubernetes is now available in Amazon CloudWatch
- Using Prometheus Metrics in Amazon CloudWatch
- AWS observability workshop
- Building dashboards for operational visibility