Delivery Hero System Engineering

This repository contains Terraform modules, Kubernetes configuration and other best practice examples to give a head start when building a new environment on AWS or Kubernetes.

The intended audience is teams or projects that are starting with AWS or Kubernetes and want to begin with a good design.

Pull requests welcome!

Guides

terraform-kubernetes-getting-started: A guide to getting started with Kubernetes on AWS using Terraform. Includes AWS and Kubernetes configuration and running an example application.

Terraform modules

aws_sso: Creates an Identity Provider in IAM and some default roles to be used for SSO.
autoscaling_group_elb_app: An example of EC2 autoscaling group and ELB.
kubernetes_kops_cluster: A Kubernetes cluster with multi-AZ master based on kops.
lambda_kubernetes_deployer: A lambda function that deploys to a Kubernetes cluster when a container image is pushed to an ECR repository.
lambda_splunk_forwarder: A lambda function for forwarding Cloudwatch logs to a Splunk HTTP events collector.
nat_gateway: Creates multi-AZ NAT gateways, associated private subnets and route tables.
ssh_bastion: SSH bastion host in an ASG with a fixed EIP.
security_monkey: IAM role and associated policy to run Netflix's Security Monkey.
vpc: A VPC setup that includes public and private subnets for each AZ, route tables.

Examples

base: A good starting point with a VPC and related resources, SSH bastion, some default security groups and S3 bucket for Terraform state.
kubernetes: Same as base example but with shared kubernetes resources and 2 kubernetes clusters added.

Kubernetes Examples

Deployment, Service and Autoscaling: A simple example of a Kubernetes deployment, different service types and a horizontal pod autoscaler.
Helm templating: A simple example using Helm to template Kubernetes resources.
Ingress Controller: Using an Ingress controller and resource to split traffic across multiple applications.
ExternalDNS: A tool that automatically creates DNS records for Kubernetes resources.
Kubernetes Best Practices: A great presentation from a Google engineer.
Kubernetes Autoscaling: How to set up autoscaling for both deployment and cluster nodes.

Other useful tools

Keymaker

Lightweight SSH key management on AWS EC2. Add public SSH keys to IAM users and then they can log into EC2 hosts.

https://github.com/kislyuk/keymaker

Elasticsearch on Kubernetes

https://github.com/pires/kubernetes-elasticsearch-cluster

Invokust

Run Locust load tests on AWS Lambda.

https://github.com/FutureSharks/invokust

Helm

Helm is a powerful tool for creating templates for Kubernetes resources, creating reproducible builds or for packaging and installing predefined configurations for services.

Dos and Don'ts

Do make pull requests to this repository.
Don't bother using a NAT gateway unless you specifically need a fixed source IP address for outgoing traffic.
Do enable MFA for your IAM accounts.
Do store your Terraform state in a bucket.
Do use an internal Route53 zone to hold records for RDS endpoints, ES endpoints, Elasticache endpoints etc.
Do terminate SSL on ELBs and forward as HTTP in VPC. This means you never need to deal with SSL or certificates on instances.
Do use a SSH bastion for all SSH connections and restrict SSH access by IP ranges.
Consider registering an external domain in Route53. It only costs a few dollars and you can have a free SSL certificate. Then use this domain and certificate for all external ELBs.
Don't have instances that are not part of an Austoscaling Group.
Do write Terraform modules to reduce duplicated code.

jmound/dhh-system-engineering