tEKS is a set of Terraform / Terragrunt modules designed to get you everything you need to run a production EKS cluster on AWS. It ships with sensible defaults, and add a lot of common addons with their configurations that work out of the box.
eks: bootstrap a managed EKS cluster with a managed or existing VPC. Similar to the official module.eks-addons: provides various addons that are often used on Kubernetes and specifically on EKS.eks-namespaces: allows administrator to manage namespaces and quotas from a centralized configuration with Terraform.
When this projet started, it did not rely on the official Terraform EKS module which is now quite stable and allows advanced configurations. The goal is now to migrate parts of this project to the upstream one and offloading part of the work to official modules and integrating them with Terragrunt:
- AWS VPC
- EKS
eksmodule will be kept for compatibility and all the feature will be PR upstream if they do not already exist.
eks-addons is now decoupled from eks module. It will soon be compatible with upstream module. v3.X is working toward this goal.
master: Backward incompatible with v1.X but compatible with v2.X, releases bumped to v3.X because a lot has changed.release-1.X: Compatible with Terraform < 0.12 and Terragrunt < 0.19. Be sure to target the same modules version.release-2.X: Compatible with Terraform >= 0.12 and Terragrunt >= 0.19. Be sure to target the same modules version.
v1.X is compatible with Terraform < 0.12 and Terragrunt < 0.19. The upgrade path to v2.x is simple:
- update tooling locally
- migrate from
terraform.tfvarsfiles toterragrunt.hclas shown inlivefolder
v2.X and v3.X are not really incompatible per se, there was a lot of changes and a careful migration is needed to avoid breaking running cluster so a change of version was simpler.
- update the
eks-addonsmodule. - run the
eks-addonsmodule with the Kiam configuration if needed. eks-addonsmodule will create the new IAM user and policies.- update the
eksmodule. - run the
eksmodule. ekswill destroy the previous IAM role and policies.
- Node pools with customizable labels / taints
- Fully customizable kubelet args
- Supports new or existing VPC
- Calico for network policies
- Common addons with associated IAM permissions if needed:
- cluster-autoscaler: scale worker nodes based on workload.
- external-dns: sync ingress and service records in route53.
- cert-manager: automatically generate TLS certificates, supports ACME v2.
- kiam: prevents pods to access EC2 metadata and enables pods to assume specific AWS IAM roles.
- nginx-ingress: processes Ingress object and acts as a HTTP/HTTPS proxy (compatible with cert-manager).
- metrics-server: enable metrics API and horizontal pod scaling (HPA).
- prometheus-operator: Monitoring / Alerting / Dashboards.
- virtual-kubelet: enables using ECS Fargate as a provider to run workload without EC2 instances.
- fluentd-cloudwatch: forwards logs to AWS Cloudwatch.
- node-problem-detector: Forwards node problems to Kubernetes events
- flux: Continous Delivery with Gitops workflow.
- sealed-secrets: Technology agnostic, store secrets on git.
- istio: Service mesh for Kubernetes.
- cni-metrics-helper: Provides cloudwatch metrics for VPC CNI plugins.
- kong: API Gateway ingress controller.
- rancher: UI for easy cluster management.
- keycloak : Identity and access management
User guides, feature documentation and examples are available here
Kiam prevents pods from accessing EC2 instances IAM role and therefore using the instances role to perform actions on AWS. It also allows pods to assume specific IAM roles if needed. To do so kiam-agent acts as an iptables proxy on nodes. It intercepts requests made to EC2 metadata and redirect them to a kiam-server that fetches IAM credentials and pass them to pods.
Kiam is running with an IAM user and use a secret key and a access key (AK/SK).
Some addons interface with AWS API, for example:
cluster-autoscalerexternal-dnscert-managervirtual-kubeletcni-metric-helper
