Amazon EKS SSP for Terraform

Welcome to the Amazon EKS Shared Services Platform (SSP) for Terraform.

This repository contains the source code for a Terraform framework that aims to accelerate the delivery of a batteries-included, multi-tenant container platform on top of Amazon EKS. This framework can be used by AWS customers, partners, and internal AWS teams to implement the foundational structure of a SSP according to AWS best practices and recommendations.

This project leverages the community terraform-aws-eks modules for deploying EKS Clusters.

Getting Started

The easiest way to get started with this framework is to follow our Getting Started guide.

Documentation

For complete project documentation, please visit our documentation directory.

Patterns

To view examples for how you can leverage this framework, see the deploy directory.

Usage Example

The below demonstrates how you can leverage this framework to deploy an EKS cluster, a managed node group, and various Kubernetes add-ons.

module "eks-ssp" {
    source = "git@github.com:aws-samples/aws-eks-accelerator-for-terraform.git"

    # EKS CLUSTER
    kubernetes_version       = "1.21"
    vpc_id             = "<vpcid>"     # Enter VPC ID
    private_subnet_ids = ["<subnet-a>", "<subnet-b>", "<subnet-c>"]     # Enter Private Subnet IDs

    # EKS MANAGED ADD-ON VARIABLES
    enable_vpc_cni_addon     = true
    enable_coredns_addon     = true
    enable_kube_proxy_addon  = true

    # KUBERNETES ADD-ON VARIABLES
    cluster_autoscaler_enable           = true
    metrics_server_enable               = true
    aws_lb_ingress_controller_enable    = true
    aws_for_fluent_bit_enable           = true
    cert_manager_enable                 = true
    nginx_ingress_controller_enable     = true

    # EKS MANAGED NODE GROUPS
    enable_managed_nodegroups = true    # default false
    managed_node_groups = {
        mg_m4l = {
            node_group_name = "managed-ondemand"
            instance_types  = ["m4.large"]
            subnet_ids      = ["<subnet-a>", "<subnet-b>", "<subnet-c>"]
        }
    }
}

The code above will provision the following:

✅ A new EKS Cluster with a managed node group.
✅ EKS managed add-ons vpc-cni, CoreDNS, and kube-proxy.
✅ Cluster Autoscaler and Metrics Server for scaling your workloads.
✅ Fluent Bit for routing metrics.
✅ AWS Load Balancer Controller for distributing traffic.
✅ cert-manager for managing SSL/TLS certificates.
✅ Nginx for managing ingress.

Add-ons

This framework provides out of the box support for a wide range of popular Kubernetes add-ons. By default, the Terraform Helm provider is used to deploy add-ons with publicly available Helm Charts. The framework provides support however for leveraging self-hosted Helm Chart as well.

For complete documentation on deploying add-ons, please visit our add-on documentation

Submodules

The root module calls into several submodules which provides support for deploying and integrating a number of external AWS services that can be used in concert with EKS. This included Amazon Managed Prometheus and EMR with EKS etc.

For complete documentation on deploying external services, please visit our submodules documentation.

Motivation

The Amazon EKS SSP for Terraform allows customers to easily configure and deploy a multi-tenant, enterprise-ready container platform on top of EKS. With a large number of design choices, deploying production-grade container platform can take a significant about of time, involve integrating a wide range or AWS services and open source tools, and require deep understand of AWS and Kubernetes concepts.

This solution handles integrating EKS with popular open source and partner tools, in addition to AWS services, in order to allow customers to deploy a cohesive container platform that can be offered as a service to application teams. It provides out-of-the-box support for common operational tasks such as auto-scaling workloads, collecting logs and metrics from both clusters and running applications, managing ingress and egress, configuring network policy, managing secrets, deploying workloads via GitOps, and more. Customers can leverage the solution to deploy a container platform and start onboarding workloads in days, rather than months.

Feedback

For architectural details, step-by-step instructions, and customization options, see our official documentation site.

To post feedback, submit feature ideas, or report bugs, use the Issues section of this GitHub repo.

To submit code for this Quick Start, see the AWS Quick Start Contributor's Kit.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Requirements

Name	Version
terraform	>= 1.0.0
aws	~> 3.60.0
helm	~> 2.3.0
http	2.4.1
kubernetes	~> 2.5.0
local	2.1.0
null	3.1.0

Providers

Name	Version
aws	~> 3.60.0
http	2.4.1
kubernetes	~> 2.5.0

Modules

Name	Source	Version
agones	./kubernetes-addons/agones	n/a
argocd	./kubernetes-addons/argocd	n/a
aws-for-fluent-bit	./kubernetes-addons/aws-for-fluentbit	n/a
aws_eks	terraform-aws-modules/eks/aws	v17.20.0
aws_eks_addon	./modules/aws-eks-addon	n/a
aws_eks_fargate_profiles	./modules/aws-eks-fargate-profiles	n/a
aws_eks_managed_node_groups	./modules/aws-eks-managed-node-groups	n/a
aws_eks_self_managed_node_groups	./modules/aws-eks-self-managed-node-groups	n/a
aws_managed_prometheus	./modules/aws-managed-prometheus	n/a
aws_opentelemetry_collector	./kubernetes-addons/aws-opentelemetry-eks	n/a
cert_manager	./kubernetes-addons/cert-manager	n/a
cluster_autoscaler	./kubernetes-addons/cluster-autoscaler	n/a
eks_tags	./modules/aws-resource-tags	n/a
emr_on_eks	./modules/emr-on-eks	n/a
fargate_fluentbit	./kubernetes-addons/fargate-fluentbit	n/a
lb_ingress_controller	./kubernetes-addons/lb-ingress-controller	n/a
metrics_server	./kubernetes-addons/metrics-server	n/a
nginx_ingress	./kubernetes-addons/nginx-ingress	n/a
prometheus	./kubernetes-addons/prometheus	n/a
spark-k8s-operator	./kubernetes-addons/spark-k8s-operator	n/a
traefik_ingress	./kubernetes-addons/traefik-ingress	n/a
windows_vpc_controllers	./kubernetes-addons/windows-vpc-controllers	n/a

Resources

Name	Type
aws_kms_key.eks	resource
kubernetes_config_map.aws_auth	resource
aws_availability_zones.available	data source
aws_caller_identity.current	data source
aws_eks_cluster.cluster	data source
aws_eks_cluster_auth.cluster	data source
aws_partition.current	data source
aws_region.current	data source
http_http.eks_cluster_readiness	data source

Inputs

Name	Description	Type	Default	Required
agones_enable	Enabling Agones Gaming Helm Chart	`bool`	`false`	no
agones_helm_chart	Agones GameServer Helm chart config	`any`	`{}`	no
argocd_enable	Enable ARGO CD Kubernetes Addon	`bool`	`false`	no
argocd_helm_chart	ARGO CD Kubernetes Addon Configuration	`any`	`{}`	no
aws_auth_additional_labels	Additional kubernetes labels applied on aws-auth ConfigMap	`map(string)`	`{}`	no
aws_for_fluentbit_enable	Enabling FluentBit Addon on EKS Worker Nodes	`bool`	`false`	no
aws_for_fluentbit_helm_chart	Helm chart definition for aws_for_fluent_bit	`any`	`{}`	no
aws_lb_ingress_controller_enable	enabling LB Ingress Controller on eks cluster	`bool`	`false`	no
aws_lb_ingress_controller_helm_app	Helm chart definition for aws_lb_ingress_controller	`any`	`{}`	no
aws_managed_prometheus_enable	Enable AWS Managed Prometheus service	`bool`	`false`	no
aws_managed_prometheus_workspace_name	AWS Managed Prometheus WorkSpace Name	`string`	`"aws-managed-prometheus-workspace"`	no
aws_open_telemetry_addon	AWS Open Telemetry Distro Addon Configuration	`any`	`{}`	no
aws_open_telemetry_enable	Enable AWS Open Telemetry Distro Addon	`bool`	`false`	no
cert_manager_enable	Enabling Cert Manager Helm Chart installation. It is automatically enabled if Windows support is enabled.	`bool`	`false`	no
cert_manager_helm_chart	Cert Manager Helm chart configuration	`any`	`{}`	no
cluster_autoscaler_enable	Enabling Cluster autoscaler on eks cluster	`bool`	`false`	no
cluster_autoscaler_helm_chart	Cluster Autoscaler Helm Chart Config	`any`	`{}`	no
cluster_enabled_log_types	A list of the desired control plane logging to enable	`list(string)`	[ "api", "audit", "authenticator", "controllerManager", "scheduler" ]	no
cluster_endpoint_private_access	Indicates whether or not the Amazon EKS private API server endpoint is enabled. Default to AWS EKS resource and it is false	`bool`	`false`	no
cluster_endpoint_public_access	Indicates whether or not the Amazon EKS public API server endpoint is enabled. Default to AWS EKS resource and it is true	`bool`	`true`	no
cluster_log_retention_period	Number of days to retain cluster logs	`number`	`7`	no
coredns_addon_version	CoreDNS Addon version	`string`	`"v1.8.3-eksbuild.1"`	no
create_eks	Enable Create EKS	`bool`	`false`	no
emr_on_eks_teams	EMR on EKS Teams configuration	`any`	`{}`	no
enable_coredns_addon	Enable CoreDNS Addon	`bool`	`false`	no
enable_emr_on_eks	Enabling EMR on EKS Config	`bool`	`false`	no
enable_fargate	Enable Fargate profiles	`bool`	`false`	no
enable_irsa	Indicates whether or not the Amazon EKS public API server endpoint is enabled. Default to AWS EKS resource and it is true	`bool`	`true`	no
enable_kube_proxy_addon	Enable Kube Proxy Addon	`bool`	`false`	no
enable_managed_nodegroups	Enable self-managed worker groups	`bool`	`false`	no
enable_self_managed_nodegroups	Enable self-managed worker groups	`bool`	`false`	no
enable_vpc_cni_addon	Enable VPC CNI Addon	`bool`	`false`	no
enable_windows_support	Enable Windows support	`bool`	`false`	no
environment	Environment area, e.g. prod or preprod	`string`	`"preprod"`	no
fargate_fluentbit_config	Fargate fluentbit configuration	`any`	`{}`	no
fargate_fluentbit_enable	Enabling fargate_fluent_bit module on eks cluster	`bool`	`false`	no
fargate_profiles	Fargate Profile configuration	`any`	`{}`	no
kube_proxy_addon_version	KubeProxy Addon version	`string`	`"v1.20.4-eksbuild.2"`	no
kubernetes_version	Desired Kubernetes master version. If you do not specify a value, the latest available version is used	`string`	`"1.21"`	no
managed_node_groups	Managed Node groups configuration	`any`	`{}`	no
map_accounts	Additional AWS account numbers to add to the aws-auth configmap.	`list(string)`	`[]`	no
map_roles	Additional IAM roles to add to the aws-auth configmap.	list(object({ rolearn = string username = string groups = list(string) }))	`[]`	no
map_users	Additional IAM users to add to the aws-auth configmap.	list(object({ userarn = string username = string groups = list(string) }))	`[]`	no
metrics_server_enable	Enabling metrics server on eks cluster	`bool`	`false`	no
metrics_server_helm_chart	Metrics Server Helm Addon Config	`any`	`{}`	no
nginx_helm_chart	NGINX Ingress Controller Helm Chart Configuration	`any`	`{}`	no
nginx_ingress_controller_enable	Enabling NGINX Ingress Controller on EKS Cluster	`bool`	`false`	no
org	tenant, which could be your organization name, e.g. aws'	`string`	`""`	no
private_subnet_ids	list of private subnets Id's for the Worker nodes	`list(string)`	n/a	yes
prometheus_enable	Enable Community Prometheus Helm Addon	`bool`	`false`	no
prometheus_helm_chart	Community Prometheus Helm Addon Config	`any`	`{}`	no
public_subnet_ids	list of private subnets Id's for the Worker nodes	`list(string)`	`[]`	no
self_managed_node_groups	Self-Managed Node groups configuration	`any`	`{}`	no
spark_on_k8s_operator_enable	Enabling Spark on K8s Operator on EKS Cluster	`bool`	`false`	no
spark_on_k8s_operator_helm_chart	Spark on K8s Operator Helm Chart Configuration	`any`	`{}`	no
tags	Additional tags (e.g. `map('BusinessUnit`,`XYZ`)	`map(string)`	`{}`	no
tenant	Account Name or unique account unique id e.g., apps or management or aws007	`string`	`"aws"`	no
terraform_version	Terraform Version	`string`	`"Terraform"`	no
traefik_helm_chart	Traefik Helm Addon Config	`any`	`{}`	no
traefik_ingress_controller_enable	Enabling Traefik Ingress Controller on eks cluster	`bool`	`false`	no
vpc_cni_addon_version	VPC CNI Addon version	`string`	`"v1.8.0-eksbuild.1"`	no
vpc_id	VPC id	`string`	n/a	yes
windows_vpc_controllers_helm_chart	Windows VPC Controllers Helm chart configuration	`any`	`{}`	no
zone	zone, e.g. dev or qa or load or ops etc...	`string`	`"dev"`	no

Outputs

Name	Description
amp_work_arn	AWS Managed Prometheus workspace ARN
amp_work_id	AWS Managed Prometheus workspace id
cluster_name	Kubernetes Cluster Name
cluster_oidc_url	The URL on the EKS cluster OIDC Issuer
cluster_primary_security_group_id	EKS Cluster Security group ID
cluster_security_group_id	EKS Control Plane Security Group ID
configure_kubectl	Configure kubectl: make sure you're logged in with the correct AWS profile and run the following command to update your kubeconfig
emr_on_eks_role_arn	IAM execution role ARN for EMR on EKS
emr_on_eks_role_id	IAM execution role ID for EMR on EKS
fargate_profiles	Outputs from EKS Fargate profiles groups
fargate_profiles_aws_auth_config_map	Fargate profiles AWS auth map
fargate_profiles_iam_role_arns	IAM role arn's for Fargate Profiles
managed_node_group_aws_auth_config_map	Managed node groups AWS auth map
managed_node_group_iam_role_arns	IAM role arn's of managed node groups
managed_node_groups	Outputs from EKS Managed node groups
oidc_provider_arn	The ARN of the OIDC Provider if `enable_irsa = true`.
self_managed_node_group_aws_auth_config_map	Self managed node groups AWS auth map
self_managed_node_group_iam_role_arns	IAM role arn's of self managed node groups
self_managed_node_groups	Outputs from EKS Self-managed node groups
windows_node_group_aws_auth_config_map	Windows node groups AWS auth map
worker_security_group_id	EKS Worker Security group ID created by EKS module

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

jessesanford/aws-eks-accelerator-for-terraform