/terraform-aws-eks

Terraform module for deploying a Domino on EKS

Primary LanguageHCLApache License 2.0Apache-2.0

terraform-aws-eks

CircleCI

Create SSH Key pair

Prerequisites

Command

 ssh-keygen -q -P '' -t rsa -b 4096 -m PEM -f domino.pem

Create terraform remote state bucket(OPTIONAL)

  • Authenticate with aws, make sure that environment variables: AWS_REGION, AWS_ACCESS_KEY_ID ,AWS_SECRET_ACCESS_KEY are set. If your account has MFA set up you will also need AWS_SESSION_TOKEN.

1. Create Bucket(if you already have a bucket just set the AWS_TERRAFORM_REMOTE_STATE_BUCKET to its name, and skip this step):

export AWS_ACCOUNT="$(aws sts get-caller-identity | jq -r .Account)"
export AWS_TERRAFORM_REMOTE_STATE_BUCKET="domino-terraform-rs-${AWS_ACCOUNT}-${AWS_REGION}"

aws s3api create-bucket \
    --bucket "${AWS_TERRAFORM_REMOTE_STATE_BUCKET}" \
    --region ${AWS_REGION} \
    --create-bucket-configuration LocationConstraint="${AWS_REGION}" | jq .

Verify bucket exists

aws s3api head-bucket --bucket "${AWS_TERRAFORM_REMOTE_STATE_BUCKET}"

You should NOT see an error.

2. Initialize the terraform remote-state

Create a file called terraform.tf(the name does not matter) with the following content

terraform {
  backend "s3" {}
}
### Set the deploy id. This will be used later as well.
export TF_VAR_deploy_id="domino-eks-1"  ## <-- Feel free to rename.
terraform init -migrate-state \
    -backend-config="bucket=${AWS_TERRAFORM_REMOTE_STATE_BUCKET}" \
    -backend-config="key=domino-eks/${TF_VAR_deploy_id}" \
    -backend-config="region=${AWS_REGION}"

If you need to delete the bucket

aws s3 rb s3://"${AWS_TERRAFORM_REMOTE_STATE_BUCKET}" --force

Terraform-docs

Requirements

Name Version
terraform >= 1.3.0
aws >= 4.0
local >= 2.2.0
tls >= 3.4.0

Providers

Name Version
aws >= 4.0
tls >= 3.4.0

Modules

Name Source Version
bastion ./submodules/bastion n/a
eks ./submodules/eks n/a
network ./submodules/network n/a
storage ./submodules/storage n/a

Resources

Name Type
aws_iam_policy.route53 resource
aws_iam_role_policy_attachment.route53 resource
aws_key_pair.domino resource
aws_availability_zones.available data source
aws_ec2_instance_type_offerings.nodes data source
aws_iam_policy_document.route53 data source
aws_route53_zone.hosted data source
aws_subnet.pod data source
aws_subnet.private data source
aws_subnet.public data source
tls_public_key.domino data source

Inputs

Name Description Type Default Required
additional_node_groups Additional EKS managed node groups definition. 
map(object({
    ami                  = optional(string)
    bootstrap_extra_args = optional(string, "")
    instance_types       = list(string)
    spot                 = optional(bool, false)
    min_per_az           = number
    max_per_az           = number
    desired_per_az       = number
    labels               = map(string)
    taints               = optional(list(object({ key = string, value = optional(string), effect = string })), [])
    tags                 = optional(map(string), {})
    volume = object({
      size = string
      type = string
    })
  }))
 {} no
availability_zones List of Availibility zones to distribute the deployment, EKS needs at least 2,https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html.
Note that setting this variable bypasses validation of the status of the zones data 'aws_availability_zones' 'available'.
Caller is responsible for validating status of these zones.		 list(string) [] no
bastion if specifed, a bastion is created with the specified details 
object({
    ami           = optional(string, null) # default will use the latest 'amazon_linux_2' ami
    instance_type = optional(string, "t2.micro")
  })
 null no
cidr The IPv4 CIDR block for the VPC. string "10.0.0.0/16" no
default_node_groups EKS managed node groups definition. 
object(
    {
      compute = object(
        {
          ami                  = optional(string)
          bootstrap_extra_args = optional(string, "")
          instance_types       = optional(list(string), ["m5.2xlarge"])
          spot                 = optional(bool, false)
          min_per_az           = optional(number, 0)
          max_per_az           = optional(number, 10)
          desired_per_az       = optional(number, 1)
          labels = optional(map(string), {
            "dominodatalab.com/node-pool" = "default"
          })
          taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])
          tags   = optional(map(string), {})
          volume = optional(object(
            {
              size = optional(number, 100)
              type = optional(string, "gp3")
            }),
            {
              size = 100
              type = "gp3"
            }
          )
      }),
      platform = object(
        {
          ami                  = optional(string)
          bootstrap_extra_args = optional(string, "")
          instance_types       = optional(list(string), ["m5.4xlarge"])
          spot                 = optional(bool, false)
          min_per_az           = optional(number, 1)
          max_per_az           = optional(number, 10)
          desired_per_az       = optional(number, 1)
          labels = optional(map(string), {
            "dominodatalab.com/node-pool" = "platform"
          })
          taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])
          tags   = optional(map(string), {})
          volume = optional(object(
            {
              size = optional(number, 100)
              type = optional(string, "gp3")
            }),
            {
              size = 100
              type = "gp3"
            }
          )
      }),
      gpu = object(
        {
          ami                  = optional(string)
          bootstrap_extra_args = optional(string, "")
          instance_types       = optional(list(string), ["g4dn.xlarge"])
          spot                 = optional(bool, false)
          min_per_az           = optional(number, 0)
          max_per_az           = optional(number, 10)
          desired_per_az       = optional(number, 0)
          labels = optional(map(string), {
            "dominodatalab.com/node-pool" = "default-gpu"
            "nvidia.com/gpu"              = true
          })
          taints = optional(list(object({ key = string, value = optional(string), effect = string })), [
            { key = "nvidia.com/gpu", value = "true", effect = "NO_SCHEDULE" }
          ])
          tags = optional(map(string), {})
          volume = optional(object(
            {
              size = optional(number, 100)
              type = optional(string, "gp3")
            }),
            {
              size = 100
              type = "gp3"
            }
          )
      })
  })
 
{
  "compute": {},
  "gpu": {},
  "platform": {}
}
 no
deploy_id Domino Deployment ID. string "domino-eks" no
efs_access_point_path Filesystem path for efs. string "/domino" no
eks_master_role_names IAM role names to be added as masters in eks. list(string) [] no
k8s_version EKS cluster k8s version. string "1.24" no
kubeconfig_path fully qualified path name to write the kubeconfig file string "" no
number_of_azs Number of AZ to distribute the deployment, EKS needs at least 2. number 3 no
pod_cidr The IPv4 CIDR block for the VPC. string "100.64.0.0/16" no
pod_cidr_network_bits Number of network bits to allocate to the private subnet. i.e /19 -> 8,192 IPs. number 19 no
pod_subnets Optional list of pod subnet ids list(string) null no
private_cidr_network_bits Number of network bits to allocate to the private subnet. i.e /19 -> 8,192 IPs. number 19 no
private_subnets Optional list of private subnet ids list(string) null no
public_cidr_network_bits Number of network bits to allocate to the public subnet. i.e /27 -> 32 IPs. number 27 no
public_subnets Optional list of public subnet ids list(string) null no
region AWS region for the deployment string n/a yes
route53_hosted_zone_name Optional hosted zone for External DNSone. string "" no
s3_force_destroy_on_deletion Toogle to allow recursive deletion of all objects in the s3 buckets. if 'false' terraform will NOT be able to delete non-empty buckets bool false no
ssh_pvt_key_path SSH private key filepath. string n/a yes
tags Deployment tags. map(string) {} no
update_kubeconfig_extra_args Optional extra args when generating kubeconfig string "" no
use_pod_cidr Use additional pod CIDR range (ie 100.64.0.0/16) for pod/service networking bool true no
vpc_id Optional VPC ID, it will bypass creation of such, public_subnets and private_subnets are also required. string null no

Outputs

Name Description
bastion_ip public ip of the bastion
domino_key_pair Domino key pair
efs_access_point EFS access point
efs_file_system EFS file system
hostname Domino instance URL.
kubeconfig location of kubeconfig
s3_buckets S3 buckets
ssh_bastion_command Command to ssh into the bastion host

Requirements

Name Version
terraform >= 1.3.0
aws >= 4.0
local >= 2.2.0
tls >= 3.4.0

Providers

Name Version
aws 4.33.0
tls 4.0.3

Modules

Name Source Version
bastion ./submodules/bastion n/a
eks ./submodules/eks n/a
k8s_setup ./submodules/k8s n/a
network ./submodules/network n/a
storage ./submodules/storage n/a

Resources

Name Type
aws_iam_policy.route53 resource
aws_iam_role_policy_attachment.route53 resource
aws_key_pair.domino resource
aws_availability_zones.available data source
aws_ec2_instance_type_offerings.nodes data source
aws_iam_policy_document.route53 data source
aws_iam_role.eks_master_roles data source
aws_route53_zone.hosted data source
aws_subnet.private data source
aws_subnet.public data source
tls_public_key.domino data source

Inputs

Name Description Type Default Required
additional_node_groups Additional EKS managed node groups definition. 
map(object({
    ami                  = optional(string)
    bootstrap_extra_args = optional(string, "")
    instance_types       = list(string)
    spot                 = optional(bool, false)
    min_per_az           = number
    max_per_az           = number
    desired_per_az       = number
    labels               = map(string)
    taints               = optional(list(object({ key = string, value = optional(string), effect = string })), [])
    tags                 = optional(map(string), {})
    volume = object({
      size = string
      type = string
    })
  }))
 {} no
availability_zones List of Availibility zones to distribute the deployment, EKS needs at least 2,https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html.
Note that setting this variable bypasses validation of the status of the zones data 'aws_availability_zones' 'available'.
Caller is responsible for validating status of these zones.		 list(string) [] no
bastion if specifed, a bastion is created with the specified details 
object({
    ami           = optional(string, null) # default will use the latest 'amazon_linux_2' ami
    instance_type = optional(string, "t2.micro")
  })
 null no
cidr The IPv4 CIDR block for the VPC. string "10.0.0.0/16" no
default_node_groups EKS managed node groups definition. 
object(
    {
      compute = object(
        {
          ami                  = optional(string)
          bootstrap_extra_args = optional(string, "")
          instance_types       = optional(list(string), ["m5.2xlarge"])
          spot                 = optional(bool, false)
          min_per_az           = optional(number, 0)
          max_per_az           = optional(number, 10)
          desired_per_az       = optional(number, 1)
          labels = optional(map(string), {
            "dominodatalab.com/node-pool" = "default"
          })
          taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])
          tags   = optional(map(string), {})
          volume = optional(object(
            {
              size = optional(number, 100)
              type = optional(string, "gp3")
            }),
            {
              size = 100
              type = "gp3"
            }
          )
      }),
      platform = object(
        {
          ami                  = optional(string)
          bootstrap_extra_args = optional(string, "")
          instance_types       = optional(list(string), ["m5.4xlarge"])
          spot                 = optional(bool, false)
          min_per_az           = optional(number, 1)
          max_per_az           = optional(number, 10)
          desired_per_az       = optional(number, 1)
          labels = optional(map(string), {
            "dominodatalab.com/node-pool" = "platform"
          })
          taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])
          tags   = optional(map(string), {})
          volume = optional(object(
            {
              size = optional(number, 100)
              type = optional(string, "gp3")
            }),
            {
              size = 100
              type = "gp3"
            }
          )
      }),
      gpu = object(
        {
          ami                  = optional(string)
          bootstrap_extra_args = optional(string, "")
          instance_types       = optional(list(string), ["g4dn.xlarge"])
          spot                 = optional(bool, false)
          min_per_az           = optional(number, 0)
          max_per_az           = optional(number, 10)
          desired_per_az       = optional(number, 0)
          labels = optional(map(string), {
            "dominodatalab.com/node-pool" = "default-gpu"
            "nvidia.com/gpu"              = true
          })
          taints = optional(list(object({ key = string, value = optional(string), effect = string })), [
            { key = "nvidia.com/gpu", value = "true", effect = "NO_SCHEDULE" }
          ])
          tags = optional(map(string), {})
          volume = optional(object(
            {
              size = optional(number, 100)
              type = optional(string, "gp3")
            }),
            {
              size = 100
              type = "gp3"
            }
          )
      })
  })
 
{
  "compute": {},
  "gpu": {},
  "platform": {}
}
 no
deploy_id Domino Deployment ID. string "domino-eks" no
efs_access_point_path Filesystem path for efs. string "/domino" no
eks_master_role_names IAM role names to be added as masters in eks. list(string) [] no
k8s_version EKS cluster k8s version. string "1.24" no
kubeconfig_path fully qualified path name to write the kubeconfig file string "" no
number_of_azs Number of AZ to distribute the deployment, EKS needs at least 2. number 3 no
private_cidr_network_bits Number of network bits to allocate to the private subnet. i.e /19 -> 8,192 IPs. number 19 no
private_subnets Optional list of private subnet ids list(string) null no
public_cidr_network_bits Number of network bits to allocate to the public subnet. i.e /27 -> 32 IPs. number 27 no
public_subnets Optional list of public subnet ids list(string) null no
region AWS region for the deployment string n/a yes
route53_hosted_zone_name Optional hosted zone for External DNSone. string "" no
s3_force_destroy_on_deletion Toogle to allow recursive deletion of all objects in the s3 buckets. if 'false' terraform will NOT be able to delete non-empty buckets bool false no
ssh_pvt_key_path SSH private key filepath. string n/a yes
tags Deployment tags. map(string) {} no
update_kubeconfig_extra_args Optional extra args when generating kubeconfig string "" no
vpc_id Optional VPC ID, it will bypass creation of such, public_subnets and private_subnets are also required. string null no

Outputs

Name Description
bastion_ip public ip of the bastion
domino_key_pair Domino key pair
efs_access_point EFS access point
efs_file_system EFS file system
hostname Domino instance URL.
kubeconfig location of kubeconfig
s3_buckets S3 buckets
ssh_bastion_command Command to ssh into the bastion host