/terraform-azurerm-aks

Terraform module composition (feature) for Azure Kubernetes Service

Primary LanguageHCLApache License 2.0Apache-2.0

DEPRECATED - Azure Kubernetes Service

Changelog Notice Apache V2 License TF Registry

⚠ This module is deprecated in favor of aks-light module.

This terraform module creates an Azure Kubernetes Service and its associated Azure Application Gateway as ingress controller.

Inside the cluster default node pool, velero and cert-manager are installed.

Inside each node pool, Kured is installed as a daemonset.

This module also configures logging to a Log Analytics Workspace, deploys the Azure Active Directory Pod Identity and creates some Storage Classes with different types of Azure managed disks (Standard HDD retain and delete, Premium SSD retain and delete).

Version compatibility

Module version Terraform version AzureRM version
>= 5.x.x 0.15.x & 1.0.x >= 2.10
>= 4.x.x 0.13.x >= 2.10
>= 3.x.x 0.12.x >= 2.10
>= 2.x.x 0.12.x < 2.0
< 2.x.x 0.11.x < 2.0

Usage

This module is optimized to work with the Claranet terraform-wrapper too which set some terraform variables in the environment needed by this module. More details about variables set by the terraform wrapper available in the documentation.

The helm and kubernetes providers must be defined at the root level and then passed to the module via the provider block as in the examples.

Global versioning rule for Claranet Azure modules

Module version Terraform version AzureRM version
>= 7.x.x 1.3.x >= 3.0
>= 6.x.x 1.x >= 3.0
>= 5.x.x 0.15.x >= 2.0
>= 4.x.x 0.13.x / 0.14.x >= 2.0
>= 3.x.x 0.12.x >= 2.0
>= 2.x.x 0.12.x < 2.0
< 2.x.x 0.11.x < 2.0

Contributing

If you want to contribute to this repository, feel free to use our pre-commit git hook configuration which will help you automatically update and format some files for you by enforcing our Terraform code module best-practices.

More details are available in the CONTRIBUTING.md file.

Usage

This module is optimized to work with the Claranet terraform-wrapper tool which set some terraform variables in the environment needed by this module. More details about variables set by the terraform-wrapper available in the documentation.

locals {
  allowed_cidrs = ["x.x.x.x", "y.y.y.y"]
}

module "azure_region" {
  source  = "claranet/regions/azurerm"
  version = "x.x.x"

  azure_region = var.azure_region
}

module "rg" {
  source  = "claranet/rg/azurerm"
  version = "x.x.x"

  location    = module.azure_region.location
  client_name = var.client_name
  environment = var.environment
  stack       = var.stack
}

module "azure_virtual_network" {
  source  = "claranet/vnet/azurerm"
  version = "x.x.x"

  environment    = var.environment
  location       = module.azure_region.location
  location_short = module.azure_region.location_short
  client_name    = var.client_name
  stack          = var.stack

  resource_group_name = module.rg.resource_group_name

  vnet_cidr = ["10.0.0.0/19"]
}

module "node_network_subnet" {
  source  = "claranet/subnet/azurerm"
  version = "x.x.x"

  environment    = var.environment
  location_short = module.azure_region.location_short
  client_name    = var.client_name
  stack          = var.stack

  resource_group_name  = module.rg.resource_group_name
  virtual_network_name = module.azure_virtual_network.virtual_network_name

  name_suffix = "nodes"

  subnet_cidr_list = ["10.0.0.0/20"]

  service_endpoints = ["Microsoft.Storage"]
}

module "appgw_network_subnet" {
  source  = "claranet/subnet/azurerm"
  version = "x.x.x"

  environment    = var.environment
  location_short = module.azure_region.location_short
  client_name    = var.client_name
  stack          = var.stack

  resource_group_name  = module.rg.resource_group_name
  virtual_network_name = module.azure_virtual_network.virtual_network_name

  name_suffix = "appgw"

  subnet_cidr_list = ["10.0.20.0/24"]
}

module "global_run" {
  source  = "claranet/run/azurerm"
  version = "x.x.x"

  client_name    = var.client_name
  location       = module.azure_region.location
  location_short = module.azure_region.location_short
  environment    = var.environment
  stack          = var.stack

  monitoring_function_splunk_token = var.monitoring_function_splunk_token

  resource_group_name = module.rg.resource_group_name

  tenant_id = var.azure_tenant_id
}

resource "tls_private_key" "key" {
  algorithm = "RSA"
}

module "aks" {
  source  = "claranet/aks/azurerm"
  version = "x.x.x"

  providers = {
    kubernetes = kubernetes.aks-module
    helm       = helm.aks-module
  }

  client_name = var.client_name
  environment = var.environment
  stack       = var.stack

  resource_group_name = module.rg.resource_group_name
  location            = module.azure_region.location
  location_short      = module.azure_region.location_short

  private_cluster_enabled = false
  service_cidr            = "10.0.16.0/22"
  kubernetes_version      = "1.19.7"

  vnet_id         = module.azure_virtual_network.virtual_network_id
  nodes_subnet_id = module.node_network_subnet.subnet_id
  nodes_pools = [
    {
      name            = "pool1"
      count           = 1
      vm_size         = "Standard_D1_v2"
      os_type         = "Linux"
      os_disk_type    = "Ephemeral"
      os_disk_size_gb = 30
      vnet_subnet_id  = module.node_network_subnet.subnet_id
    },
    {
      name                = "bigpool1"
      count               = 3
      vm_size             = "Standard_F8s_v2"
      os_type             = "Linux"
      os_disk_size_gb     = 30
      vnet_subnet_id      = module.node_network_subnet.subnet_id
      enable_auto_scaling = true
      min_count           = 3
      max_count           = 9
    }
  ]

  linux_profile = {
    username = "nodeadmin"
    ssh_key  = tls_private_key.key.public_key_openssh
  }

  oms_log_analytics_workspace_id = module.global_run.log_analytics_workspace_id
  azure_policy_enabled           = false

  logs_destinations_ids = [module.global_run.log_analytics_workspace_id]

  appgw_subnet_id = module.appgw_network_subnet.subnet_id

  appgw_ingress_controller_values = { "verbosityLevel" = 5, "appgw.shared" = true }
  cert_manager_settings           = { "cainjector.nodeSelector.agentpool" = "default", "nodeSelector.agentpool" = "default", "webhook.nodeSelector.agentpool" = "default" }
  velero_storage_settings         = { allowed_cidrs = local.allowed_cidrs }

  container_registries_id = [module.acr.acr_id]

  key_vault_secrets_provider = {
    secret_rotation_enabled  = true
    secret_rotation_interval = "2m"
  }
}

module "acr" {
  source  = "claranet/acr/azurerm"
  version = "x.x.x"

  location            = module.azure_region.location
  location_short      = module.azure_region.location_short
  resource_group_name = module.rg.resource_group_name
  sku                 = "Standard"

  client_name = var.client_name
  environment = var.environment
  stack       = var.stack

  logs_destinations_ids = [module.global_run.log_analytics_workspace_id]
}

Providers

Name Version
azuread ~> 2.31
azurecaf ~> 1.2, >= 1.2.22
azurerm ~> 3.39

Modules

Name Source Version
appgw ./tools/agic n/a
certmanager ./tools/cert-manager n/a
diagnostic_settings claranet/diagnostic-settings/azurerm ~> 6.5.0
infra ./modules/infra n/a
kured ./tools/kured n/a
velero ./tools/velero n/a

Resources

Name Type
azurerm_kubernetes_cluster.aks resource
azurerm_kubernetes_cluster_node_pool.node_pools resource
azurerm_resource_policy_assignment.aks_policy_kubenet_aadpodidentity_assignment resource
azurerm_role_assignment.aad_pod_identity_mio_appgw_identity resource
azurerm_role_assignment.aci_assignment resource
azurerm_role_assignment.aks_acr_pull_allowed resource
azurerm_role_assignment.aks_kubelet_uai_vnet_network_contributor resource
azurerm_role_assignment.aks_uai_private_dns_zone_contributor resource
azurerm_role_assignment.aks_uai_route_table_contributor resource
azurerm_role_assignment.aks_uai_vnet_network_contributor resource
azurerm_role_assignment.aks_user_assigned resource
azurerm_user_assigned_identity.aks_user_assigned_identity resource
azurerm_user_assigned_identity.appgw_assigned_identity resource
azuread_service_principal.aci_identity data source
azurecaf_name.aks data source
azurecaf_name.aks_identity data source
azurecaf_name.appgw data source
azurecaf_name.appgw_identity data source
azurerm_policy_definition.aks_policy_kubenet_aadpodidentity_definition data source
azurerm_subscription.current data source
azurerm_virtual_network.aks_vnet data source

Inputs

Name Description Type Default Required
aadpodidentity_chart_repository AAD Pod Identity Helm chart repository URL string "https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts" no
aadpodidentity_chart_version AAD Pod Identity helm chart version to use string "4.1.9" no
aadpodidentity_custom_name Custom name for aad pod identity MSI string "aad-pod-identity" no
aadpodidentity_extra_tags Extra Tags to add to aad pod identity MSI map(string) {} no
aadpodidentity_kubenet_policy_enabled Boolean to wether deploy or not a built-in Azure Policy at the cluster level
to mitigate potential security issue with aadpodidentity used with kubenet :
https://docs.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity#using-kubenet-network-plugin-with-azure-active-directory-pod-managed-identities "		 bool false no
aadpodidentity_namespace Kubernetes namespace in which to deploy AAD Pod Identity string "system-aadpodid" no
aadpodidentity_values Settings for AAD Pod identity helm Chart:
map(object({
  nmi.nodeSelector.agentpool    = string
  mic.nodeSelector.agentpool    = string
  azureIdentity.enabled         = bool
  azureIdentity.type            = string
  azureIdentity.resourceID      = string
  azureIdentity.clientID        = string
  nmi.micNamespace              = string
}))
 map(string) {} no
aci_subnet_id Optional subnet Id used for ACI virtual-nodes string null no
agic_chart_repository Helm chart repository URL string "https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/" no
agic_chart_version Version of the Helm chart string "1.5.2" no
agic_enabled Enable Application gateway ingress controller bool true no
agic_helm_version [DEPRECATED] Version of Helm chart to deploy string null no
aks_http_proxy_settings AKS HTTP proxy settings. URLs must be in format http(s)://fqdn:port/. When setting the no_proxy_url_list parameter, the AKS Private Endpoint domain name and the AKS VNet CIDR must be added to the URLs list. 
object({
    http_proxy_url    = optional(string)
    https_proxy_url   = optional(string)
    no_proxy_url_list = optional(list(string), [])
    trusted_ca        = optional(string)
  })
 null no
aks_network_plugin AKS network plugin to use. Possible values are azure and kubenet. Changing this forces a new resource to be created string "azure" no
aks_network_policy AKS network policy to use. string "calico" no
aks_pod_cidr CIDR used by pods when network plugin is set to kubenet. https://docs.microsoft.com/en-us/azure/aks/configure-kubenet string "172.17.0.0/16" no
aks_route_table_id Provide an existing route table when outbound_type variable is set to userdefinedrouting with kubenet : https://docs.microsoft.com/fr-fr/azure/aks/configure-kubenet#bring-your-own-subnet-and-route-table-with-kubenet string null no
aks_sku_tier aks sku tier. Possible values are Free ou Paid string "Free" no
aks_user_assigned_identity_custom_name Custom name for the aks user assigned identity resource string null no
aks_user_assigned_identity_resource_group_name Resource Group where to deploy the aks user assigned identity resource. Used when private cluster is enabled and when Azure private dns zone is not managed by aks string null no
aks_user_assigned_identity_tags Tags to add to AKS MSI map(string) {} no
api_server_authorized_ip_ranges Ip ranges allowed to interract with Kubernetes API. Default no restrictions list(string) [] no
appgw_identity_enabled Configure a managed service identity for Application gateway used with AGIC (useful to configure ssl cert into appgw from keyvault) bool false no
appgw_ingress_controller_values Application Gateway Ingress Controller settings map(string) {} no
appgw_private_ip Private IP for Application Gateway. Used when variable private_ingress is set to true. string null no
appgw_settings Application gateway configuration settings. Default dummy configuration map(any) {} no
appgw_ssl_certificates_configs Application gateway ssl certificates configuration list(map(string)) [] no
appgw_subnet_id Application gateway subnet id string "" no
appgw_user_assigned_identity_custom_name Custom name for the application gateway user assigned identity resource string null no
appgw_user_assigned_identity_resource_group_name Resource Group where to deploy the Application Gateway User Assigned Identity resource. string null no
application_gateway_id ID of an existing Application Gateway to use as an AGIC. use_existing_application_gateway must be set to true. string null no
auto_scaler_profile Configuration of auto_scaler_profile block object 
object({
    balance_similar_node_groups      = optional(bool, false)
    expander                         = optional(string, "random")
    max_graceful_termination_sec     = optional(number, 600)
    max_node_provisioning_time       = optional(string, "15m")
    max_unready_nodes                = optional(number, 3)
    max_unready_percentage           = optional(number, 45)
    new_pod_scale_up_delay           = optional(string, "10s")
    scale_down_delay_after_add       = optional(string, "10m")
    scale_down_delay_after_delete    = optional(string, "10s")
    scale_down_delay_after_failure   = optional(string, "3m")
    scan_interval                    = optional(string, "10s")
    scale_down_unneeded              = optional(string, "10m")
    scale_down_unready               = optional(string, "20m")
    scale_down_utilization_threshold = optional(number, 0.5)
    empty_bulk_delete_max            = optional(number, 10)
    skip_nodes_with_local_storage    = optional(bool, true)
    skip_nodes_with_system_pods      = optional(bool, true)
  })
 null no
azure_policy_enabled Should the Azure Policy Add-On be enabled? bool false no
cert_manager_chart_repository Helm chart repository URL string "https://charts.jetstack.io" no
cert_manager_chart_version Cert Manager helm chart version to use string "v1.8.0" no
cert_manager_namespace Kubernetes namespace in which to deploy Cert Manager string "system-cert-manager" no
cert_manager_settings Settings for cert-manager helm chart map(string) {} no
client_name Client name/account used in naming string n/a yes
container_registries_id List of Azure Container Registries ids where AKS needs pull access. list(string) [] no
custom_aks_name Custom AKS name string "" no
custom_appgw_name Custom name for AKS ingress application gateway string "" no
custom_diagnostic_settings_name Custom name of the diagnostics settings, name will be 'default' if not set. string "default" no
default_node_pool Default node pool configuration 
object({
    name                   = optional(string, "default")
    node_count             = optional(number, 1)
    vm_size                = optional(string, "Standard_D2_v3")
    os_type                = optional(string, "Linux")
    workload_runtime       = optional(string, null)
    zones                  = optional(list(number), [1, 2, 3])
    enable_auto_scaling    = optional(bool, false)
    min_count              = optional(number, 1)
    max_count              = optional(number, 10)
    type                   = optional(string, "VirtualMachineScaleSets")
    node_taints            = optional(list(any), null)
    node_labels            = optional(map(any), null)
    orchestrator_version   = optional(string, null)
    priority               = optional(string, null)
    enable_host_encryption = optional(bool, null)
    eviction_policy        = optional(string, null)
    max_pods               = optional(number, 30)
    os_disk_type           = optional(string, "Managed")
    os_disk_size_gb        = optional(number, 128)
    enable_node_public_ip  = optional(bool, false)
    scale_down_mode        = optional(string, "Delete")
  })
 {} no
default_node_pool_tags Specific tags for default node pool map(string) {} no
default_tags_enabled Option to enable or disable default tags bool true no
docker_bridge_cidr IP address for docker with Network CIDR. string "172.16.0.1/16" no
enable_cert_manager Enable cert-manager on AKS cluster bool true no
enable_kured Enable kured daemon on AKS cluster bool true no
enable_pod_security_policy Enable pod security policy or not. https://docs.microsoft.com/fr-fr/azure/AKS/use-pod-security-policies bool false no
enable_velero Enable velero on AKS cluster bool true no
environment Project environment string n/a yes
extra_tags Extra tags to add map(string) {} no
http_application_routing_enabled Whether HTTP Application Routing is enabled. bool false no
key_vault_secrets_provider Enable AKS built-in Key Vault secrets provider. If enabled, an identity is created by the AKS itself and exported from this module. 
object({
    secret_rotation_enabled  = optional(bool)
    secret_rotation_interval = optional(string)
  })
 null no
kubernetes_version Version of Kubernetes to deploy string "1.17.9" no
kured_chart_repository Helm chart repository URL string "https://kubereboot.github.io/charts" no
kured_chart_version Version of the Helm chart string "2.2.0" no
kured_settings Settings for kured helm chart:
map(object({
  image.repository         = string
  image.tag                = string
  image.pullPolicy         = string
  extraArgs.reboot-days    = string
  extraArgs.start-time     = string
  extraArgs.end-time       = string
  extraArgs.time-zone      = string
  rbac.create              = string
  podSecurityPolicy.create = string
  serviceAccount.create    = string
  autolock.enabled         = string
}))
 map(string) {} no
linux_profile Username and ssh key for accessing AKS Linux nodes with ssh. 
object({
    username = string,
    ssh_key  = string
  })
 null no
location Azure region to use string n/a yes
location_short Short name of Azure regions to use string n/a yes
logs_categories Log categories to send to destinations. list(string) null no
logs_destinations_ids List of destination resources IDs for logs diagnostic destination.
Can be Storage Account, Log Analytics Workspace and Event Hub. No more than one of each can be set.
If you want to specify an Azure EventHub to send logs and metrics to, you need to provide a formated string with both the EventHub Namespace authorization send ID and the EventHub name (name of the queue to use in the Namespace) separated by the `		 ` character. list(string) n/a
logs_kube_audit_enabled Whether to include kube-audit and kube-audit-admin logs from diagnostics settings collection. Enabling this can increase your Azure billing. bool false no
logs_metrics_categories Metrics categories to send to destinations. list(string) null no
name_prefix Optional prefix for the generated name string "" no
name_suffix Optional suffix for the generated name string "" no
node_pool_tags Specific tags for node pool map(string) {} no
node_resource_group Name of the resource group in which to put AKS nodes. If null default to MC_ string null no
nodes_pools A list of nodes pools to create, each item supports same properties as local.default_agent_profile list(any) [] no
nodes_subnet_id ID of the subnet used for nodes string n/a yes
oidc_issuer_enabled Whether to enable OpenID Connect issuer or not. https://learn.microsoft.com/en-us/azure/aks/use-oidc-issuer bool false no
oms_log_analytics_workspace_id The ID of the Log Analytics Workspace used to send OMS logs string n/a yes
outbound_type The outbound (egress) routing method which should be used for this Kubernetes Cluster. Possible values are loadBalancer and userDefinedRouting. string "loadBalancer" no
private_cluster_enabled Configure AKS as a Private Cluster: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#private_cluster_enabled bool true no
private_dns_zone_id Id of the private DNS Zone when <private_dns_zone_type> is custom string null no
private_dns_zone_role_assignment_enabled Option to enable or disable Private DNS Zone role assignment. bool true no
private_dns_zone_type Set AKS private dns zone if needed and if private cluster is enabled (privatelink..azmk8s.io)
- "Custom" : You will have to deploy a private Dns Zone on your own and pass the id with <private_dns_zone_id> variable
If this settings is used, aks user assigned identity will be "userassigned" instead of "systemassigned"
and the aks user must have "Private DNS Zone Contributor" role on the private DNS Zone
- "System" : AKS will manage the private zone and create it in the same resource group as the Node Resource Group
- "None" : In case of None you will need to bring your own DNS server and set up resolving, otherwise cluster will have issues after provisioning.

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#private_dns_zone_id		 string "System" no
private_ingress Private ingress boolean variable. When true, the default http listener will listen on private IP instead of the public IP. bool false no
resource_group_name Name of the AKS resource group string n/a yes
service_cidr CIDR used by kubernetes services (kubectl get svc). string n/a yes
stack Project stack name string n/a yes
use_caf_naming Use the Azure CAF naming provider to generate default resource name. custom_aks_name override this if set. Legacy default name is used if this is set to false. bool true no
use_existing_application_gateway True to use an existing Application Gateway instead of creating a new one.
If true you may use appgw_ingress_controller_values = { appgw.shared = true } to tell AGIC to not erase the whole Application Gateway configuration with its own configuration.
You also have to deploy AzureIngressProhibitedTarget CRD.
https://github.com/Azure/application-gateway-kubernetes-ingress/blob/072626cb4e37f7b7a1b0c4578c38d1eadc3e8701/docs/setup/install-existing.md#multi-cluster--shared-app-gateway		 bool false no
velero_chart_repository URL of the Helm chart repository string "https://vmware-tanzu.github.io/helm-charts" no
velero_chart_version Velero helm chart version to use string "2.29.5" no
velero_identity_custom_name Name of the Velero MSI string "velero" no
velero_identity_extra_tags Extra tags to add to velero MSI map(string) {} no
velero_namespace Kubernetes namespace in which to deploy Velero string "system-velero" no
velero_storage_settings Settings for Storage account and blob container for Velero 
object({
    name                     = optional(string)
    resource_group_name      = optional(string)
    location                 = optional(string)
    account_tier             = optional(string)
    account_replication_type = optional(string)
    tags                     = optional(map(any))
    allowed_cidrs            = optional(list(string))
    allowed_subnet_ids       = optional(list(string))
    container_name           = optional(string)
  })
 {} no
velero_values Settings for Velero helm chart:
map(object({
  configuration.backupStorageLocation.bucket                = string
  configuration.backupStorageLocation.config.resourceGroup  = string
  configuration.backupStorageLocation.config.storageAccount = string
  configuration.backupStorageLocation.name                  = string
  configuration.provider                                    = string
  configuration.volumeSnapshotLocation.config.resourceGroup = string
  configuration.volumeSnapshotLocation.name                 = string
  credential.exstingSecret                                  = string
  credentials.useSecret                                     = string
  deployRestic                                              = string
  env.AZURE_CREDENTIALS_FILE                                = string
  metrics.enabled                                           = string
  rbac.create                                               = string
  schedules.daily.schedule                                  = string
  schedules.daily.template.includedNamespaces               = string
  schedules.daily.template.snapshotVolumes                  = string
  schedules.daily.template.ttl                              = string
  serviceAccount.server.create                              = string
  snapshotsEnabled                                          = string
  initContainers[0].name                                    = string
  initContainers[0].image                                   = string
  initContainers[0].volumeMounts[0].mountPath               = string
  initContainers[0].volumeMounts[0].name                    = string
  image.repository                                          = string
  image.tag                                                 = string
  image.pullPolicy                                          = string
}))
 map(string) {} no
vnet_id Vnet id that Aks MSI should be network contributor in a private cluster string null no

Outputs

Name Description
aad_pod_identity_azure_identity Identity object for AAD Pod Identity
aad_pod_identity_namespace Namespace used for AAD Pod Identity
agic_namespace Namespace used for AGIC
aks_id AKS resource id
aks_kube_config Kube configuration of AKS Cluster
aks_kube_config_raw Raw kube config to be used by kubectl command
aks_kubelet_user_managed_identity The Kubelet User Managed Identity used by the AKS cluster.
aks_name Name of the AKS cluster
aks_nodes_pools_ids Ids of AKS nodes pools
aks_nodes_pools_names Names of AKS nodes pools
aks_nodes_rg Name of the resource group in which AKS nodes are deployed
aks_user_managed_identity The User Managed Identity used by the AKS cluster.
application_gateway_id Id of the application gateway used by AKS
application_gateway_identity_principal_id Id of the managed service identity of the application gateway used by AKS
application_gateway_name Name of the application gateway used by AKS
cert_manager_namespace Namespace used for Cert Manager
key_vault_secrets_provider_identity The User Managed Identity used by the Key Vault secrets provider.
kured_namespace Namespace used for Kured
oidc_issuer_url The URL of the OpenID Connect issuer.
public_ip_id Id of the public ip used by AKS application gateway
public_ip_name Name of the public ip used by AKS application gateway
velero_identity Azure Identity used for Velero pods
velero_namespace Namespace used for Velero
velero_storage_account Storage Account on which Velero data is stored.
velero_storage_account_container Container in Storage Account on which Velero data is stored.

Related documentation