| title | titleSuffix | description | services | ms.topic | ms.date |
|---|---|---|---|---|---|
Secure secrets with Key Vault |
Azure Kubernetes Service |
Learn how to get Secrets from Azure Key Vault in Azure Kubernetes Service (AKS) using AAD Pod Identity and Secrets Store CSI provider |
container-service |
tutorial |
05/03/2020 |
A Secret in Kubernetes is meant to save sensitive and secure data like passwords, certificates, and keys. These Secret objects aren't secret or secure! They're encoded using Base64 and saved into etcd. Also, by default, any pod have access to them.
To secure these secrets, one solution could be to encrypt data at rest in Kubernetes as explained in Kubernetes documentation. But this feature (KMS) is not yet supported in AKS. However some work is in progress. Check the AKS Roadmap and the KMS Plugin for Key Vault.
Another solution would be using Azure Key Vault. It can securely encrypt sensitive data like keys, secrets, files, and certificates. Even more than just encryption, they offer powerful features like key rotation, expiration date and access policies.
Now, to access Key Vault, a password is needed. Thus it will be non-securely saved in a Secret object in etcd! We returned back to the first problem. Fortunately in Azure, AKS can connect to Key Vault or SQL Database… using an Identity. This connection could be done using the open-source project Pod Identity. It uses Azure AD to create an Identity and assign the roles and resources.
Now that we have access to Key Vault, we can use its SDK or REST API in the application to retrieve the secrets. The SDK has support for .NET, Java, Python, JS, Ruby, PHP, etc. Or we can retrieve the secrets from a mounted volume. Historically, in Azure, this solution was implemented through Kubernetes Key Vault Flex Volume. Now it's being deprecated. The new solution is Azure Key Vault provider for Secret Store CSI driver. Which is the Azure implementation of Secrets Store CSI driver.
This tutorial will help you to securely retrieve secrets in Key Vault right from the Pod using Secrets Store CSI and AAD Pod Identity.
To complete this workshop, we need az, kubectl and helm CLI. Also, we need to create the following resources in Azure: • AKS cluster that uses Service Principal or Managed Identity through the variable $isAKSWithManagedIdentity. • Key Vault with the following secrets: “DatabaseLogin: DbUserName” and “DatabasePassword: MyP@ssword123456”. • Container Registry (ACR). We provision these resources from the Azure Portal or using the following Powershell script:
$suffix = "demo01"
$subscriptionId = (az account show | ConvertFrom-Json).id
$tenantId = (az account show | ConvertFrom-Json).tenantId
$location = "westeurope"
$resourceGroupName = "rg-" + $suffix
$aksName = "aks-" + $suffix
$aksVersion = "1.16.13"
$keyVaultName = "keyvaultaks" + $suffix
$secret1Name = "DatabaseLogin"
$secret2Name = "DatabasePassword"
$secret1Alias = "DATABASE_LOGIN"
$secret2Alias = "DATABASE_PASSWORD"
$identityName = "identity-aks-kv"
$identitySelector = "azure-kv"
$secretProviderClassName = "secret-provider-kv"
$acrName = "acrforaks" + $suffix
$isAKSWithManagedIdentity = "true"
# echo "Creating Resource Group..."
$resourceGroup = az group create -n $resourceGroupName -l $location | ConvertFrom-Json
# echo "Creating ACR..."
$acr = az acr create --resource-group $resourceGroupName --name $acrName --sku Basic | ConvertFrom-Json
az acr login -n $acrName --expose-token
If ($isAKSWithManagedIdentity -eq "true") {
echo "Creating AKS cluster with Managed Identity..."
$aks = az aks create -n $aksName -g $resourceGroupName --kubernetes-version $aksVersion --node-count 1 --attach-acr $acrName --enable-managed-identity | ConvertFrom-Json
} Else {
echo "Creating AKS cluster with Service Principal..."
$aks = az aks create -n $aksName -g $resourceGroupName --kubernetes-version $aksVersion --node-count 1 --attach-acr $acrName | ConvertFrom-Json
}
# retrieve the existing or created AKS
$aks = (az aks show -n $aksName -g $resourceGroupName | ConvertFrom-Json)
# echo "Connecting/authenticating to AKS..."
az aks get-credentials -n $aksName -g $resourceGroupName
echo "Creating Key Vault..."
$keyVault = az keyvault create -n $keyVaultName -g $resourceGroupName -l $location --enable-soft-delete true --retention-days 7 | ConvertFrom-Json
# $keyVault = az keyvault show -n $keyVaultName | ConvertFrom-Json # retrieve existing KV
echo "Creating Secrets in Key Vault..."
az keyvault secret set --name $secret1Name --value "DbUserName" --vault-name $keyVaultName
az keyvault secret set --name $secret2Name --value "P@ssword123456" --vault-name $keyVaultName
Important
Key Vault, AKS and Identity are in the same resource group here for simplicity. But they can be deployed on different ones.
We’ll start by installing Secrets Store CSI driver using Helm charts into a separate namespace.
helm repo add csi-secrets-store-provider-azure https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts
"secrets-store-csi-driver-provider-azure" has been added to your repositories
kubectl create ns csi-driver
namespace/csi-driver created
helm install csi-azure csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --namespace csi-driver
NAME: csi-azure
LAST DEPLOYED: Thu Jun 11 11:57:28 2020
NAMESPACE: csi-driver
STATUS: deployed
REVISION: 1
TEST SUITE: None
Let's check the new created pods:
kubectl get pods --namespace=csi-driver
NAME READY STATUS RESTARTS AGE
csi-azure-csi-secrets-store-provider-azure-9mf84 0/1 ContainerCreating 0 3s
csi-azure-secrets-store-csi-driver-rpn7f 0/3 ContainerCreating 0 3s
Now the driver is installed. Let's use the SecretProviderClass to configure the Key Vault instance to connect to the specific keys, secrets, or certificates to retrieve. Note how we are providing the Key Vault name, resource group, subscription Id, tenant Id, and then the name of the secrets.
$secretProviderKV = @"
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
name: $($secretProviderClassName)
spec:
provider: azure
parameters:
usePodIdentity: "true"
useVMManagedIdentity: "false"
userAssignedIdentityID: ""
keyvaultName: $keyVaultName
cloudName: AzurePublicCloud
objects: |
array:
- |
objectName: $secret1Name
objectAlias: $secret1Alias
objectType: secret
objectVersion: ""
- |
objectName: $secret2Name
objectAlias: $secret2Alias
objectType: secret
objectVersion: ""
resourceGroup: $resourceGroupName
subscriptionId: $subscriptionId
tenantId: $tenantId
"@
$secretProviderKV | kubectl create -f -
secretproviderclass.secrets-store.csi.x-k8s.io/secret-provider-kv created
Note
Note: This sample used only Secrets in Key Vault, but we can also retrieve Keys and Certificates.
If we are using AKS with Managed Identity, then we should create the following two role assignments:
# Run the following 2 commands only if using AKS with Managed Identity
If ($isAKSWithManagedIdentity -eq "true") {
az role assignment create --role "Managed Identity Operator" --assignee $aks.identityProfile.kubeletidentity.clientId --scope /subscriptions/$subscriptionId/resourcegroups/$($aks.nodeResourceGroup)
az role assignment create --role "Virtual Machine Contributor" --assignee $aks.identityProfile.kubeletidentity.clientId --scope /subscriptions/$subscriptionId/resourcegroups/$($aks.nodeResourceGroup)
# If user-assigned identities that are not within the cluster resource group
# az role assignment create --role "Managed Identity Operator" --assignee $aks.identityProfile.kubeletidentity.clientId --scope /subscriptions/$subscriptionId/resourcegroups/$resourceGroupName
}
The Azure Key Vault Provider offers four modes for accessing a Key Vault instance: Service Principal, Pod Identity, VMSS User Assigned Managed Identity and VMSS System Assigned Managed Identity. Here we'll be using Pod Identity. Azure AD Pod Identity will be used to create an Identity in AAD and assign the right roles and resources. Let's first install it into the cluster.
We deploy Pod Identity using a Helm chart. The chart will install Node Managed Identity (NMI) and Managed Identity Controllers (MIC) on each node.
helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
helm install pod-identity aad-pod-identity/aad-pod-identity
kubectl get pods
NAME READY STATUS RESTARTS AGE
aad-pod-identity-mic-c558d8649-4vmj2 1/1 Running 0 29s
aad-pod-identity-mic-c558d8649-rrhvd 1/1 Running 0 29s
aad-pod-identity-nmi-t62zh 1/1 Running 0 29s
If we are using an AKS cluster with Managed Identity, then Azure has already created the Identity resource with AKS. So, we’ll go to retrieve it. But if we created an AKS cluster with Service Principal, then we need to create a new Identity.
# If using AKS with Managed Identity, retrieve the existing Identity
If ($isAKSWithManagedIdentity -eq "true") {
echo "Retrieving the existing Azure Identity..."
$existingIdentity = az resource list -g $aks.nodeResourceGroup --query "[?contains(type, 'Microsoft.ManagedIdentity/userAssignedIdentities')]" | ConvertFrom-Json
$identity = az identity show -n $existingIdentity.name -g $existingIdentity.resourceGroup | ConvertFrom-Json
} Else {
# If using AKS with Service Principal, create new Identity
echo "Creating an Azure Identity..."
$identity = az identity create -g $resourceGroupName -n $identityName | ConvertFrom-Json
}
$identity
{
"clientId": "a0c038fd-3df3-4eaf-bb34-abdd4f78a0db",
"clientSecretUrl": "https://control-westeurope.identity.azure.net/subscriptions/<AZURE_SUBSCRIPTION_ID>/resourcegroups/rg-demo/providers/Microsoft.ManagedIdentity/userAssignedIdentities/identity-aks-kv/crede
ntials?tid=<AZURE_TENANT_ID>&oid=f8bb59bd-b704-4274-8391-3b0791d7a02c&aid=a0c038fd-3df3-4eaf
"id": "/subscriptions/<AZURE_SUBSCRIPTION_ID>/resourcegroups/rg-demo/providers/Microsoft.Managed
Identity/userAssignedIdentities/identity-aks-kv",
"location": "westeurope",
"name": "identity-aks-kv",
"principalId": "000000-b704-4274-8391-3b0791d7a02c",
"resourceGroup": "rg-demo",
"tags": {},
"tenantId": "<AZURE_TENANT_ID>",
"type": "Microsoft.ManagedIdentity/userAssignedIdentities"
}
The Identity we created earlier will be used by AKS Pods to read secrets from Key Vault. Thus, it should have permissions to do so. We will assign it the Reader role to the KV scope.
az role assignment create --role "Reader" --assignee $identity.principalId --scope $keyVault.id
{
"canDelegate": null,
"id": "/subscriptions/<AZURE_SUBSCRIPTION_ID>/resourcegroups/rg-demo/providers/Microsoft.KeyVault/vaults/
az-key-vault-demo/providers/Microsoft.Authorization/roleAssignments/d6bd00b8-9734-4c53-9de3-5a5b203c3286",
"name": "d6bd00b8-9734-4c53-9de3-5a5b203c3286",
"principalId": "f8bb59bd-b704-4274-8391-3b0791d7a02c",
"principalName": "a0c038fd-3df3-4eaf-bb34-abdd4f78a0db",
"principalType": "ServicePrincipal",
"resourceGroup": "rg-demo",
"roleDefinitionId": "/subscriptions/<AZURE_SUBSCRIPTION_ID>/providers/Microsoft.Authorization/roleDefinit
ions/acdd72a7-3385-48ef-bd42-f606fba81ae7",
"roleDefinitionName": "Reader",
"scope": "/subscriptions/<AZURE_SUBSCRIPTION_ID>/resourcegroups/rg-demo/providers/Microsoft.KeyVault/vaul
ts/az-key-vault-demo",
"type": "Microsoft.Authorization/roleAssignments"
}
In case you chose AKS with Service Principal, you need also to grant permissions to the AKS cluster with role “Managed Identity Operator”. For that we will need the Service Principal (SPN) used with AKS.
# Run the following command only if using AKS with Service Principal
If ($isAKSWithManagedIdentity -eq "false") {
echo "Providing required permissions for MIC..."
az role assignment create --role "Managed Identity Operator" --assignee $aks.servicePrincipalProfile.clientId --scope $identity.id
}
{
"canDelegate": null,
"id": "/subscriptions/<AZURE_SUBSCRIPTION_ID>/resourcegroups/rg-demo/providers/Microsoft.ManagedIdentity/
userAssignedIdentities/identity-aks-kv/providers/Microsoft.Authorization/roleAssignments/c018c932-c06b-446c-863e-bc85c68
7cf69",
"name": "c018c932-c06b-446c-863e-bc85c687cf69",
"principalId": "2736b5eb-e79e-48fa-9348-19f9c64ce7b3",
"principalName": "http://aks-demoSP-20200430052736",
"principalType": "ServicePrincipal",
"resourceGroup": "rg-demo",
"roleDefinitionId": "/subscriptions/<AZURE_SUBSCRIPTION_ID>/providers/Microsoft.Authorization/roleDefinit
ions/f1a07417-d97a-45cb-824c-7a7467783830",
"roleDefinitionName": "Managed Identity Operator",
"scope": "/subscriptions/<AZURE_SUBSCRIPTION_ID>/resourcegroups/rg-demo/providers/Microsoft.ManagedIdenti
ty/userAssignedIdentities/identity-aks-kv",
"type": "Microsoft.Authorization/roleAssignments"
}
We should tell Key Vault to allow the Identity to do only specific actions on the secrets like get, list, delete, update. In our case, we need only permissions for GET. This permission is granted by using a Policy.
az keyvault set-policy -n $keyVaultName --secret-permissions get --spn $identity.clientId
{
"id": "/subscriptions/<AZURE_SUBSCRIPTION_ID>/resourceGroups/demo-rg/providers/Microsoft.KeyVault/vaults/kv-aks-demo",
"location": "westeurope",
"name": "kv-aks-demo",
"properties": {
"accessPolicies": [
{
… code removed for brievety …
"permissions": {
"certificates": null,
"keys": null,
"secrets": [
"get"
"softDeleteRetentionInDays": null,
"tenantId": "<AZURE_TENANT_ID>",
"vaultUri": "https://kv-aks-demo.vault.azure.net/"
},
"resourceGroup": "demo-rg",
"tags": {},
"type": "Microsoft.KeyVault/vaults"
}
Note
Note: To set policy to access keys or certs in keyvault, replace: --secret-permissions by: --key-permissions or --certificate-permissions.
The Pod needs to use the Identity to access to Key Vault. We’ll point to that Identity in AKS using AzureIdentity object and then we’ll assign it to the Pod through AzureIdentityBinding.
$aadPodIdentityAndBinding = @"
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
name: $($identityName)
spec:
type: 0
resourceID: $($identity.id)
clientID: $($identity.clientId)
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
name: $($identityName)-binding
spec:
azureIdentity: $($identityName)
selector: $($identitySelector)
"@
$aadPodIdentityAndBinding | kubectl apply -f -
azureidentity.aadpodidentity.k8s.io/identity-aks-kv created
azureidentitybinding.aadpodidentity.k8s.io/identity-aks-kv-binding created
Tip
Note: Set type: 0 for Managed Service Identity; type: 1 for Service Principal. In this case, we are using managed service identity, type: 0.
Note
Note the selector here as we’ll reuse it later with the pod that needs access to the Identity.
At this stage, we can create a Pod and mount CSI driver on which we’ll find the login and password retrieved from Key Vault. Let's deploying a Nginx Pod for testing
$nginxPod = @"
kind: Pod
apiVersion: v1
metadata:
name: nginx-secrets-store
labels:
aadpodidbinding: $($identitySelector)
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- name: secrets-store-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: $($secretProviderClassName)
"@
$nginxPod | kubectl apply -f -
pod/nginx-secrets-store created
Now, we’ll validate all what we have done before. Let’s list and read the content of the mounted CSI volume. If all is fine, we should see the secret values from our Key Vault.
kubectl exec -it nginx-secrets-store ls /mnt/secrets-store/
DATABASE_LOGIN DATABASE_PASSWORD
kubectl exec -it nginx-secrets-store cat /mnt/secrets-store/DATABASE_LOGIN
DbUserName
kubectl exec -it nginx-secrets-store cat /mnt/secrets-store/DATABASE_PASSWORD
MyP@ssword123456
To clean up the resources, you will purge Key Vault and delete the created resource groups.
az keyvault purge -n $keyVaultName
az group delete --no-wait --yes -n $resourceGroupName
az group delete --no-wait --yes -n $aks.nodeResourceGroup