Key Vault FlexVolume for Kubernetes - Integrates Key Management Systems with Kubernetes via a FlexVolume.
With the Key Vault FlexVolume, developers can mount multiple secrets, keys, and certs stored in Key Management Systems into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container's file system.
- Azure Key Vault
💡 NOTE: To enable encryption at rest of Kubernetes data in etcd using Azure Key Vault, use Kubernetes KMS plugin for Azure Key Vault.
The detailed design of this solution:
💡 Make sure you have a Kubernetes cluster
Follow this to use aks-engine to create a new Kubernetes cluster with the Key Vault FlexVolume already deployed.
kubectl create -f https://raw.githubusercontent.com/Azure/kubernetes-keyvault-flexvol/master/deployment/kv-flexvol-installer.yaml
To validate the installer is running as expected, run the following commands:
kubectl get pods -n kv
You should see the keyvault flexvolume installer pods running on each agent node:
keyvault-flexvolume-f7bx8 1/1 Running 0 3m
keyvault-flexvolume-rcxbl 1/1 Running 0 3m
keyvault-flexvolume-z6jm6 1/1 Running 0 3m
The KeyVault FlexVolume offers two modes for accessing a Key Vault instance: Service Principal and Pod Identity.
Add your service principal credentials as a Kubernetes secrets accessible by the KeyVault FlexVolume driver.
kubectl create secret generic kvcreds --from-literal clientid=<CLIENTID> --from-literal clientsecret=<CLIENTSECRET> --type=azure/kv
Ensure this service principal has all the required permissions to access content in your key vault instance. If not, you can run the following using the Azure cli:
# Assign Reader Role to the service principal for your keyvault
az role assignment create --role Reader --assignee <principalid> --scope /subscriptions/<subscriptionid>/resourcegroups/<resourcegroup>/providers/Microsoft.KeyVault/vaults/<keyvaultname>
az keyvault set-policy -n $KV_NAME --key-permissions get --spn <YOUR SPN CLIENT ID>
az keyvault set-policy -n $KV_NAME --secret-permissions get --spn <YOUR SPN CLIENT ID>
az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn <YOUR SPN CLIENT ID>
Fill in the missing pieces in this deployment for your own deployment, make sure to:
- reference the service principal kubernetes secret created in the previous step
secretRef:
name: kvcreds
- pass in properties for the Key Vault instance to the flexvolume driver.
Name | Required | Description | Default Value |
---|---|---|---|
usepodidentity | no | specify access mode: service principal or pod identity | "false" |
keyvaultname | yes | name of KeyVault instance | "" |
keyvaultobjectnames | yes | names of KeyVault objects to access | "" |
keyvaultobjectaliases | no | filenames to use when writing the objects | keyvaultobjectnames |
keyvaultobjecttypes | yes | types of KeyVault objects: secret, key or cert | "" |
keyvaultobjectversions | no | versions of KeyVault objects, if not provided, will use latest | "" |
resourcegroup | yes | name of resource group containing key vault instance | "" |
subscriptionid | yes | name of subscription containing key vault instance | "" |
tenantid | yes | name of tenant containing key vault instance | "" |
keyvaultobjectnames, keyvaultobjecttypes and keyvaultobjectversions are semi-colon (;) separated.
- Specify mount path of flexvolume to mount key vault objects
volumeMounts:
- name: test
mountPath: /kvmnt
readOnly: true
Example of an nginx pod accessing a secret from a key vault instance:
apiVersion: v1
kind: Pod
metadata:
name: nginx-flex-kv
spec:
containers:
- name: nginx-flex-kv
image: nginx
volumeMounts:
- name: test
mountPath: /kvmnt
readOnly: true
volumes:
- name: test
flexVolume:
driver: "azure/kv"
secretRef:
name: kvcreds # [OPTIONAL] not required if using Pod Identity
options:
usepodidentity: "false" # [OPTIONAL] if not provided, will default to "false"
keyvaultname: "testkeyvault" # [REQUIRED] the name of the KeyVault
keyvaultobjectnames: "testsecret" # [REQUIRED] list of KeyVault object names (semi-colon separated)
keyvaultobjectaliases: "secret.json" # [OPTIONAL] list of KeyVault object aliases
keyvaultobjecttypes: secret # [REQUIRED] list of KeyVault object types: secret, key, cert
keyvaultobjectversions: "testversion" # [OPTIONAL] list of KeyVault object versions (semi-colon separated), will get latest if empty
resourcegroup: "testresourcegroup" # [REQUIRED] the resource group of the KeyVault
subscriptionid: "testsub" # [REQUIRED] the subscription ID of the KeyVault
tenantid: "testtenant" # [REQUIRED] the tenant ID of the KeyVault
Deploy your app
kubectl create -f deployment/nginx-flex-kv.yaml
Validate the pod has access to the secret from key vault:
kubectl exec -it nginx-flex-kv cat /kvmnt/testsecret
testvalue
💡 Make sure you have installed pod identity to your Kubernetes cluster
This project makes use of the aad-pod-identity project located here to handle the identity management of the pods. Reference the aad-pod-identity README if you need further instructions on any of these steps.
Not all steps need to be followed on the instructions for the aad-pod-identity project as we will also complete some of the steps on our installation here.
-
Install the aad-pod-identity components to your cluster
-
Install the RBAC enabled aad-pod-identiy infrastructure components:
kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
-
(Optional) Providing required permissions for MIC
- If the SPN you are using for the AKS cluster was created separately (before the cluster creation - i.e. not part of the MC_ resource group) you will need to assign it the "Managed Identity Operator" role.
az role assignment create --role "Managed Identity Operator" --assignee <sp id> --scope <full id of the managed identity>
- If the SPN you are using for the AKS cluster was created separately (before the cluster creation - i.e. not part of the MC_ resource group) you will need to assign it the "Managed Identity Operator" role.
-
-
Create an Azure User Identity
Create an Azure User Identity with the following command. Get
clientId
andid
from the output.az identity create -g <resourcegroup> -n <idname>
-
Assign permissions to new identity Ensure your Azure user identity has all the required permissions to read the keyvault instance and to access content within your key vault instance. If not, you can run the following using the Azure cli:
# Assign Reader Role to new Identity for your keyvault az role assignment create --role Reader --assignee <principalid> --scope /subscriptions/<subscriptionid>/resourcegroups/<resourcegroup>/providers/Microsoft.KeyVault/vaults/<keyvaultname> # set policy to access keys in your keyvault az keyvault set-policy -n $KV_NAME --key-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID> # set policy to access secrets in your keyvault az keyvault set-policy -n $KV_NAME --secret-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID> # set policy to access certs in your keyvault az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID>
-
Add a new
AzureIdentity
for the new identity to your clusterEdit and save this as
aadpodidentity.yaml
Set
type: 0
for Managed Service Identity;type: 1
for Service Principal In this case, we are using managed service identity,type: 0
. Create a new name for the AzureIdentity. SetResourceID
toid
of the Azure User Identity created from the previous step.apiVersion: "aadpodidentity.k8s.io/v1" kind: AzureIdentity metadata: name: <any-name> spec: type: 0 ResourceID: /subscriptions/<subid>/resourcegroups/<resourcegroup>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<idname> ClientID: <clientid>
kubectl create -f aadpodidentity.yaml
-
Add a new
AzureIdentityBinding
for the new Azure identity to your clusterEdit and save this as
aadpodidentitybinding.yaml
apiVersion: "aadpodidentity.k8s.io/v1" kind: AzureIdentityBinding metadata: name: demo1-azure-identity-binding spec: AzureIdentity: <name_of_AzureIdentity_created_from_previous_step> Selector: <label value to match in your app>
kubectl create -f aadpodidentitybinding.yaml
-
Add the following to your deployment yaml, like deployment/nginx-flex-kv-podidentity.yaml:
a. Include the
aadpodidbinding
label matching theSelector
value set in the previous step so that this pod will be assigned an identitymetadata: labels: aadpodidbinding: "NAME OF the AzureIdentityBinding SELECTOR"
b. make sure to update
usepodidentity
totrue
usepodidentity: "true"
-
Deploy your app
kubectl create -f deployment/nginx-flex-kv-podidentity.yaml
-
Validate the pod has access to the secret from key vault:
kubectl exec -it nginx-flex-kv-podid cat /kvmnt/testsecret testvalue
NOTE When using the Pod Identity
option mode, there can be some amount of delay in obtaining the objects from keyvault. During the pod creation time, in this particular mode aad-pod-identity
will need to create the AzureAssignedIdentity
for the pod based on the AzureIdentity
and AzureIdentityBinding
, retrieve token for keyvault. This proccess can take time to complete and it's possible for the pod volume mount to fail during this time. When the volume mount fails, kubelet will keep retrying until it succeeds. So the volume mount will eventually succeed after the whole process for retrieving the token is complete.
The Key Vault FlexVolume interacts with keyvault objects by using the keyvault API. If you need to understand the difference between Keys, Secrets and Certificate objects, we recommend that you start by reading the thorough documentation available on Keyvault : About keys, secrets, and certificates
It is important to understand how a certificate is structured in keyvault. As mentioned in the REST API docs here and here, Azure Key Vault (AKV) represents a given X.509 certificate via three interrelated resources: an AKV-certificate, an AKV-key, and an AKV-secret. All three will share the same name and the same version and can be fetched independently.
- The AKV-certificate provides the public key and certificate metadata. Specifying
cert
inkeyvaultobjecttypes
will fetch the public key and certificate metadata. - The AKV-key provides the private key of the X.509 certificate. It can be useful for performing cryptographic operations such as signing if the corresponding certificate was marked as non-exportable. Specifying
key
inkeyvaultobjecttypes
will fetch the private key of the certificate if its policy allows for private key exporting. - The AKV-secret provides a way to export the full X.509 certificate, including its private key (if its policy allows for private key exporting). Specifying
secret
inkeyvaultobjecttypes
will fetch the base64-encoded certificate bundle.
The Key Vault FlexVolume project welcomes contributions and suggestions. Please see CONTRIBUTING for details.
This project has adopted the Microsoft Open Source Code of Conduct. For more information, see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.