[PROPOSAL] Support adding CSI volumes to operator to pull secrets from Key vault
Closed this issue · 4 comments
What/Why
What are you proposing?
Provide the capability to add CSI volumes to the operator pod inorder to pull secrets from Azure Key vault.
What users have asked for this feature?
What problems are you trying to solve?
At present, the admin credentials secret has to be created in the cluster before it can be used by the operator. But with this change, this secret can be added to the key vault and the CSI secrets store driver will get the secret from the key vault and create a secret automatically
What is the developer experience going to be?
No REST API changes
Are there any security considerations?
No
Are there any breaking changes to the API
No
What is the user experience going to be?
Users will have 2 new fields they can add to the operator's Helm values file. These can be used to add a CSI secrets store volume.
extraVolumes: []
extraVolumeMounts: []
Are there breaking changes to the User Experience?
No
Why should it be built? Any reason not to?
This will enable users to store the admin credentials secret in a keyvault and use it
What will it take to execute?
The operator's deployment manifest needs to be modified to add the volumeMounts
field to the container spec and the volumes
field to the pod spec
I can contribute this
Any remaining open questions?
No
Hi @nilushancosta.
I'm not really sure how extra volumes for the operator pod can help with the admin credentials secret? That secret is per opensearch-cluster, so there are potentially any number of them per operator instance.
Could you please explain how you intend this to work? Maybe also give an example?
Hi @swoehrl-mw ,
I want to be able to use a CSI secrets store volume to create the adminCredentialsSecret
and the securityConfigSecret
used in the OpenSearchCluster
manifest - https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/docs/userguide/main.md#custom-admin-user
security:
config:
adminCredentialsSecret:
name: admin-credentials-secret # The secret with the admin credentials for the operator to use
securityConfigSecret:
name: securityconfig-secret # The secret containing your customized securityconfig
tls:
transport:
generate: true
http:
generate: true
I initially tried to use the CSI secret store driver through the additionalVolumes of the OpenSearchCluster
. However the CSI secrets store driver will only create the secrets when the pod where the volumeMount
is added starts (i.e. OpenSearch pod starts). But the operator will not start the OpenSearch pod unless the secret exists. It will keep printing the following logs
{"level":"info","ts":"2024-05-20T18:13:55.678Z","msg":"Waiting for secret 'securityconfig-secret' that contains the securityconfig to be created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"my-first-cluster","namespace":"test"},"namespace":"test","name":"my-first-cluster","reconcileID":"412104d5-6c66-4994-8aa5-2eed921b5130"}
{"level":"info","ts":"2024-05-20T18:14:27.079Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"my-first-cluster","namespace":"test"},"namespace":"test","name":"my-first-cluster","reconcileID":"05e35c83-31a1-4d67-b376-1633ceadcefc","interface":"transport"}
{"level":"info","ts":"2024-05-20T18:14:27.079Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"my-first-cluster","namespace":"test"},"namespace":"test","name":"my-first-cluster","reconcileID":"05e35c83-31a1-4d67-b376-1633ceadcefc","interface":"http"}
{"level":"error","ts":"2024-05-20T18:14:27.081Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"my-first-cluster","namespace":"test"},"namespace":"test","name":"my-first-cluster","reconcileID":"05e35c83-31a1-4d67-b376-1633ceadcefc","error":"Secret \"admin-credentials-secret\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
To solve this, I could add the following secret provider class and mount the CSI secrets store volume to the operator pod using the proposed fields as follows
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: opensearch
namespace: test
spec:
provider: azure
secretObjects:
- secretName: securityconfig-secret
type: Opaque
data:
- key: internal_users.yml
objectName: SECURITY_CONFIG_INTERNAL_USERS
- secretName: admin-credentials-secret
type: Opaque
data:
- key: username
objectName: USERNAME
- key: password
objectName: PASSWORD
parameters:
keyvaultName: test-kv
tenantId: aaa-bbb
usePodIdentity: "false"
objects: |
array:
- |
objectName: internal-users
objectType: secret
objectAlias: SECURITY_CONFIG_INTERNAL_USERS
objectVersion: 1111
- |
objectName: password
objectType: secret
objectAlias: PASSWORD
objectVersion: 2222
- |
objectName: username
objectType: secret
objectAlias: USERNAME
objectVersion: 3333
extraVolumeMounts:
- name: csi-volume-opensearch-secrets
mountPath: "/mnt/csi-secret-store/opensearch-secrets"
readOnly: true
extraVolumes:
- name: csi-volume-opensearch-secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: opensearch
nodePublishSecretRef:
name: azure
Then when the operator pod starts, these two secrets will get created. It would be possible to add secrets of all clusters to this secret provider class manifest and they will get created.
@nilushancosta. To be honest this sounds like quite the hack and not like something the operator should support.
I can see two options: Either you find a way to get the secrets created some other way (does it have to be mounted to the operator pod or would it suffice to have some random pod mount it so the secret is created?), or the operator gets proper support for handling other secret stores (which would likely be a bigger implementation).
okay @swoehrl-mw . I will check if there is another way to do this.
It is not required to mount it to the operator pod itself. If we could create another pod at the same time, that should create the secret.