[Feature] TLS support

Question

[Feature] TLS support

Opened this issue a year ago · 8 comments

Is your feature request related to a problem? Please describe.
I would like to manage TLS setup for MariaDB pods in a Kubernetes-native way :)

Describe the solution you'd like
I would like to be able to specify tls property when deploying a MariaDB manifest. It should be possible to specify an existing TLS secret or to ask the operator for an auto-generated one. Updates to this Secret should be reflected in the pod (process or pod restarted?)

Describe alternatives you've considered
I think it is possible now with some myCnf magic and mounting correct Secret into the pod

Additional context

#237

Answer 1 · 2023-06-10T21:23:41.000Z

Hey @wasd171 ! Thanks for bringing this up 🙏🏻

Indeed, providing TLS secrets is something we don't directly support, but it can be achieved by mounting an extra volume and pointing to it in the myCnf:

apiVersion: mariadb.mmontes.io/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  ...
  volumes: 
    - name: tls
      secret:
          secretName: tls
  volumeMounts: 
    - name: tls
      mountPath: /certs

  myCnf: |
    [mysqld]
    ...
    ssl_cert = /certs/server-cert.pem
    ssl_key = /certs/server-key.pem
    ssl_ca = /certs/ca-cert.pem

This is far from ideal and not the best cloud native experience, so I would suggest the following changes in the MariaDB CRD to support this:

apiVersion: mariadb.mmontes.io/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
   ...
   tls:
      secretRef: # reference to an existing TLS secret
          name: tls
      certificateRef: # reference to a cert-manager Certificate resource
           name: cert
      selfSigned: true # the operator will generate a self-signed certificate
      autoReload: true # reload the StatefulSet whenever the certificate changes

spec.secretRef, spec.certificateRef and spec.selfSigned will be mutually exclusive i.e. only one certificate source can be provided.

I think your suggestion makes total sense, but it would be worth integrating with cert-manager, which is the de facto standard project for managing certificates inside Kubernetes. This will be and opt-in feature that can be enabled via a value in the Helm chart.

Regarding reloading certificates, the operator will watch the certificate sources and trigger an update in the StatefulSet whenever they are re-issued. To make it as safe as possible, the user will be able to specify a updateStrategy for the StatefulSet in the MariaDB, so the updates are totally under control. Alternatively, we could leverage our sidecar agent to watch the directory where the certificates are mounted and reload the mariadbd process reactively but this can potentially lead to outages as the agents don't work in a coordinated way. For example, reloading certificates in all replicas at the same time in a Galera cluster will require recovering the Galera cluster.

Thoughts? 🤔

Answer 2 · 2023-06-11T17:53:09.000Z

Hi @mmontes11! Thanks for super detailed feedback :)

I think integrating cert-manager makes total sense, this removes the hassle of rotating certificates, etc.
I am not sure whether the certificateRef field is really needed – in the end the Certificate would anyway point to a Secret, so just having secretRef should be enough.

Format selfSigned: true is probably not enough to distinguish between the "use cert-manager" and "self-signed". Perhaps something like already used for webhook, tls:certManager = boolean? If tls field is present and there is no secretRef then we either generate the certificate by the operator or via cert-manager

I cannot say much about reloading strategies, I can just point out, that PostgreSQL operators typically allow to set 2 keys: one for the certificate itself and another for the ca.crt (see https://cloudnative-pg.io/documentation/1.20/certificates/#user-provided-certificates-mode, https://access.crunchydata.com/documentation/postgres-operator/4.7.2/tutorial/tls/). I think this allows for a following optimization: if the certificate has changed but the ca.crt is the same then we can simply reload all the processes with new certificates (since clients typically rely on the ca.crt for TLS checks). However, if the ca.crt has changed then the processes need to be reloaded in a specific order – new master should get the correct ca.crt first, replicas should follow. I am not sure what would be the best strategy for Galera mode :(

As a final note, I guess the backup / restore CRDs should also be TLS-aware if the MariaDB is configured as such

Answer 3 · 2023-06-19T16:20:47.000Z

Hey @wasd171 !

I am not sure whether the certificateRef field is really needed – in the end the Certificate would anyway point to a Secret, so just having secretRef should be enough.

I agree with this. More than a reference to a secret we may want to pass a reference to a cert-manager Issuer or ClusterIssuer so the operator is able to reconcile a Certificate. This way the user don't need to worry about issuing the certificate, only configure an issuer in the first place.

Considering this, and also your other suggestions, I think we can re-iterate my previous API spec. On one hand, we should allow the user to provide a secret with an existing certificate:

apiVersion: mariadb.mmontes.io/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
   ...
   tls:
      certificateSecretRef: # reference to a Kubernetes TLS secret
           name: my-tls-secret
      caSecretKeyRef: # reference to a Kubernetes secret key containing the CA public key used to establish trust in the client side
           name:  my-tls-secret
           key: ca.crt

As an alternative, the user could just provide a reference to a issuer which will result in the operator reconciling an extra Certificate resource which later on will be used by MariaDB:

apiVersion: mariadb.mmontes.io/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
   ...
   tls:
      certManager:
           issuerRef:  # reference to a cert-manager Issuer/ClusterIssuer resource
              name: vault-issuer
              namespace: vault
              kind: ClusterIssuer
           secretName: mariadb-tls # secretName to be used

For this case, the operator will perform late initialization by filling up spec.certificateSecretRef and spec.caSecretKeyRef with the details of the Certificate issued by cert-manager:

apiVersion: mariadb.mmontes.io/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
   ...
   tls:
      certManager:
           issuerRef:  # reference to a cert-manager Issuer/ClusterIssuer resource
              name: vault-issuer
              namespace: vault
              kind: ClusterIssuer
           secretName: mariadb-tls # secretName to be used
      certificateSecretRef: # reference to a Kubernetes TLS secret
           name: mariadb-tls
      caSecretKeyRef: # reference to a Kubernetes secret key containing the CA public key used to establish trust in the client side
           name: mariadb-tls
           key: ca.crt

Regarding reloading, there will be a controller watching the secrets provided in spec.certificateSecretRef and spec.caSecretKeyRef and trigger a StatefulSet restart when that happens. To make it as safe as possible, the user will be able to specify a updateStrategy for the StatefulSet in the MariaDB, so the updates are totally under control.

Does this make sense to you?

Answer 4 · 2023-06-20T11:48:32.000Z

Hi @mmontes11

Yes, this proposal makes total sense. The only thing that it is missing is a support for non-cert-manager self-generated certificates – but since I do not plan to use it myself and believe that TLS should be implemented properly anyway, this would not matter much for me :)

Answer 5 · 2023-06-20T12:12:14.000Z

Hey @wasd171 ! That's a good point, we could levarage crypto/x509 to issue a self-signed certificate. I would argue that this is not the best practice for production but definetly useful if we are provisioning MariaDB instances for development purposes.

I think we will initially go with the aforementioned approaches (cert-manager + user provided cert) but happy to introduce spec.tls.selfSigned later on 👍🏻 .

Answer 6 · 2023-06-20T12:14:18.000Z

Besides, cert-manager also supports setting up a self-signed issuer:

https://cert-manager.io/docs/configuration/selfsigned/

Answer 7 · 2024-06-10T13:59:21.000Z

Hi,

thanks for your good work!
Can you estimate how long it will take until the TLS support has been added to the operator?

Thanks and regards,
Christian

Answer 8 · 2024-06-25T14:08:07.000Z

Hey there @ChristianHeich-bosch ! No ETA unfortunately, but it is the highest priority of the next release, stay tuned!