Provide CDC security materials via secrets

[Miles-Garnsey]

What is missing?

CDC security materials (truststorePassword, keystorePassword, pulsarAuthParams) are all provided direct in the manifest at present.

There should be an option to provide these via secrets instead. We should also retain the ability to provide them via the manifest (this is useful for testing, and required for backwards compatibility).

┆Issue is synchronized with this Jira Story by Unito

[burmanm]

I think the only correct approach here would be to ensure that the cdc-java-agent takes it input as a configuration file. Like we do with the metrics configuration and vector configuration. After that, things become a lot simpler.

In the cdc-agent configuration (additional-java-opts) we could allow overriding the configuration filename, but otherwise it probably has a default one. With that inplace, the only requirement to hide the configuration is to create a secret with the intended configuration and mount it all as a AdditionalVolumeSource:

    additionalVolumes:
      - name: cdc-config
        mountPath: /opt/cdc/config
        volumeSource:
          secret:
            name: "my-cdc-config"

Any other approach seems like a bad hack.

[Miles-Garnsey]

I agree with @burmanm that passing all configuration parameters via the startup argument to Cassandra is sketchy, to say the least. Providing a config file would be preferred indeed.

Any time startup arguments are used, a simple ps on the host will reveal the secrets.

However, we are not in control of the CDC agent project and if we we want to expedite a partial fix we have two options:

1. Shim out the existing UpdateConfig logic to draw encryption materials from the secrets and update the internal CDCConfiguration struct. The problem here is that the security materials will still appear in the StatefulSet's environment variables, because that's how they get passed in to config builder (and config builder is what manipulates the cassandra-env.sh to add the startup arguments).
2. Modify the config builder to read some files from disk and load their contents into the cassandra-env.sh, then mount secrets into the designated locations.

I don't write Clojure, so don't ask me how to execute (2). I had a look yesterday and it is vogon poetry to me.

[burmanm]

Alright, time for some larger workaround then. I think we can do this without changing anything right now. I first had a complex scenario that I described to John using the magic of container-runtime's override, but suddenly I simplified it in my mind..

Create Secret that includes a script that sets properties

The script itself would write the required changes to cassandra-env.sh, like:

        echo "" >> ${CASSANDRA_CONF}/cassandra-env.sh
        echo "JVM_OPTS=\"\$JVM_OPTS -javaagent:/opt/cdc_agent/cdc-agent.jar=topicPrefix=moo,pulsarAuthParams=cow\"" >> ${CASSANDRA_CONF}/cassandra-env.sh

Add a new initContainer

In the PodTemplateSpec:

  podTemplateSpec:
    spec:
      initContainers:
        - name: runmyscript
           image: busybox
           command: ["/bin/sh", "-c", "./secret/cdc-config-generator.sh"]

Profit

Cassandra will load this configuration and the java-agent. Obviously, can't use CDCConfiguration, but other than that, this will work without any modifications to cass-operator or mgmt-api images.

[jsanda]

@burmanm Will that container run before config builder or do we need:

  podTemplateSpec:
    spec:
      initContainers:
        - name: server-config-init
        - name: runmyscript
           image: busybox
           command: ["/bin/sh", "-c", "./secret/cdc-config-generator.sh"]

[burmanm]

We might need that. I think cass-config-builder fetches cassandra 3.11.6 cassandra-env.sh and then applies its modifications to that one, thus wiping out the existing cassandra-env.sh

https://github.com/datastax/cass-config-builder/blob/master/src/lcm/utils/oss_definitions_gen.clj#L194

[jsanda]

Why would the script need to run in the cassandra container? I can do those things with the init container.

[Miles-Garnsey]

I deleted my comment, I forgot that the config is stored in an emptyDir, and didn't realise that the initContainers ran one at a time. (I'm not sure if that was always the case, but maybe I'm misremembering a related problem I had with sidecar + initContainer interactions).

In any event, under these circumstances I think Michael's proposal works better than what I was suggesting.