Support SSL in Kafka connectivity

[mishamo]

We connect to Kafka via SSL. It should be possible to (optionally) add SSL config to the producer and/or consumer.

The only complication is how we would make the whole SSL block optional within application.conf.

Do you have any strong opinions of how to achieve this? Otherwise we can put together a PR.

[lacarvalho91]

@mishamo with pureconfig you can just create a new case class for the ssl related config and make it an optional field in the scheduler config case class.

You might also be able to supply any kafka properties in the consumer/producer kafka-clients config block. Reactive kafka will read all of these and add them as properties to the kafka consumer/producer I believe.

I'm not sure how you configure ssl on kafka producer/consumer though, if it's just some additional properties it should be pretty trivial.

Feel raise to raise a PR. If you have any problems I'd be more than happy to try and help.

[mishamo]

The easiest way I can see is to add the following block under each kafka-clients section:

       security.inter.broker.protocol=${?SECURITY_INTER_BROKER_PROTOCOL}

        ssl {
          key.password=${?KEY_PASSWORD}

          keystore {
            location=${?KEYSTORE_LOCATION}
            password=${?KEYSTORE_PASSWORD}
          }

          truststore {
            location=${?TRUSTSTORE_LOCATION}
            password=${?TRUSTSTORE_PASSWORD}
          }
        }

However, the complexity would come in when making this optional. As this is inside the actorsystem config, we never directly interact with this config, and I'm unsure how the producer/consumer would react to being passed a bunch of blank properties.

[lacarvalho91]

@mishamo do they have to be passed as blank? They should either exist or they shouldn't. You can inject typesafe config entries as java opts when you run the app, or even inject an external file

[lacarvalho91]

when running this component as a Docker container you can create your own image which extends from the one we provide and bake in your own application.conf which can be templated using confd if you only need those ssl fields for specific environments etc

[mishamo]

Resolved by #18