CFK Schema Registry Basic Auth Vault Secret Auto-reload Failed
nvolynets opened this issue · 0 comments
Hi,
Details
First of all, issue described below belongs more to CFK operator but unfortunately posting it here as failed to find corresponding escalation point for CFK.
Trying to deploy Schema Registry with enabled HTTP Basic Authentication on K8S via CFK operator with enabled Vault integration according to Provide and Manage Sensitive Data for Confluent Platform in Confluent for Kubernetes.
All works fine as expected except basic auth credentials/ secrets rotation/ reload referred in jaas_config.conf
file.
SchemaRegistry-Props {
org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
file="/mnt/secrets/basic.txt"
debug="true";
};
To apply updated credentials Schema Registry pod and/or stateful-set restart required.
In the same time there is as follows statement in official Provide and Manage Sensitive Data for Confluent Platform in Confluent for Kubernetes doc:
Vault injects secrets into the CFK or Confluent Platform component pods on specific directory paths in the containers. Then, in the Confluent Platform component custom resources (CRs), the secrets are referenced via those directory paths (directoryPathInContainer).
If secrets are changed, Vault updates them in the pod, and CFK dynamically reads the updated info.
But it doesn't work as stated above.
STRs:
- Enable HTTP Basic Authentication in Schema Registry as depicted in Confluent for Kubernetes API Reference (2.8) specification.
- Deploy Schema Registry with Vault integration enabled according to Provide and Manage Sensitive Data for Confluent Platform in Confluent for Kubernetes.
- Successfully sign-in into Schema Registry (make some test request with using appropriate credentials specified in
basic.txt
Vault secret to verify Vault integration). - Update
basic.txt
Vault secret (just change password for someadmin
user). - Successfully check if corresponding updated secret has been delivered/ modified in corresponding Schema Registry pod.
- Wait 10 minutes (next step will fail even if wait an hour).
- Fail to sign-in into Schema Registry via using new creds (old still works).
- Schema Registry pod and/or stateful-set restart required to reload
basic.txt
referred injaas_config.conf
file.
Having secrets exposed into Vault it is expected to rotate them from time-to-time. Unfortunately, forced each time restart Schema Registry is not an option here :(
Btw, for Kafka Broker & KsqlDB spinned up via CFK according to Provide and Manage Sensitive Data for Confluent Platform in Confluent for Kubernetes. Secrets reload works as expected (automatically). It is yet another point to fix it for Schema Registry too to resurrect consistency.
Finally, the same issue applicable for Kafka Connect deployed via CFK but it looks like that it should escalated separately.
Analysis
Performed small investigation in this direction looks found out the root cause why updated secret/ creds are not reloaded automatically.
Confirmed that the secret automatically is updated in a corresponding Schema Registry pod by Vault Agent Injector operator.
But issue root cause looks like is in a jaas_config.conf
file provided/ generated by CFK operator for Jetty app server: missed reloadInterval
option. By default it is 0
which means "hot" props reload is disabled.
Providing below references to corresponding Jetty's classes source code:
Consequently, as a proposal to sort it out please consider to extend CFK operator's Schema Registry CRD schema with providing possibility to specify reloadInterval
option.