replicaCount > 1 safe?
Nick-Triller opened this issue · 7 comments
Hi,
I believe the operator doesn't have any kind of leader election or other coordination. Is it safe to run external secrets operator with multiple replicas? No matter if it's safe or not, it should probably be documented in the README.
Thanks
Nick
As you say KES does not do any leader election or similar so running multiple replicas (watching the same namespace) will lead to race conditions and collisions when updating secrets. (Eg concurrent modifications will cause one to fail)
I would not recommend running multiple replicas as it would just increase the costs (multiple instances polling your secret store) and yield more errors.
Thanks for the insights. I am thinking about implementing coordination via a leader election sidecar. Is this a contribution the KES maintainers would be interested in?
Yes, having leader election in place would be nice. Its something that will be available in the rewrite as well (https://github.com/external-secrets/external-secrets).
Not sure about the sidecar approach, how does it interact with the KES container? It seems like a convenient way to do it tho, allows it to be optionally enabled in the helm chart when running multiple replicas.
My idea was to have the leader election sidecars provide an HTTP endpoint to the application container in the same pod. The application can use the endpoint to check if it's the leader currently. Therefore, some changes to the application code are required.
This approach is described here: https://kubernetes.io/blog/2016/01/simple-leader-election-with-kubernetes/
Makes sense, think I've read that post at some point now that I saw it again!
PR welcome! :)
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.
This issue was closed because it has been stalled for 30 days with no activity.