Cluster config not persistent

Question

Cluster config not persistent

Closed this issue 4 years ago · 15 comments

Hello!
Due to zookeeper has no storage, and cluster config is applied only once on creation via job, cluster list is not persistent and will disappear after every CMAK pod recreation.
There is 2 possible ways to solve this:

Add persistent storage to Zookeeper.
Run cluster population script in an initContainer.
Thanks.

Answer 1 · 2020-11-09T15:24:02.000Z

Thnx for opening a bug. I am still thinking about proper fix for the issue.

Answer 2 · 2020-11-18T14:02:39.000Z

Ya this is a pretty big bug as it just makes the tool unusable. Perhaps adding an option to the chart to either add a disk or init is a flexible solution.

Answer 3 · 2020-11-18T14:08:49.000Z

PRs are welcomed, btw.

Answer 4 · 2020-11-18T14:31:48.000Z

I am making one don't worry, I just was giving you the two approaches I may take. I just made changes in my flux repo. If it works as expected I will add as a PR. Would you also be receptive to having a toggleable ingress?

…

On Wed, 18 Nov 2020, 15:09 Ievgenii Shepeliuk, ***@***.***> wrote: PRs are welcomed, btw. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG6KEUMKUBRIS5PWQWIZSFLSQPIQDANCNFSM4TNHZNGA> .

Answer 5 · 2020-11-18T14:40:49.000Z

Would you also be receptive to having a toggleable ingress?

Sounds good!

Answer 6 · 2020-11-18T15:27:42.000Z

Would you also be receptive to having a toggleable ingress?

Absolutely, I'd just prefer to have it in the different PR

Answer 7 · 2020-11-18T16:52:09.000Z

I will test it over the next day or two. If it keeps working I will submit my changes as a PR. I will also submit a separate one for the ingress then as well.

To get the initcontainer approach to work I had to split the zk and cmak deployments into two with separate services because the cmak job needs zk running or it won't execute. The job runs as an initcontainer to cmak-ui and watches for zk service to be active then runs and it all boots.

Answer 8 · 2020-11-18T19:48:52.000Z

Do you still need k8s Job while using initContainer for putting config into ZK ?
I suppose, don't.

initContainer should be added to ZK, not to cmak-ui.
Since the data disappeares when ZK container restarts.

Well, if you think about recinciliation, then it is not an initContainer approach, but a separate deployment should be created that reconciles ConfigMap to ZK if the data disappers.

Answer 9 · 2020-11-18T19:49:27.000Z

Maybe I just misunderstood the approach, i am open for further design discussion.

Answer 10 · 2020-11-19T23:17:01.000Z

Okay so you were 100% correct, putting it as an init container for the UI wouldn't work longterm. I instead went for an approach where the zookeeper pod has an init container that creates the setup job from a mounted configmap. This way on each reboot the job will run until the zookeeper is up. So far it seems to work. If it has no issues I will try and have a PR this week.

Answer 11 · 2020-11-20T08:42:06.000Z

Good news @sharkymcdongles , thanks

Answer 12 · 2020-11-28T15:53:54.000Z

@sharkymcdongles any news on this?

Answer 13 · 2020-11-28T21:02:04.000Z

Sorry I took the week off for Thanksgiving. It seems to work though as I heard no complaints after making the change. I will submit a PR for the job tweak first then the ingress separate. I hope to do it Monday or Tuesday.

…

On Sat, 28 Nov 2020, 16:54 Alexey T, ***@***.***> wrote: @sharkymcdongles <https://github.com/sharkymcdongles> any news on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG6KEUOJAK6H6JOPELOUG2LSSEMJ5ANCNFSM4TNHZNGA> .

Answer 14 · 2020-11-30T11:24:19.000Z

Please take a look at PR #3

Answer 15 · 2020-12-02T09:48:02.000Z

@sharkymcdongles @lexasoft123 released 0.7.0

Please give it a try.