Cannot deploy new cluster without disabling elasticsearch post-start

Question

Cannot deploy new cluster without disabling elasticsearch post-start

Closed this issue 8 years ago · 8 comments

Seems like an initial cluster when deploying the canary node can't spin up properly, because there's only one data-node, and it's in a yellow-state. The user experience for "deploy initially disabling post start, then redeploy to enable it" seems sub-optimal.

Would things work better if it moved from post-start to post-deploy?

Answer 1 · 2017-04-14T19:44:01.000Z

@geofffranks Are you trying to upgrade existing Logsearch cluster or deploying from scratch?

Answer 2 · 2017-04-14T19:44:25.000Z

Deploying from scratch

…

On Apr 14, 2017, at 3:44 PM, Andrei Krasnitski ***@***.***> wrote: @geofffranks <https://github.com/geofffranks> Are you trying to upgrade existing Logsearch cluster or deploying from scratch? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#62 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG2SUlruS8rfToC_JHoRPq1jiDTrQX7mks5rv8yCgaJpZM4M-BxP>.

Answer 3 · 2017-04-14T20:22:33.000Z

Both jobs elasticsearch_master/elasticsearch_data have serial: true in update section and cluster should not get any data during the deployment process since ingestor/parser jobs are beginning to update only when elasticsearch is in healthy and empty state.
https://github.com/cloudfoundry-community/logsearch-boshrelease/blob/v206.0.1/templates/logsearch-jobs.yml#L145

Answer 4 · 2017-04-14T20:23:33.000Z

Something was generating a .kibana index, and it was yellow.

…

On Apr 14, 2017, at 4:22 PM, Andrei Krasnitski ***@***.***> wrote: Both jobs elasticsearch_master/elasticsearch_data have serial: true in update section and cluster should not get any data during the deployment process since ingestor/parser jobs are beginning to update only when elasticsearch is in healthy and empty state. https://github.com/cloudfoundry-community/logsearch-boshrelease/blob/v206.0.1/templates/logsearch-jobs.yml#L145 <https://github.com/cloudfoundry-community/logsearch-boshrelease/blob/v206.0.1/templates/logsearch-jobs.yml#L145> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#62 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG2SUmYOsWZCMoYmpMUsSJhioCPNUbJTks5rv9WKgaJpZM4M-BxP>.

Answer 5 · 2017-04-14T20:28:35.000Z

Make sure that all jobs are updated in the correct order.

Answer 6 · 2017-04-14T20:29:59.000Z

They seem to be. Had a kibana app running in the background, as logsearch got in such an unworkable state I had to bosh delete-deployment + recreate (but the cf app for kibana stayed up). Regardless, would moving the post-start script to post-deploy fix this, and the issues encountered when deploying using bosh2 azs?

…

On Apr 14, 2017, at 4:28 PM, Andrei Krasnitski ***@***.***> wrote: Make sure that all jobs are updated in the correct order. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#62 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG2SUkp7ks4B2VJaYesf6GmqvvSVkEOeks5rv9b0gaJpZM4M-BxP>.

Answer 7 · 2017-04-14T20:47:56.000Z

I guess that this Kibana app is part of a previous Logsearch deployment ;)
Bosh2 require additional steps only for upgrade ELK 2 -> ELK 5.
I don't like idea with post-deploy scripts until they will be enabled by default.

Answer 8 · 2017-04-21T19:27:22.000Z

@geofffranks: the recommended procedure for an elastic rolling restart is to wait for each node to rejoin the cluster and finish reallocating shards before moving on to the next: https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html. That's why we added the post-start script. Do you need to start kibana before the cluster is healthy?

By the way, which issues did you run into with bosh2 azs?