Elasticsearch issue starting after K8s node upgrade UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

[suciuandrei94]

Chart version: 7.17.3

Kubernetes version: v1.22.12+vmware.1

Kubernetes provider: Self-managed VMware Tanzu Kubernetes Grid

Helm Version: v3.8.0

Describe the bug: I installed elasticsearch with helm and it worked fine initially, however when I upgraded my K8s version it stopped working: efk-elasticsearch-master-0 0/1 Running 0 20h

Steps to reproduce:

1. Install Elasticsearch
2. Upgrade K8s version on the cluster

Expected behavior: Elasticsearch would be able to restart without a problem on the new nodes.

Provide logs and/or server output (if relevant):

{"type": "server", "timestamp": "2022-11-02T15:42:52,110Z", "level": "ERROR", "component": "o.e.x.s.a.e.ReservedRealm", "cluster.name": "efk-elasticsearch", "node.name": "efk-elasticsearch-master-0", "message": "failed to ret
rieve password hash for reserved user [elastic]", "cluster.uuid": "x103Xij-RQ2f24CWR1KF4Q", "node.id": "tPvzVmUmSGqauMq5Wls8tw" , 
"stacktrace": ["org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable",
{"type": "server", "timestamp": "2022-11-02T15:42:52,201Z", "level": "INFO", "component": "o.e.x.s.a.RealmsAuthenticator", "cluster.name": "efk-elasticsearch", "node.name": "efk-elasticsearch-master-0", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "cluster.uuid": "x103Xij-RQ2f24CWR1KF4Q", "node.id": "tPvzVmUmSGqauMq5Wls8tw"  }

Additional info: I was going through this article https://discuss.elastic.co/t/at-least-one-primary-shard-for-the-index-security-7-is-unavailable-issue/316149 , it looks like ES didn't do a graceful shutdown and this is why it cannot start.

Thanks in advance for the help

[andreatera]

same here

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.