HiveMQ-k8s-4.7.0 crashloop when upgrading from 4.6.4 => 4.7.0

[stefanjay]

Hey there,

I'm facing the issue when upgrading my HiveMQ cluster operator from v4.6.4 =to=> v4.7.0. Please see below code block for full stack trace. The main exception is

Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -XX:InitialRAMPercentage=30 -XX:MaxRAMPercentage=80 -XX:MinRAMPercentage=30
�[36m08:05:55.384�[0;39m �[1;30m[main]�[0;39m �[34mINFO �[0;39m �[35mcom.hivemq.Application�[0;39m - Preparing SSL files
�[36m08:05:57.590�[0;39m �[1;30m[main]�[0;39m �[34mINFO �[0;39m �[35mcom.hivemq.Application�[0;39m - Wrote converted key store to /tmp/store.p12
__ __ _ _
| \/ (_) ___ _ __ ___ _ __ __ _ _ _| |_
| |\/| | |/ __| '__/ _ \| '_ \ / _` | | | | __|
| | | | | (__| | | (_) | | | | (_| | |_| | |_
|_| |_|_|\___|_| \___/|_| |_|\__,_|\__,_|\__|
Micronaut (v2.4.2)
�[36m08:06:01.794�[0;39m �[1;30m[main]�[0;39m �[1;31mERROR�[0;39m �[35mio.micronaut.runtime.Micronaut�[0;39m - Error starting Micronaut server: Error instantiating bean of type [io.micronaut.http.server.netty.NettyHttpServer]: An error occurred configuring SSL
io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [io.micronaut.http.server.netty.NettyHttpServer]: An error occurred configuring SSL
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:1972)
at io.micronaut.context.DefaultBeanContext.createAndRegisterSingletonInternal(DefaultBeanContext.java:2724)
at io.micronaut.context.DefaultBeanContext.createAndRegisterSingleton(DefaultBeanContext.java:2710)
at io.micronaut.context.DefaultBeanContext.getBeanForDefinition(DefaultBeanContext.java:2382)
at io.micronaut.context.DefaultBeanContext.getBeanInternal(DefaultBeanContext.java:2356)
at io.micronaut.context.DefaultBeanContext.findBean(DefaultBeanContext.java:1282)
at io.micronaut.context.DefaultBeanContext.findBean(DefaultBeanContext.java:752)
at io.micronaut.context.BeanLocator.findBean(BeanLocator.java:149)
at io.micronaut.runtime.Micronaut.start(Micronaut.java:73)
at io.micronaut.runtime.Micronaut.run(Micronaut.java:311)
at io.micronaut.runtime.Micronaut.run(Micronaut.java:297)
at com.hivemq.Application.main(Application.java:48)
Caused by: io.micronaut.http.ssl.SslConfigurationException: An error occurred configuring SSL
at io.micronaut.http.ssl.SslBuilder.getKeyManagerFactory(SslBuilder.java:109)
at io.micronaut.http.server.netty.ssl.CertificateProvidedSslBuilder.build(CertificateProvidedSslBuilder.java:85)
at io.micronaut.http.server.netty.ssl.CertificateProvidedSslBuilder.build(CertificateProvidedSslBuilder.java:79)
at io.micronaut.http.server.netty.ssl.CertificateProvidedSslBuilder.build(CertificateProvidedSslBuilder.java:72)
at io.micronaut.http.server.netty.NettyHttpServer.<init>(NettyHttpServer.java:211)
at io.micronaut.http.server.netty.$NettyHttpServerDefinition.build(Unknown Source)
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:1943)
... 11 common frames omitted
Caused by: java.io.IOException: keystore password was incorrect
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source)
at java.base/sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source)
at java.base/java.security.KeyStore.load(Unknown Source)
at io.micronaut.http.ssl.SslBuilder.load(SslBuilder.java:144)
at io.micronaut.http.ssl.SslBuilder.getKeyStore(SslBuilder.java:124)
at io.micronaut.http.server.netty.ssl.CertificateProvidedSslBuilder.getKeyStore(CertificateProvidedSslBuilder.java:135)
at io.micronaut.http.ssl.SslBuilder.getKeyManagerFactory(SslBuilder.java:98)
... 17 common frames omitted
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
... 24 common frames omitted

Are there any necessary configuration changes which needs to take place before upgrading the operator? I was able to update the HiveMQ broker cluster w/o any issues from v4.6.4 =to=> v4.7.0 by doing a calm rolling restart.

Any help would be highly appreciated.

[sbaier1]

Which chart version did you install, can you post your values.yaml or part of it?
It looks like you might have tried installing chart <0.9.0 with operator version 4.7.0.
For this minor version upgrade you should definitely also use the new chart version.

[patrickjahns]

Thanks for reporting the issue.

Can you please elaborate a bit further, how exactly you performed the upgrade?

If you are deploying the operator via the helm-charts, please ensure that you are using the latest helm-chart (0.9.1).
If you are using the helm-chart - can you share your values.yaml (without secrets)

[stefanjay]

This issue can be closed as it was a the wrong helm chart version as spotted by @patrickjahns . Thanks for your answers. :)