Node key not being exported to pkcs8
jacobreid opened this issue · 0 comments
jacobreid commented
I can create a cluster with the upmcenterprises/docker-elasticsearch-kubernetes:6.1.3_1 image fine, but when I try to use the searchguard image (https://github.com/while1eq1/elasticsearch-kubernetes-searchguard), elasticsearch fails to start up because the node key can not be found. The searchguard config specifies this to be in pkcs8 format (https://github.com/while1eq1/elasticsearch-kubernetes-searchguard/blob/master/config/elasticsearch.yml#L47) and this should be written out (https://github.com/upmc-enterprises/elasticsearch-operator/blob/master/pkg/k8sutil/certs.go#L206)
Changing ownership of /elasticsearch folder
chown: ./config/certs/..2019_03_18_15_47_15.821934694/ca.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/kibana.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/cerebro-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/ca-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node-keystore.jks: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/kibana-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/cerebro.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/truststore.jks: Read-only file system
Changing ownership of /data folder
chown: ./config/certs/..2019_03_18_15_47_15.821934694/ca.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/kibana.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/cerebro-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/ca-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/node-keystore.jks: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/kibana-key.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/cerebro.pem: Read-only file system
chown: ./config/certs/..2019_03_18_15_47_15.821934694/truststore.jks: Read-only file system
Waiting for Elasticsearch to become ready before running sgadmin...
[2019-03-18T15:47:18,803][INFO ][o.e.n.Node ] [aae567bf-aa89-4558-b2fe-7c78083abd99] initializing ...
[2019-03-18T15:47:18,862][INFO ][o.e.e.NodeEnvironment ] [aae567bf-aa89-4558-b2fe-7c78083abd99] using [1] data paths, mounts [[/data (/dev/nvme0n1p2)]], net usable_space [89.7gb], net total_space [119.9gb], types [ext4]
[2019-03-18T15:47:18,862][INFO ][o.e.e.NodeEnvironment ] [aae567bf-aa89-4558-b2fe-7c78083abd99] heap size [1007.3mb], compressed ordinary object pointers [true]
[2019-03-18T15:47:18,863][INFO ][o.e.n.Node ] [aae567bf-aa89-4558-b2fe-7c78083abd99] node name [aae567bf-aa89-4558-b2fe-7c78083abd99], node ID [cbnqrXMlT66u6oE927Y0GA]
[2019-03-18T15:47:18,863][INFO ][o.e.n.Node ] [aae567bf-aa89-4558-b2fe-7c78083abd99] version[6.4.1], pid[19], build[default/tar/e36acdb/2018-09-13T22:18:07.696808Z], OS[Linux/4.9.0-7-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12]
[2019-03-18T15:47:18,863][INFO ][o.e.n.Node ] [aae567bf-aa89-4558-b2fe-7c78083abd99] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms1024m, -Xmx1024m, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2019-03-18T15:47:20,262][INFO ][o.e.p.p.PrometheusExporterPlugin] starting Prometheus exporter plugin
[2019-03-18T15:47:20,405][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /elasticsearch/config
[2019-03-18T15:47:20,447][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
[2019-03-18T15:47:20,454][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /elasticsearch/config/, from there the key- and truststore files are resolved relatively
[2019-03-18T15:47:20,508][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [aae567bf-aa89-4558-b2fe-7c78083abd99] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
... 6 more
Caused by: org.elasticsearch.ElasticsearchException: Unable to read /elasticsearch/config/certs/node-key.pkcs8.pem (/elasticsearch/config/certs/node-key.pkcs8.pem). Please make sure this files exists and is readable regarding to permissions. Property: searchguard.ssl.transport.pemkey_filepath
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.checkPath(DefaultSearchGuardKeyStore.java:809) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.resolve(DefaultSearchGuardKeyStore.java:210) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:327) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:151) ~[?:?]
at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193) ~[?:?]
at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
... 6 more
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
On one of the nodes:
bash-4.4# ls /elasticsearch/config/certs/
ca-key.pem cerebro-key.pem kibana-key.pem node-key.pem node.pem
ca.pem cerebro.pem kibana.pem node-keystore.jks truststore.jks
bash-4.4#
If there is some configuration required for the key to be exported to pkcs8 this should be documented in the README.
Using elasticsearch-operator 0.3.0 on Kubernetes v1.10.11.