ES_PLUGINS_INSTALL breaks boot sequence
rewt opened this issue · 2 comments
After several different stages of debugging why my fresh instances are failing to boot, I've tracked it down to the ES_PLUGINS_INSTALL option. Without ES_PLUGINS_INSTALL, pods boot fine wihtout issue. Once ES_PLUGINS_INSTALL added, pods fail. I'm using the stateful master yaml files.
It seems any Pires image from 6.3.0 to latest shares same behaviour.
Steps to re-create
- Download fresh es-master-stateful.yaml
- Update image version to 6.4.0 (6.3.0 throws license error)
- Update storage settings to match platform
- Add ES_PLUGINS_INSTALL
ex.
- name: "ES_PLUGINS_INSTALL"
value: "mapper-murmur3"
Pod will load plugin, but then die.
Is there issue with stateful storage path? Unfortunately, I'm not able to find any logs.
Same issue on stateless configuration as well. Very curious - this was just working fine recently. What happened?
Pod log:
-> Downloading mapper-murmur3 from elastic
[=================================================] 100%
-> Installed mapper-murmur3
[2018-10-10T19:04:23,328][INFO ][o.e.n.Node ] [es-master-stateless-68bc4b596-6h5j7] initializing ...
[2018-10-10T19:04:24,207][INFO ][o.e.e.NodeEnvironment ] [es-master-stateless-68bc4b596-6h5j7] using [1] data paths, mounts [[/data (/dev/sda1)]], net usable_space [17.4gb], net total_space [29gb], types [ext4]
[2018-10-10T19:04:24,207][INFO ][o.e.e.NodeEnvironment ] [es-master-stateless-68bc4b596-6h5j7] heap size [247.5mb], compressed ordinary object pointers [true]
[2018-10-10T19:04:24,209][INFO ][o.e.n.Node ] [es-master-stateless-68bc4b596-6h5j7] node name [es-master-stateless-68bc4b596-6h5j7], node ID [CADIFrrJRRSSk5mD8sMFOA]
[2018-10-10T19:04:24,209][INFO ][o.e.n.Node ] [es-master-stateless-68bc4b596-6h5j7] version[6.3.1], pid[1], build[default/tar/eb782d0/2018-06-29T21:59:26.107521Z], OS[Linux/4.15.0-1023-azure/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_171/25.171-b11]
[2018-10-10T19:04:24,209][INFO ][o.e.n.Node ] [es-master-stateless-68bc4b596-6h5j7] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms256m, -Xmx256m, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-10-10T19:04:28,216][WARN ][o.e.d.c.s.Settings ] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [aggs-matrix-stats]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [analysis-common]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [ingest-common]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [lang-expression]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [lang-mustache]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [lang-painless]
[2018-10-10T19:04:31,728][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [mapper-extras]
[2018-10-10T19:04:31,802][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [parent-join]
[2018-10-10T19:04:31,802][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [percolator]
[2018-10-10T19:04:31,802][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [rank-eval]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [reindex]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [repository-url]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [transport-netty4]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [tribe]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-core]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-deprecation]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-graph]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-logstash]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-monitoring]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-rollup]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-security]
[2018-10-10T19:04:31,803][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-sql]
[2018-10-10T19:04:31,804][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-upgrade]
[2018-10-10T19:04:31,804][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded module [x-pack-watcher]
[2018-10-10T19:04:31,805][INFO ][o.e.p.PluginsService ] [es-master-stateless-68bc4b596-6h5j7] loaded plugin [mapper-murmur3]
[2018-10-10T19:04:31,806][INFO ][o.e.x.s.a.s.FileRolesStore] [es-master-stateless-68bc4b596-6h5j7] parsed [0] roles from file [/elasticsearch/config/roles.yml]