Figure out what's up with 5.0

Question

Figure out what's up with 5.0

Closed this issue 8 years ago · 68 comments

Full log from running it:

$ docker run ...
[2016-04-14 16:41:30,521][WARN ][bootstrap                ] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: your kernel is buggy and you should upgrade
    at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:279)
    at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:616)
    at org.elasticsearch.bootstrap.JNANatives.trySeccomp(JNANatives.java:215)
    at org.elasticsearch.bootstrap.Natives.trySeccomp(Natives.java:99)
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:98)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:151)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:263)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:88)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
Exception in thread "main" java.lang.RuntimeException: please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster.
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:79)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:60)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:187)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:263)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:88)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
Refer to the log for complete error details.

(from #96)

Answer 1 · 2016-04-14T16:47:14.000Z

The code in question: https://github.com/elastic/elasticsearch/blob/0c9aca94d4cd295e690a2d522c2b729770d486b5/core/src/main/java/org/elasticsearch/bootstrap/Seccomp.java#L276-L280

Answer 2 · 2016-04-14T16:48:12.000Z

Wagering a guess, I'm thinking that Docker's seccomp filter is blocking that unimplemented syscall outright instead of returning ENOSYS.

Answer 3 · 2016-04-14T16:52:38.000Z

If I run it on Docker 1.9.1 (before seccomp filtering was implemented), it gets past that line and I get errors about missing cluster config instead, so that's pretty good confirmation of my guess.

Answer 4 · 2016-04-14T16:53:50.000Z

Same with --security-opt seccomp=unconfined -- so we might have to come up with a seccomp profile or try to convince upstream that this check should handle blocked syscalls too. 😢

Answer 5 · 2016-04-14T16:54:31.000Z

Here's the next hurdle (which is more one of new functionality than Docker blocking us):

Exception in thread "main" java.lang.RuntimeException: please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster.
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:79)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:60)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:187)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:263)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:88)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
Refer to the log for complete error details.

Answer 6 · 2016-04-14T17:00:57.000Z

https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking_50_settings_changes.html#_discovery_settings

The discovery.zen.minimum_master_node must bet set for nodes that have network.host, network.bind_host, network.publish_host, transport.host, transport.bind_host, or transport.publish_host configuration options set. We see those nodes as in "production" mode and thus require the setting.

Answer 7 · 2016-05-05T19:15:06.000Z

The saga continues with alpha2:

Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed
initial heap size [268435456] not equal to maximum heap size [1073741824]; this can cause resize pauses and prevents mlockall from locking the entire heap
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:93)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:66)
    at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:191)
    at org.elasticsearch.node.Node.start(Node.java:323)
    at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:206)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:269)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
Refer to the log for complete error details.

Answer 8 · 2016-05-06T02:45:04.000Z

Ok, we might want to add -E es.bootstrap.seccomp=false to our default config to overcome the seccomp filtering issue, but alpha3 of 5.0 might include -E es.bootstrap.ignore_system_bootstrap_checks=true, if we're lucky. 😅

Answer 9 · 2016-05-06T02:45:36.000Z

(At least for our tests -- I don't think our default config should reasonably include es.bootstrap.ignore_system_bootstrap_checks.)

Answer 10 · 2016-05-06T02:46:24.000Z

(from elastic/elasticsearch@8e178c4)

Answer 11 · 2016-05-09T13:18:25.000Z

@tianon hi, the error you show from alpha2, how can one fix that? Running the current image gives me the same error.

Answer 12 · 2016-05-09T16:02:28.000Z

I think adjusting vm.max_map_count is going to have to be something that happens on the host system.

I'm not sure yet how to overcome the heap size check properly 😞

Answer 13 · 2016-05-09T16:02:59.000Z

$ docker run -it --rm --sysctl vm.max_map_count=262144 elasticsearch:5.0 -E es.bootstrap.seccomp=false
invalid value "vm.max_map_count=262144" for flag --sysctl: sysctl 'vm.max_map_count=262144' is not whitelisted
See 'docker run --help'.

Answer 14 · 2016-05-09T19:46:06.000Z

I see, so alpha2 image is unusable right now. I will look around tomorrow if I can be of any help

Answer 15 · 2016-05-10T06:38:20.000Z

@xificurC I increased max_map_count on my docker host via sudo sysctl -w vm.max_map_count=262144 and then started elasticsearch using docker run -e ES_JAVA_OPTS="-Xms1g -Xmx1g" elasticsearch:5.0.0-alpha2 and it seems to work.

Without the ES_JAVA_OPTS elasticsearch is complaining about

initial heap size [268435456] not equal to maximum heap size [1073741824]; this can cause resize pauses and prevents mlockall from locking the entire heap

and then just crashes.

Answer 16 · 2016-05-10T08:04:54.000Z

@tobstarr Thanks, these changes make alpha2 work

Answer 17 · 2016-06-03T20:09:26.000Z

Just ran across this as well. Is there a fix or a workaround?

I am using the Docker beta app and don't know of a way to adjust vm.max_map_count.

Answer 18 · 2016-06-10T13:40:53.000Z

@jmreicha I am running into the same issue and using the Beta app as well....did you find a workaround to this?

Answer 19 · 2016-06-10T14:23:12.000Z

@owjprice No I didn't spend much time on it, I just reverted back to 2.3 for now.

Answer 20 · 2016-06-10T14:33:26.000Z

I'm able to launch alpha3 successfully with the following, although it's far from ideal:

$ docker run -it --rm -e ES_JAVA_OPTS='-Xms1g -Xmx1g' elasticsearch:5 -E bootstrap.ignore_system_bootstrap_checks=true
....

Answer 21 · 2016-06-10T14:47:01.000Z

I can confirm @tianon 's command works, but container seems to hang after a while.

Answer 22 · 2016-06-17T19:15:40.000Z

@owjprice Got a chance to play around with this a little bit. They recently added an option to update the sysctl.conf in the VM. I added the following config in Preferences and hit apply changes and now the ES container comes up at least.

I'm still having trouble getting Kibana to connect but changing the max memory does help.

Answer 23 · 2016-06-21T13:58:57.000Z

@tianon 's solution from here works perfectly here, no problems with elastic or kibana. Somewhat strangely putting these options into the respective configuration files (jvm.options and elasticsearch.yml) didn't work for me.

Answer 24 · 2016-07-04T15:42:01.000Z

I'm not quite sure, why this is still open. There's an easy way to set two environment specific variables resp. docker settings with docker-1.11.2 and elastic-5.0.0-alpha4.

Problem:

$ docker run -it --name es -p 9200:9200 -p 9300:9300 es
[2016-07-04 14:42:45,427][WARN ][bootstrap                ] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: your kernel is buggy and you should upgrade
    at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:279)
    at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:616)
    at org.elasticsearch.bootstrap.JNANatives.trySeccomp(JNANatives.java:215)
    at org.elasticsearch.bootstrap.Natives.trySeccomp(Natives.java:99)
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:94)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:147)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:250)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
[2016-07-04 14:42:45,601][INFO ][node                     ] [Tenpin] version[5.0.0-alpha4], pid[1], build[3f5b994/2016-06-27T16:23:46.861Z], OS[Linux/4.4.0-28-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_91/25.91-b14]
[2016-07-04 14:42:45,601][INFO ][node                     ] [Tenpin] initializing ...
[2016-07-04 14:42:46,727][INFO ][plugins                  ] [Tenpin] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy], plugins []
[2016-07-04 14:42:47,619][INFO ][env                      ] [Tenpin] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda4)]], net usable_space [244.9gb], net total_space [271.2gb], spins? [possibly], types [ext4]
[2016-07-04 14:42:47,620][INFO ][env                      ] [Tenpin] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-07-04 14:42:49,567][INFO ][node                     ] [Tenpin] initialized
[2016-07-04 14:42:49,567][INFO ][node                     ] [Tenpin] starting ...
[2016-07-04 14:42:49,678][INFO ][transport                ] [Tenpin] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed
initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:125)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:85)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:65)
    at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:178)
    at org.elasticsearch.node.Node.start(Node.java:373)
    at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:193)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:252)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)

The warning about seccomp can be fixed by setting --security-opt seccomp=unconfined or by providing a seccomp profile.
The error about heap size can be fixed by providing "elastic search" javaopts like this: -e ES_JAVA_OPTS="-Xms1g -Xmx1g". Elastic wants min/max heapspace reservation for the JVM to be equal. That's all.
Generally, the host (linux/unix like OS, also MacOS) must support a higher amount of memory map areas by setting something like: sysctl -w vm.max_map_count=262144. As Elastic describes it on their site this is a necessary setting.

After having your memory map areas set to a higher amount (see last point), an example of my run of elastic-5.0 could look like this:

$ docker run -it --name es -p 9200:9200 -p 9300:9300 --security-opt seccomp=unconfined -e ES_JAVA_OPTS="-Xms1g -Xmx1g"  elasticsearch:5.0.0-alpha4

Answer 25 · 2016-07-09T03:34:21.000Z

@f3rdy Unfortunately your solution is not working for me:

xamox@xl2 ~% docker --version                                                     
Docker version 1.11.2, build b9f10c9

xamox@xl2 ~% docker run -it --name es -p 9200:9200 -p 9300:9300 --security-opt seccomp=unconfined -e ES_JAVA_OPTS="-Xms1g -Xmx1g"  elasticsearch:5.0.0-alpha4              
[2016-07-09 03:31:37,057][INFO ][node                     ] [Adam X] version[5.0.0-alpha4], pid[1], build[3f5b994/2016-06-27T16:23:46.861Z], OS[Linux/4.4.0-28-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_91/25.91-b14]
[2016-07-09 03:31:37,057][INFO ][node                     ] [Adam X] initializing ...
[2016-07-09 03:31:37,899][INFO ][plugins                  ] [Adam X] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy], plugins []
[2016-07-09 03:31:38,663][INFO ][env                      ] [Adam X] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda2)]], net usable_space [23.9gb], net total_space [226.3gb], spins? [possibly], types [ext4]
[2016-07-09 03:31:38,664][INFO ][env                      ] [Adam X] heap size [990.7mb], compressed ordinary object pointers [true]
[2016-07-09 03:31:39,928][INFO ][node                     ] [Adam X] initialized
[2016-07-09 03:31:39,928][INFO ][node                     ] [Adam X] starting ...
[2016-07-09 03:31:39,988][INFO ][transport                ] [Adam X] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
  at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:125)
  at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:85)
  at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:65)
  at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:178)
  at org.elasticsearch.node.Node.start(Node.java:373)
  at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:193)
  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:252)
  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
  at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
  at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
  at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
  at org.elasticsearch.cli.Command.main(Command.java:53)
  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
Refer to the log for complete error details.
[2016-07-09 03:31:39,998][INFO ][node                     ] [Adam X] stopping ...
[2016-07-09 03:31:40,008][INFO ][node                     ] [Adam X] stopped
[2016-07-09 03:31:40,008][INFO ][node                     ] [Adam X] closing ...
[2016-07-09 03:31:40,018][INFO ][node                     ] [Adam X] closed

xamox@xl2 ~% lsb_release -a                                                                                                                                                
No LSB modules are available.
Distributor ID: Ubuntu
Description:  Ubuntu 16.04 LTS
Release:  16.04
Codename: xenial

xamox@xl2 ~% uname -a                                                                                                                                                      
Linux xl2 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Answer 26 · 2016-07-09T06:47:06.000Z

From your stacktrace:

Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

You'll need to fix up your docker host to support more vm.max_map_count. For reference:

Elasticsearch also uses a mix of NioFS and MMapFS for the various files. Ensure that you configure the maximum map count so that there is ample virtual memory available for mmapped files. This can be set temporarily:

sysctl -w vm.max_map_count=262144

https://www.elastic.co/guide/en/elasticsearch/guide/current/_file_descriptors_and_mmap.html

Answer 27 · 2016-07-14T09:16:12.000Z

I'm getting this error, too:

master_1     | Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed
master_1     | max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

running ES5 on docker on Mac OS X. And I have no clue how to set vm.max_map_count there...

Answer 28 · 2016-07-14T09:31:02.000Z

@drahtGitter plz read above. Is there no sysctl in Mac OS X?

Answer 29 · 2016-07-14T14:29:28.000Z

@f3rdy I've set my memory in /etc/elasticsearch/jvm.options inside my container and in my compose-file I use:

security_opt:
            - seccomp:unconfined 

# for some reason -seccomp: unconfined (with a space) leads to this error:
services.data_node.security_opt contains an invalid type, it should be a string
# - label:seccomp:unconfined is accepted by compose but same result in the end

Still getting the error - and I don't see any hints on how to configure vm.max_map_count on my docker host (Mac OSX).

Creating a non existing /etc/sysctl.conf in OS X and adding vm.max_map_count=262144 has no effect.

Answer 30 · 2016-07-14T14:57:11.000Z

@f3rdy This sysctl method worked. Thanks!

Answer 31 · 2016-07-24T11:26:00.000Z

I resolved about Docker for mac in this way.

$ cd ~/Library/Containers/com.docker.docker/Data/database/com.docker.driver.amd64-linux/etc
$ echo "vm.max_map_count=262144" >> sysctl.conf
$ git add sysctl.conf
$ git commit -m 'Change sysctl.conf for elasticsearch 5'

Docker for mac will restart by the commit, the error disappeared:

Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

By the way, I could change vm.max_map_count by sysctl on Alpine linux.

http://larstechnica.com/2016/07/connect-to-docker-for-mac-alpine-host

But the changing reset by restarting of Docker for mac. The commit needed to persist it.

Answer 32 · 2016-07-24T23:57:06.000Z

I added this to my docker compose file

environment:    
    ES_JAVA_OPTS: "-Xms1g -Xmx1g"

And used the confuguration change suguested by @miyanaga on my docker for Mac. It stays up now.

Answer 33 · 2016-07-29T15:11:05.000Z

any fix? as the above is working until I'll try to connect via kibana

Answer 34 · 2016-08-02T17:22:36.000Z

@somuraijoe

I fixed this by editing /etc/elasticsearch/jvm.options file from this:

-Xms1024m
-Xmx2g

to this:

-Xms2g
-Xmx2g

Answer 35 · 2016-08-02T23:11:27.000Z

Successfully tested with Docker 1.10.3 on Ubuntu 16.04 using the ES_JAVA_OPTS variable:

docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e ES_JAVA_OPTS="-Xms512m -Xmx512m" elasticsearch:5

Answer 36 · 2016-08-09T23:29:45.000Z

I was curious why the way we run elasticsearch was not working, but using /etc/init.d/elasticsearch (provided by upstream's package) does work, and I've narrowed it down a bit. It would appear that if elasticsearch.yml is empty/default contents (ie, network.host is not set, and thus elasticsearch is listening on localhost only) that bootstrap.ignore_system_bootstrap_checks is implied somehow, so these errors are only warnings instead. This makes me wonder if we should just add bootstrap.ignore_system_bootstrap_checks to our elasticsearch.yml for 5.0 or whether we should instead document the workarounds (which is odd given that the default jvm.options has unmatching values for -Xms and -Xmx).

Answer 37 · 2016-08-19T02:28:27.000Z

See also: elastic/elasticsearch#17974 + #105

Answer 38 · 2016-08-25T12:49:16.000Z

Got this docker compose file working with 5.0.0-alpha5 . It has 1 master, 1 data node and kibana.

https://gist.github.com/shavo007/7f54f503fa7480dbabde753847cb7d71

Answer 39 · 2016-09-13T21:05:12.000Z

I am running this using Redhat on EC2 on alpha5 doing the rpm install.
So I've set /etc/elasticsearch/jvm.options (2g for min/max) and service elasticsearch restart with the same error:

[2016-09-13 16:55:19,731][WARN ][bootstrap                ] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupError: java.lang.RuntimeException: bootstrap checks failed
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

I also manually exported the option:
export ES_JAVA_OPTS="-Xms2g -Xmx2g

Still no luck.

Answer 40 · 2016-09-14T05:21:37.000Z

@carnalim Please read my post above. You need to set max_map_count like this apart from setting Java opts:

sysctl -w vm.max_map_count=262144

Answer 41 · 2016-09-14T07:27:25.000Z

@f3rdy how can I do this with docker-compose ?

Answer 42 · 2016-09-14T13:56:04.000Z

@somuraijoe Please read above, your docker host needs to set this. Quoting myself:

Generally, the host (linux/unix like OS, also MacOS) must support a higher amount of memory map areas by setting something like: sysctl -w vm.max_map_count=262144. As Elastic describes it on their site this is a necessary setting.

Answer 43 · 2016-09-29T10:17:06.000Z

I have a bit different setup, since I am using a Dockerfile invoking the Elastic Dockerfile FROM elasticsearch:5.0 plus, I am using ansible to deploy it.
I thought @tianon solution to set the environment variable would work, but it didn't sadly, even though the container is getting the variables values.

Use jvm.options or ES_JAVA_OPTS to configure the JVM
ES_HEAP_SIZE=1g: set -Xms1g and -Xmx1g in jvm.options or add "-Xms1g -Xmx1g" to ES_JAVA_OPTS

"Env": [
                "ES_HEAP_SIZE=1g",
                "ES_JAVA_OPTS=-Xms1g -Xmx1g",

I think it has to be in the "root" Dockerfile maybe, but since I wanted the heapsize and the xms to be configurable on deploy time, I haven't been able to do it yet.

What it worked for me before, was adding the jvm.options file to the config directory and rebuild the image with it.

Answer 44 · 2016-10-14T12:45:44.000Z

Test elasticsearch rc1 / apha5
It work with elasticsearch apha5

  elasticsearch: 
    image: elasticsearch:5.0.0-alpha5
    command: "-E bootstrap.ignore_system_bootstrap_checks=true"
    environment:
      ES_JAVA_OPTS: "-Xms2g -Xmx2g"

No error 👍

It not work with elasticsearch rc1
With option : bootstrap.ignore_system_bootstrap_checks

  elasticsearch: 
    image: elasticsearch:5.0.0-rc1
    command: "-E bootstrap.ignore_system_bootstrap_checks=true"
    environment:
      ES_JAVA_OPTS: "-Xms2g -Xmx2g"

Error 👎 :

elasticsearch_1  | org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [bootstrap.ignore_system_bootstrap_checks] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
elasticsearch_1  |  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:116) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:103) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  | Caused by: java.lang.IllegalArgumentException: unknown setting [bootstrap.ignore_system_bootstrap_checks] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
elasticsearch_1  |  at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:271) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:239) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.node.Node.<init>(Node.java:220) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:191) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]
elasticsearch_1  |  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112) ~[elasticsearch-5.0.0-rc1.jar:5.0.0-rc1]

Without option bootstrap.ignore_system_bootstrap_checks

  elasticsearch: 
    image: elasticsearch:5.0.0-rc1
    environment:
      ES_JAVA_OPTS: "-Xms2g -Xmx2g"

Error 👎 :

elasticsearch_1  | max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
elasticsearch_1  | [2016-10-14T12:32:29,385][INFO ][o.e.n.Node               ] [F3rMIvp] stopping ...
elasticsearch_1  | [2016-10-14T12:32:29,443][INFO ][o.e.n.Node               ] [F3rMIvp] stopped
elasticsearch_1  | [2016-10-14T12:32:29,445][INFO ][o.e.n.Node               ] [F3rMIvp] closing ...
elasticsearch_1  | [2016-10-14T12:32:29,461][INFO ][o.e.n.Node               ] [F3rMIvp] closed

I make this test with instance aws : t2.medium

Answer 45 · 2016-10-14T12:51:02.000Z

It is working for me with the RC1 and adding the xms. I removed the bootstrap ignore command and also removed the jvm.options file that I have added for testing.

    env:
      ES_JAVA_OPTS: "-Xms1g -Xmx1g"

Answer 46 · 2016-10-17T19:36:35.000Z

@pablogrs for me this docker-compose file does not work

version: '2'
services:
  elasticsearch: 
    image: elasticsearch:5.0.0-rc1
    environment:
      ES_JAVA_OPTS: "-Xms1g -Xmx1g"

conf

Docker-compose version 1.8.0, build f3628c7
Docker version 1.12.1, build 23cf638
Debian/jessie

Answer 47 · 2016-10-17T20:52:04.000Z

@vincent-herlemont, What error did you get in the logs? I'm guessing you still need to set vm.max_map_count on the docker host machine.

$ sysctl -w vm.max_map_count=262144

Answer 48 · 2016-10-18T12:13:55.000Z

@vincent-herlemont I didn't have to use sysctl -w vm.max_map_count=262144 but I am not using docker-compile, I am using Ansible.

- name: elasticsearch docker container 
  docker:
    name: elasticsearch
    image: "elastic5:latest" # I have a custom dockerfile because of proxy and other bits
    restart_policy: always
    entrypoint: "/docker-entrypoint.sh"
    command: "-E http.port=9200 -E cluster.name=my_cluster"
    volumes:
    - "/opt/elasticsearch:/usr/share/elasticsearch/data"
    - "/opt/elasticsearch/config:/usr/share/elasticsearch/config"
    ports:
    - "9200:9200"
    - "9300:9300"
    state: reloaded
    env:
      ES_JAVA_OPTS: "-Xms1g -Xmx1g"
    log_driver: "json-file"

Answer 49 · 2016-10-18T22:18:16.000Z

@pablogrs I guess your host must've already had an acceptable value 😄

I just tested, and -e ES_JAVA_OPTS='-Xms1g -Xmx1g' by itself wasn't enough, but was sufficient after I ran sysctl -w vm.max_map_count=262144 on the host. 👍

Answer 50 · 2016-10-18T22:19:35.000Z

Oh actually, it looks like the later releases fixed jvm.options too, so I don't even need ES_JAVA_OPTS anymore! Setting sysctl -w vm.max_map_count=262144 on the host is enough with default arguments otherwise!

Answer 51 · 2016-10-27T09:14:19.000Z

@tianon ,
I'm sorry for commenting in a closed issue. But I'm using Docker Toolbox on Windows. I can set the sysctl -w vm.max_map_count=262144 but the setting is reset each time I restart my Docker host machine. Is there any way to deal with it? :(

Answer 52 · 2016-10-31T13:51:28.000Z

bootstrap.ignore_system_bootstrap_checks doesn't exist in 5.0.0.

[2016-10-31T13:50:41,262][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [bootstrap.ignore_system_bootstrap_checks] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:116) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:103) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73) ~[elasticsearch-5.0.0.jar:5.0.0]
Caused by: java.lang.IllegalArgumentException: unknown setting [bootstrap.ignore_system_bootstrap_checks] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
    at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:271) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:239) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.node.Node.<init>(Node.java:220) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:191) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286) ~[elasticsearch-5.0.0.jar:5.0.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112) ~[elasticsearch-5.0.0.jar:5.0.0]
    ... 6 more

Answer 53 · 2016-11-01T16:34:23.000Z

Increase max_map_count on the Host System (Pls. note not in the Docker) using the following command
sudo sysctl -w vm.max_map_count=262144.

I have added this to the Dockerfile itself which has Fixed the Problem

Answer 54 · 2016-11-01T16:47:22.000Z

@nguoianphu with Docker Toolbox, your VM is likely boot2docker, so you'll want to add it to /var/lib/boot2docker/profile so that it runs at boot; if you're using Docker for Mac or Docker for Windows, I believe their settings have a place to add additional sysctl values for the VM host 👍

Answer 55 · 2016-11-19T04:31:44.000Z

hey, folks. just chiming an opinion in.

Elasticsearch is (or was) and extremely easy storage setup. No weird knowledge required. I'd highly recommend someone from elasticsearch trying to figure out a pragmatic update to ES itself which allows it to work with "listen all" and without require host modifications. Otherwise, downstream users, especially those with limited experience will perceive Elasticsearch as expert-only and/or fail. Being able to quickly start without wizard skills is probably a part of ES' success and something that it should be able to retain.

Answer 56 · 2016-11-29T19:52:02.000Z

I also noticed that a few of the options (e.g. default.http.port) are also missing from 5.0. Do we have a changelog on those changes?

Answer 57 · 2016-11-29T23:00:16.000Z

@nubunto I think https://www.elastic.co/guide/en/elasticsearch/reference/5.0/es-release-notes.html might be what you're looking for.

https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking-changes-5.0.html is probably also useful for what you're looking for.

Answer 58 · 2016-12-12T10:37:39.000Z

sysctl -w vm.max_map_count=262144

Worked for me, thank you very much.

First try: (Failure)

elasticsearch | [2016-12-12T10:31:36,346][INFO ][o.e.n.Node ] [Fovo3av] starting ...
elasticsearch | [2016-12-12T10:31:36,730][INFO ][o.e.t.TransportService ] [Fovo3av] publish_address {172.19.0.3:9300}, bound_addresses {[::]:9300}
elasticsearch | [2016-12-12T10:31:36,736][INFO ][o.e.b.BootstrapCheck ] [Fovo3av] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elasticsearch | ERROR: bootstrap checks failed
elasticsearch | max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
elasticsearch | [2016-12-12T10:31:36,777][INFO ][o.e.n.Node ] [Fovo3av] stopping ...

Second try after updating vm.max_map_count: (Success)

elasticsearch | [2016-12-12T10:34:54,902][INFO ][o.e.n.Node ] [Fovo3av] initialized
elasticsearch | [2016-12-12T10:34:54,909][INFO ][o.e.n.Node ] [Fovo3av] starting ...
elasticsearch | [2016-12-12T10:34:55,368][INFO ][o.e.t.TransportService ] [Fovo3av] publish_address {172.19.0.3:9300}, bound_addresses {[::]:9300}
elasticsearch | [2016-12-12T10:34:55,374][INFO ][o.e.b.BootstrapCheck ] [Fovo3av] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elasticsearch | [2016-12-12T10:34:56,978][WARN ][o.e.m.j.JvmGcMonitorService] [Fovo3av] [gc][young][1][2] duration [1.2s], collections [1]/[1.9s], total [1.2s]/[1.8s], memory [217.2mb]->[74.1mb]/[1.9gb], all_pools {[young] [198.3mb]->[36.4mb]/[266.2mb]}{[survivor] [18.9mb]->[33.1mb]/[33.2mb]}{[old] [0b]->[8.1mb]/[1.6gb]}

Answer 59 · 2016-12-12T15:27:05.000Z

This is still not solved...

For those who "solve" this by modifying host settings: seriously? What is the point of running this in a container if you have to modify host settings?

As of right now the master is completely unusable with docker-compose. Docker compose doesn't allow one to specify sysctl values, and this ridiculous requirement on the ES part is making this a no-go.

Answer 60 · 2016-12-12T17:24:14.000Z

@least-olegs the value in question has to be modified on the host -- it is not scoped to any namespace, and Elasticsearch 5.x is more aggressive about enforcing it than previous versions where. There's not really anything further we can do from the Dockerization itself to solve this.

Answer 61 · 2016-12-13T07:57:19.000Z

There's not really anything further we can do from the Dockerization itself to solve this.

this is the key sentence. nothing to do in this repo at the moment. If a core code change to elasticsearch occurred, which would prevent the need of host modification (at the expense of performance or some other tradeoff), then we'd change this repo to enable that feature. Until then, nothing to do.

Answer 62 · 2016-12-15T11:32:14.000Z

@tianon you are plain wrong. For these reasons:

This value is mindless nonsense and is not a real requirement. Some idiot in Elasticsearch discovered there was a setting they didn't really understand the purpose of, and decided to make it mandatory.
And, if you cannot do anything: you cannot close the ticket. Just go home and do something else. Your code doesn't work and you have no way of making it work, so how is the problem solved if it isn't?
But, in fact, you can make it work! Just don't use docker-compose, or roll your own version of docker-compose. The problem is not in the Docker, however crappy that program is. The problem is simply that you cannot specify a configuration parameter through the configuration tool that you chose. The answer? - ditch / modify your tool. It's that simple.

Answer 63 · 2016-12-15T12:10:46.000Z

@least-olegs FYI you're in a docker repo complaining about ES's new requirement to a docker person.

Answer 64 · 2016-12-19T13:57:20.000Z

I've just reproduced on elasticsearch:5 and elasticsearch:5.1.

Could anyone please explain why is it closed? Host modification can not be a way to run a container.

Answer 65 · 2016-12-19T17:27:05.000Z

Upstream's startup checks in 5.x are now more explicit about the host environment settings. There is nothing we can do from the image to configure those for you, which is why the image documentation now explicitly mentions configuring those bits on your host yourself. The settings in question are not namespaced to containers, and thus cannot be adjusted by a container, or by a docker run flag, and must be set on the host. There is nothing else we, the Docker image maintainers, can do here.

Answer 66 · 2016-12-19T18:07:54.000Z

See also elastic/elasticsearch#4978 (comment) for some notes from upstream on why they've made this change:

As of 5.0, Elasticsearch will not start in production mode if vm.max_map_count is not high enough. Silencing the warning when we are not able to apply the MAX_MAP_COUNT setting on openvz will just make debugging the issue harder, as it will not be obvious why the setting is not being applied. Instead, if you are running on a system where you cannot set vm.max_map_count, but it is set to be high enough for Elasticsearch's bootstrap checks, then you can silence the warning by removing the MAX_MAP_COUNT setting. If the value on your system is NOT high enough, then your cluster is going to crash and burn at some stage and you will lose data. Instead of trying to work around it with hacks like #4978 (comment), you should either speak to your sysadmin to configure vm.max_map_count correctly, or move to a platform where you can set it.

Answer 67 · 2017-01-11T17:14:16.000Z

See #153 for a possible pending fix to this issue (essentially no longer having the image in "production" mode by default). 👍

Answer 68 · 2017-01-11T22:55:25.000Z

We have merged a fix in #153 so that elasticsearch will run anywhere by default (by not needing to run the bootstrap checks). The minimal set to make it work for clustering is the following:

$ docker run -d --name elas elasticsearch -Etransport.host=0.0.0.0 -Ediscovery.zen.minimum_master_nodes=1
$ # note, running it with these flags would require the bootstrap checks to pass (vm.max_map_count, etc)

These same configs could also be put in a custom config file to replace what is currently there.

This will be available on the Docker Hub once a PR to official-images has been made and merged. (hopefully later today).