elastic/elasticsearch

0-day in log4j package

Closed this issue · 47 comments

aSapien commented

Hi Elastic,

A 0-day exploit in log4j package has been published and it looks like ElasticSearch could be affected by a vulnerable version:

elasticsearch/build-tools-internal/version.properties

Lines 16 to 18 in 68836bb

# when updating log4j, please update also docs/java-api/index.asciidoc
log4j = 2.11.1
slf4j = 1.6.2

Vulnerability:
apache/logging-log4j2#608

Please look at it and advice on the best course of action to secure an elastic cluster and prevent compromise ASAP.

Thanks!

Deleted user commented

Duplicate of #81618
Mitigation in there as well.

  • 👍3
JerryGuos commented

We are currently using two versions, 5.5.2 and 7.7.1.
For log4j-2.10.x , I can add -Dlog4j2.formatMsgNoLookups=true to ES_JAVA_OPTS.
However, in version 5.5.2, the version is 2.8.2 and -Dlog4j2.formatMsgNoLookups=true is invalid.

  • 👍3
jaywilliams commented

Added this to my /etc/elasticsearch/jvm.options file to set the flag.

# CVE-2021-44228
-Dlog4j2.formatMsgNoLookups=true
  • 👍18
akx commented

Looks like #81622, #81623, #81624, #81625 are addressing this. :)

mehta-ankit commented

I see #81624 merged, does that mean this also release a new docker image here: https://www.docker.elastic.co/r/elasticsearch/elasticsearch
Or will that happen when a new release is cut for 7.16 (like 7.16.1) ? nvm i see the label 7.16.1 on the PR
image

jkowall commented

OpenSearch is patched, feel free to adopt :)

opensearch-project/OpenSearch#1698

  • 👍8
  • 👎11
  • 😄8
p-ob commented

We are currently using two versions, 5.5.2 and 7.7.1.
For log4j-2.10.x , I can add -Dlog4j2.formatMsgNoLookups=true to ES_JAVA_OPTS.
However, in version 5.5.2, the version is 2.8.2 and -Dlog4j2.formatMsgNoLookups=true is invalid.

@JerryGuos We have some 5.6.x Elastic infrastructure still running, and have dropped the resolved jar file in and removed the 2.8.x version (after stopping the service). Started the service afterwards and run some tests against it. This appears functional (albeit less-than-ideal).

ymmv though.

enoch85 commented

cc @Ark74, what about your Docker?

tanguofu commented

I see #81624 merged, does that mean this also release a new docker image here: https://www.docker.elastic.co/r/elasticsearch/elasticsearch Or will that happen when a new release is cut for 7.16 (like 7.16.1) ? nvm i see the label 7.16.1 on the PR image

is there any es v7.16.1 release plan ?

  • 👍3
t0klian commented

Some sources state that simply adding "-Dlog4j2.formatMsgNoLookups=true" is not an acceptable mitigation, because there are too many variables that may make it weak or ineffective.

When can we expect update of the log4j dependency for Elasticsearch? What versions will receive that update?

aSapien commented

@t0klian can you please provide links to these sources?

t0klian commented

@aSapien I've just re-phrased it to "some".

aSapien commented

@t0klian please share which variables other than log4j-core version might make the flag weak or ineffective, in which circumstances and according to which sources. If there is a bypass to the mitigation strategy we all need to know about it asap.

xuxiaowei-com-cn commented

我使用的 elasticsearch 版本是 7.13.3，安装方式：yum，log4j相关 jar 包是：log4j-api-2.11.1.jar、log4j-core-2.11.1.jar，位置是：/usr/share/elasticsearch/lib。
升级步骤：

  1. 将 /usr/share/elasticsearch/lib 中的 log4j-api-2.11.1.jar、log4j-core-2.11.1.jar 文件备份到其他文件夹后删除 /usr/share/elasticsearch/lib 中的 log4j-api-2.11.1.jar、log4j-core-2.11.1.jar。
  2. 重启 elasticsearch，发现报错：
[root@log lib]# systemctl status elasticsearch.service && journalctl -xe
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2021-12-13 08:25:50 CST; 16s ago
     Docs: https://www.elastic.co
  Process: 25607 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 25607 (code=exited, status=1/FAILURE)

Dec 13 08:25:50 log systemd-entrypoint[25607]: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/status/StatusListener
Dec 13 08:25:50 log systemd-entrypoint[25607]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:79)
Dec 13 08:25:50 log systemd-entrypoint[25607]: Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.status.StatusListener
Dec 13 08:25:50 log systemd-entrypoint[25607]:         at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:636)
Dec 13 08:25:50 log systemd-entrypoint[25607]:         at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
Dec 13 08:25:50 log systemd-entrypoint[25607]:         at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:519)
Dec 13 08:25:50 log systemd-entrypoint[25607]:         ... 1 more
Dec 13 08:25:50 log systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 08:25:50 log systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Dec 13 08:25:50 log systemd[1]: Failed to start Elasticsearch.
  1. 下载 log4j-api-2.16.0.jar、log4j-core-2.16.0.jar 并放到 /usr/share/elasticsearch/lib 文件夹中。
  2. 重启 elasticsearch 后正常。
sevenJustFine commented

你这两个jar包是哪里来的?版本变了没影响吗?不改其他的参数设置的情况下

xuxiaowei-com-cn commented

你这两个jar包是哪里来的?版本变了没影响吗?不改其他的参数设置的情况下

上面有网址呀

vCassius commented

jar报的下载已经不行了,官方撤回了?

sevenJustFine commented

jar报的下载已经不行了,官方撤回了?

https://logging.apache.org/log4j/2.x/download.html?spm=a2c4g.11174386.n2.3.37314c07isUvXE# 从这里下载

xuxiaowei-com-cn commented

apache官方把 2.15.1 删除了，发布了 2.16.0

vCassius commented

嗯嗯 谢谢 我也刚发现他发布了2.16 rc1

cuishuaigit commented

请问怎么验证这个问题被解决了呢？或者怎么复现

Ark74 commented

cc @Ark74, what about your Docker?

@enoch85 I'll review later today, thanks for the heads up.

  • 👍1
gastonsaid-ey commented

I made docker pull for Elasticsearch 7.16.1 and I still having problems with Log4j2 according with Aquasec scan.
Any other have this problem too?

xuxiaowei-com-cn commented

Elasticsearch 版本 7.16.1 已发布

FuviTeam commented

I've just upgraded to 7.16.1 in Ubuntu, and it's still the log4j-api-2.11.1.jar in the share lib ?
Capture_d_écran_13_12_2021_17_58

jheiselman commented

I've just upgraded to 7.16.1 in Ubuntu, and it's still the log4j-api-2.11.1.jar in the share lib ? Capture_d_écran_13_12_2021_17_58

The version of log4j2 wasn't bumped, but the out-of-box logging configuration was updated to take into account the recommended mitigation strategies.

https://www.elastic.co/guide/en/elasticsearch/reference/current/release-notes-7.16.1.html

  • 👍3
rdnm commented

For the latest updates on this issue, please refer to https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

Elastic's security reporting guidelines are available at https://www.elastic.co/community/security. Per those guidelines, all reports of potential security issues or vulnerabilities should be sent via email to security@elastic.co. We are unable to discuss potential issues of this nature here. Please send your report to the email address above, where it can be appropriately handled.

  • 👍1
xuxiaowei-com-cn commented

经过测试：

如果要升级 elasticsearch，需要将自己升级（调整）过的 jar 包还原。否则将无法重启。例如：
本人昨天已经将 elasticsearch 中的 log4j-api、log4j-core 升级至 2.16.0，今天将 elasticsearch 从 7.13.3 升级至 7.16.1 后，重启出现如下问题：

[2021-12-14T10:11:30,107][ERROR][o.e.b.Bootstrap          ] [log] Exception
java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.Logger
jar1: /usr/share/elasticsearch/lib/log4j-api-2.16.0.jar
jar2: /usr/share/elasticsearch/lib/log4j-api-2.11.1.jar
	at org.elasticsearch.jdk.JarHell.checkClass(JarHell.java:297) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:75) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.16.1.jar:7.16.1]
[2021-12-14T10:11:30,115][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [log] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.Logger
jar1: /usr/share/elasticsearch/lib/log4j-api-2.16.0.jar
jar2: /usr/share/elasticsearch/lib/log4j-api-2.11.1.jar
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.1.jar:7.16.1]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.Logger
jar1: /usr/share/elasticsearch/lib/log4j-api-2.16.0.jar
jar2: /usr/share/elasticsearch/lib/log4j-api-2.11.1.jar
	at org.elasticsearch.jdk.JarHell.checkClass(JarHell.java:297) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:75) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.1.jar:7.16.1]
	... 6 more

删除 自己添加的 /usr/share/elasticsearch/lib/log4j-api-2.16.0.jar 后，重启出现新错误：

[2021-12-14T10:14:44,477][ERROR][o.e.b.Bootstrap          ] [log] Exception
java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.core.appender.rewrite.MapRewritePolicy$Mode
jar1: /usr/share/elasticsearch/lib/elasticsearch-log4j-7.16.1.jar
jar2: /usr/share/elasticsearch/lib/log4j-core-2.16.0.jar
	at org.elasticsearch.jdk.JarHell.checkClass(JarHell.java:297) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:75) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.16.1.jar:7.16.1]
[2021-12-14T10:14:44,488][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [log] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.core.appender.rewrite.MapRewritePolicy$Mode
jar1: /usr/share/elasticsearch/lib/elasticsearch-log4j-7.16.1.jar
jar2: /usr/share/elasticsearch/lib/log4j-core-2.16.0.jar
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.1.jar:7.16.1]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.core.appender.rewrite.MapRewritePolicy$Mode
jar1: /usr/share/elasticsearch/lib/elasticsearch-log4j-7.16.1.jar
jar2: /usr/share/elasticsearch/lib/log4j-core-2.16.0.jar
	at org.elasticsearch.jdk.JarHell.checkClass(JarHell.java:297) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:75) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.1.jar:7.16.1]
	... 6 more

删除 自己添加的 /usr/share/elasticsearch/lib/log4j-core-2.16.0.jar 后重启正常。

通过对上述报错与 /usr/share/elasticsearch/lib/ 文件夹分析可知，elasticsearch-7.16.1 （CentOS 使用 yum 安装，并从 7.13.3 升级至 7.16.1）使用 log4j-api-2.11.1.jar，同楼上回复一致。并且 elasticsearch-7.16.1 使用了 elasticsearch-log4j-7.16.1.jar 替换了 log4j-core。

下图是阿里云对 elasticsearch-7.13.3 扫描的结果，仅提示了 log4j-core-2.11.1.jar 存在漏洞，并未说明 log4j-api-2.11.1.jar 是否有问题。
image

如果大家觉得 log4j-api-2.11.1.jar 可能存在风险，可使用 log4j-api-2.16.0.jar 将其替换（经过测试，暂未发现问题）。

另外说明：
在同时升级 elasticsearch、filebeat、kibana 至 7.16.1 后，需要将 kibana 在停止的状态下执行 curl -XDELETE http://localhost:9200/.kibana*，否则访问 kibana 后，页面仅显示 Kibana server is not ready yet

julichan commented

Hello, may you consider this bug incompletely resolved as 6.8.21 Docker version sill contain log4j-core-2.11.1.jar

MHenn1g commented

Hi @julichan indeed Log4j has not been removed in total.
They removed the vulnerable JndiLookup class from the Log4j package.

See this satement:
As of December 13, 2021, we have released Elasticsearch 6.8.21 and 7.16.1 which set the JVM option identified below and remove the vulnerable JndiLookup class from Log4j out of an abundance of caution.
(https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476)

So if you looked inside the jar in 6.8.20 and earlier...
jar tvf /usr/share/elasticsearch/lib/log4j-core-2.11.1.jar | grep JndiLookup.class

This class would show up
2937 Sun Jul 22 20:45:20 CEST 2018 org/apache/logging/log4j/core/lookup/JndiLookup.class

But now, there should be nothing to find

  • 👍3
julichan commented

Thanks for your quick reply

vishallalwani commented

As per the latest thread https://lists.apache.org/thread/83y7dx5xvn3h5290q1twn16tltolv88f, setting log4j.formatMsgNoLookups property to true does not mitigate the issue.

julichan commented

@vishallalwani , the variable log4j.formatMsgNoLookups alone does not mitigate it. But if the faulty class JNDILookup is removed from the jar, it does!

fishjam commented

I have try following method for my es 6.2.2:

  1. find all log4j jar files in your es folder ( include plugins and other folders)
  $ find ./ -type f | grep log4j.*jar$
  ./lib/log4j-core-2.9.1.jar
  ./lib/log4j-1.2-api-2.9.1.jar
  ./lib/log4j-api-2.9.1.jar
  ./plugins/search-guard-6/log4j-slf4j-impl-2.9.1.jar
  1. download log4j 2.16.0 jar files , and you can find all other files on apache
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-core/2.16.0/log4j-core-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-api/2.16.0/log4j-api-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-1.2-api/2.16.0/log4j-1.2-api-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-slf4j-impl/2.16.0/log4j-slf4j-impl-2.16.0.jar
  1. stop the es and backup your old log4j files
  mkdir ../es_bak_libs && find ./ -type f | grep log4j.*jar$ | xargs -i{} mv {} ../es_bak_libs/
  1. mv the log4j 2.16.0 jar to the right folder, then restart es , and it works
  2. lsof to confirm it, it use only log4j 2.16.0
    image

  • Now I want to know:

    • is this workable? can I fix the security issue by this method? how can I try to verify it?
    • because I don't want to upgrade my ES version, and want to solve this problem fundamentally without introducing new problems

  • Consider ES_CLASSPATH="$ES_HOME/lib/*" in elasticsearch-env , and es load all jar from it, it's version independent if it can load & run the function successful

[fishjam bin]$fgrep ES_CLASSPATH elasticsearch-env
ES_CLASSPATH="$ES_HOME/lib/*"
"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JavaVersionChecker

@kimchy seems you are es team, could you ask some body to check this solution? thank you very much.

  • 👍1
sandeepk-veritas commented

I tried solution suggested by @fishjam above for 7.14.1, but it is not working, ES does not startup properly.

[2021-12-15T14:02:22,454][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SK-FS-2K19] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.14.1.jar:7.14.1]
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2049) ~[?:?]
at java.lang.ClassLoader.getParent(ClassLoader.java:1799) ~[?:?]
at org.apache.logging.log4j.util.LoaderUtil.getClassLoaders(LoaderUtil.java:113) ~[log4j-api-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.getEventServices(WatchManager.java:160) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.(WatchManager.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.AbstractConfiguration.(AbstractConfiguration.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.DefaultConfiguration.(DefaultConfiguration.java:46) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:768) ~[log4j-core-2.16.0.jar:2.16.0]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.(EcsJsonLayout.java:47) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.createLayout(EcsJsonLayout.java:124) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout$Builder.build(EcsJsonLayout.java:150) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent.(DeprecationIndexingComponent.java:70) ~[?:?]
at org.elasticsearch.xpack.deprecation.Deprecation.createComponents(Deprecation.java:83) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:615) ~[elasticsearch-7.14.1.jar:7.14.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.(Node.java:619) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.node.Node.(Node.java:281) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.14.1.jar:7.14.1]
... 6 more

sandeepk-veritas commented

To add to my above comment, the same exact steps do work with 7.12.0 it seems. Unfortunately that is not an option for us because our application bundles 7.14.1 only and it is already out in market.
Seems like some issue with transient dependencies.

fishjam commented

reproduce the exception as sandeepk-veritas for 7.14.1.
After compare log4j-core 2.11 and 2.16, the WatchManager.getEventServices function is new added function, and maybe it's forbidden by xpack.
And my solution is not common solution for all ES version, so maybe you have to use the temporary measures and wait for official solution.

sandeepk-veritas commented

@fishjam thanks a lot for quick response. Yes, we will wait for official solution.

BillLucky commented

I have try following method for my es 6.2.2:

  1. find all log4j jar files in your es folder ( include plugins and other folders)
  $ find ./ -type f | grep log4j.*jar$
  ./lib/log4j-core-2.9.1.jar
  ./lib/log4j-1.2-api-2.9.1.jar
  ./lib/log4j-api-2.9.1.jar
  ./plugins/search-guard-6/log4j-slf4j-impl-2.9.1.jar
  1. download log4j 2.16.0 jar files , and you can find all other files on apache
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-core/2.16.0/log4j-core-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-api/2.16.0/log4j-api-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-1.2-api/2.16.0/log4j-1.2-api-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-slf4j-impl/2.16.0/log4j-slf4j-impl-2.16.0.jar
  1. stop the es and backup your old log4j files
  mkdir ../es_bak_libs && find ./ -type f | grep log4j.*jar$ | xargs -i{} mv {} ../es_bak_libs/
  1. mv the log4j 2.16.0 jar to the right folder, then restart es , and it works
  2. lsof to confirm it, it use only log4j 2.16.0
    image

  • Now I want to know:

    • is this workable? can I fix the security issue by this method? how can I try to verify it?
    • because I don't want to upgrade my ES version, and want to solve this problem fundamentally without introducing new problems

  • Consider ES_CLASSPATH="$ES_HOME/lib/*" in elasticsearch-env , and es load all jar from it, it's version independent if it can load & run the function successful

[fishjam bin]$fgrep ES_CLASSPATH elasticsearch-env
ES_CLASSPATH="$ES_HOME/lib/*"
"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JavaVersionChecker

@kimchy seems you are es team, could you ask some body to check this solution? thank you very much.

We also plan to use this solution to fix the CVE. Is it an official recommendation?

xeraa commented

There is an advisory for fixes on 5.0.0-5.6.10 and 6.0.0-6.3.2: https://discuss.elastic.co/t/elasticsearch-5-0-0-5-6-10-and-6-0-0-6-3-2-log4j-cve-2021-44228-cve-2021-45046-remediation/292054

PS: Don't drop in the latest Log4j JAR — it's not that simple as you can see in #47298. But we're working on a bigger fix to move to the latest version.

fishjam commented

after analyze the org.apache.logging.log4j.core.util.WatchManager.getEventServices function, consider there is already try...catch for each ServiceLoader , and the AccessControlException is foreseeable error scenarios , So I modify the code as following , and build & replace the WatchManager.class in log4j-core-2.16.0.jar, then it works.

Steps:

    1. add try catch for org.apache.logging.log4j.core.util.WatchManager.getEventServices and build the WatchManager.class
      image
    1. replace the WatchManager.class in es home path
       mkdir -p org/apache/logging/log4j/core/util
   cp WatchManager.class org/apache/logging/log4j/core/util/
   zip -u lib/log4j-core-2.16.0.jar org/apache/logging/log4j/core/util/WatchManager.class
    1. start es (7.14.1) and check, it's successful , but I'm not sure is there any other issue, you need do more check for you case.
    $ curl -X GET http://localhost:9200/_cat/health
1639617900 01:25:00 elasticsearch green 1 1 1 1 0 0 0 0 - 100.0%

log4j-core-2.16.0.jar.zip

In my opinion, the log4j security issue in #47298 should be fixed in log4j. I will try to create a PR for it.
but I'm not sure whethere is it acceptable and when they accept it.

sandeepk-veritas commented

With 7.14.1 and log4j2 2.16.0 jars in place, one option that I tried today is remove x-pack-deprecation and try to see if we ES is able to startup or not.
Found that it does work but not sure whether that is a possible solution to use. @xeraa any comments on it?

jianchen06 commented

There is an advisory for fixes on 5.0.0-5.6.10 and 6.0.0-6.3.2: https://discuss.elastic.co/t/elasticsearch-5-0-0-5-6-10-and-6-0-0-6-3-2-log4j-cve-2021-44228-cve-2021-45046-remediation/292054

PS: Don't drop in the latest Log4j JAR — it's not that simple as you can see in #47298. But we're working on a bigger fix to move to the latest version.

Hi @xeraa , I have a version 6.8.1 running on Linux. I don't see instructions in the official advisory on that version, other than upgarde ES. I've tried @fishjam 's method to replace the 2.11.1 jars with 2.16.0. Restart the elasticsearch service and it runs fine without any error. Should I go with that approach if I don't want to upgrade ES right now? Thanks.

fishjam commented

there is already LOG4J2-3236 for "java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")" yesterday.

xeraa commented

To close the loop here: Elasticsearch 7.16.2 and 6.8.22 are out and they use Log4j 2.17.0. This is the version you really want :)

  • 👍5
zdenkers commented

does anyone know when 7.16.2 will be published to docker hub?

xeraa commented

Looks like it has been merged (docker-library/elasticsearch@53dedd3), but I'm not sure what it will take now to appear.

In the meantime, you can always pull it from the Elastic repo: docker pull docker.elastic.co/elasticsearch/elasticsearch:7.16.2