Live Minecraft Zero Day security issue! Mitigation possible via startup flag
Closed this issue Β· 50 comments
According to many sources there is a currently active zero day vulnerability. See more here:
https://arstechnica.com/information-technology/2021/12/minecraft-and-other-apps-face-serious-threat-from-new-code-execution-bug/
Attackers are already scanning the net for vulnerable servers.
The issue leads to remote code execution on the server!
According to many sources the JVM flag -Dlog4j2.formatMsgNoLookups=true mitigates this issue.
I suggest quickly adding this flag as default for all itzg /docker-minecraft-server versions and installations.
Edit:
Apparently Fabric released loader version 0.12.9 with a fix.
Edit2:
CVE: CVE-2021-44228
https://www.randori.com/blog/cve-2021-44228/
apparently, this fix only works for 1.17 servers if I understand this correctly:
https://www.minecraft.net/en-us/article/important-message--security-vulnerability-java-edition
Im not sure this is ready to close @itzg
I didn't mean to close it. It got auto-closed with the PR.
...however, beyond the mitigation included in #1178 I need advice on what all I can do.
I think it might make sense to also pin this issue to make sure people find the correct information.
FWIW, the change has been merged and published to all the variants and tagged at the respective 2021.23.0 for each variant.
I'll just drop some information here for people who might be interested:
Minecraft blog post with Mitigationβ:
https://www.minecraft.net/en-us/article/important-message--security-vulnerability-java-edition
Description of the exploit:
https://www.lunasec.io/docs/blog/log4j-zero-day/
Github Advisory with more links:
GHSA-jfh8-c2jp-5v3q
In the meantime, I would encourage everyone to do the following:
1.18.1: You are safe.
1.17.x - 1.18: Make sure you run the latest version of the docker. You should be safe with that.
1.7 - 1.16.5: Take them offline or make sure they are not accessible from the internet. It has apparently been confirmed that bots are already scanning for vulnerable servers. outdated. Now same as 1.17.x - 1.18:
According to https://www.minecraft.net/en-us/article/important-message--security-vulnerability-java-edition there are workaround for v1.7-1.11.2 & v1.12-1.16.5 too. Could we get those added? Thanks!
@StoneLabs and @sheepa Thank you for sharing that. Yes, I'll work on getting that added.
Additionally Paper requires special xml files, for example paper 1.11.2 wont boot with the vanilla xml files.
I've pasted the discord message below, I don't know if you want to wait for a more official release but thought I'd let you know.
1. Download the following file (for your version) and place it in the main server directory.
2. Run server with -Dlog4j.configurationFile=log4j2.xml flag (before -jar)
1.8-1.9: https://cdn.discordapp.com/attachments/722785635881975910/918816545705189386/log4j2.xml
1.10: https://cdn.discordapp.com/attachments/722785635881975910/918816283414384670/log4j2.xml
1.11: https://cdn.discordapp.com/attachments/722785635881975910/918816627636731914/log4j2.xml
1.12: https://cdn.discordapp.com/attachments/722785635881975910/918816730061606912/log4j2.xml
1.13: https://cdn.discordapp.com/attachments/722785635881975910/918816806624440380/log4j2.xml
1.14: https://cdn.discordapp.com/attachments/722785635881975910/918816971800334396/log4j2.xml
1.15: https://cdn.discordapp.com/attachments/722785635881975910/918817030512205874/log4j2.xml
Thanks @sheepa . Yeah, I'm thinking I would prefer waiting for an official file location; however, I will adjust these changes I'm doing now to avoid disruption of Paper. Do you know if the vanilla configs are fine for Forge and non-Paper variants in the Bukkit family?
@itzg Unrelated but fabric 1.17-1.18 is fixed as long as loader version 1.12.9+ is used.
FabricMC/fabric-loader@7f8086f
I personally suspect Mojang will release fixed server jar files soon as the command line arguments are only not really long-term solutions.
Additionally, i've seen some people say that
This exploit does not work on Java 8 and above by default because of this change: https://www.oracle.com/java/technolog... (see "Improved protection for JNDI remote class loading"). This means you are most likely safe from code being executed on your machine.
which would ofc be great if it is true.
Yeah that would be awesome and pretty much make this whole thing a non-issue since Java 8+ has been the norm (other than the Java 7 variant I provide).
In any case, updated images will be ready soon: #1179
Thanks @sheepa . Yeah, I'm thinking I would prefer waiting for an official file location; however, I will adjust these changes I'm doing now to avoid disruption of Paper. Do you know if the vanilla configs are fine for Forge and non-Paper variants in the Bukkit family?
I have not tested all but for forge the vanilla ones at least let you start the server, unsure about protection. Spigot doesnt work but have released updated jars for buildtools.
Additionally, i've seen some people say that
This exploit does not work on Java 8 and above by default because of this change: https://www.oracle.com/java/technolog... (see "Improved protection for JNDI remote class loading"). This means you are most likely safe from code being executed on your machine.
which would ofc be great if it is true.
That would be awesome indeed, just odd that there is not mention of that on the official minecraft.net article. Is this confirmed?
Merges, build, and tag at 2021.23.1 are on the way.
The proposed fixes by mojang have been implemented in #1179.
Vanilla should be fine now. Things might be different for non-vanilla servers so take care!
Help wanted though: If we missed something please let us know!
Latest images are now published to include Mojang's published fix. They have also been tagged with their respective tag at 2021.23.1
Do the proxy images need updating too? https://github.com/itzg/docker-bungeecord
Do the proxy images need updating too? https://github.com/itzg/docker-bungeecord
Is there any specific guidance from bungee and the others? As mentioned in Discord, any software updates on their part can be picked up by restarting the container.
This might be a stupidly simple question, for which I apologise in advance, but I'm the sort of docker user that can do simple stuff like docker run and exec, and following "how-to"s for complicated things, but doesn't really know the intricacies of how it all works ... so... how do I upgrade an existing installation so as the fixes are installed? Thanks!
@irrelevantdotcom This guide should help you.
https://phoenixnap.com/kb/update-docker-image-container
Ofc, if you use some GUI to manage your images/containers (e.g. your NAS's web interface, portainer, etc..) then you should do it through gui.
Thank you for the reply, but that guide appears to boil down to "docker rm" your existing container and "docker run" a new one (using a latest pull) with the same name ... this sounds dangerous to the data within....
Edit: looks like I started the running copy it with a local mapping for /data, so it's possibly safe... I am still terrified though!!
Thank you for the reply, but that guide appears to boil down to "docker rm" your existing container and "docker run" a new one (using a latest pull) with the same name ... this sounds dangerous to the data within....
Oh, that's fine. That's why any persistent data is saved on mapped paths. These are not part of the container.
That means, even if you delete the Minecraft server container, the data is still on your drive because it was never saved inside the container. You can then simply start a new container with the dame /data mapping and it will be like nothing ever changed.
If you're still scared, you can always make a backup of the data from your /data mount before removing the container. :)
And if you didn't use mapped volumes in the first place then that's what this section is for
https://github.com/itzg/docker-minecraft-server#converting-anonymous-data-volume-to-named-volume
Thank you both; I'll do backups, and try that later... (servers are shut down for now.)
@itzg thank you for your prompt attention regarding this. I can confirm, by using the tool created by Huntress
Weβve created a tool to help you test whether your applications are vulnerable to CVE-2021-44228.
You can access the tool here: https://log4shell.huntress.com/
that pulling itzg/minecraft-server:latest appears to have patched my system. I ran a test before upgrading and was found to be vulnerable. After pulling the latest image and restarting Minecraft Launcher the test did not log any further entries.
Edit: As an additional test, I took my client back to 1.18 and tried to exploit and there were still no entries logged showing potential vulnerability on the server.
Additionally, i've seen some people say that
This exploit does not work on Java 8 and above by default because of this change: https://www.oracle.com/java/technolog... (see "Improved protection for JNDI remote class loading"). This means you are most likely safe from code being executed on your machine.
which would ofc be great if it is true.
We mustn't assume this is the case, unfortunately. Though some reputable sources were saying only early Java 8 versions are susceptible to RCE - and it is true that a vulnerability was mitigated then - alternate paths very likely exist and will be used.
Additionally, i've seen some people say that
This exploit does not work on Java 8 and above by default because of this change: https://www.oracle.com/java/technolog... (see "Improved protection for JNDI remote class loading"). This means you are most likely safe from code being executed on your machine.
which would ofc be great if it is true.
We mustn't assume this is the case, unfortunately. Though some reputable sources were saying only early Java 8 versions are susceptible to RCE - and it is true that a vulnerability was mitigated then - alternate paths very likely exist and will be used.
I agree. A tester would be nice? Do anyone know of a tester? If not I could try and create one.
Thanks @ebell451 !
@sheepa perhaps not as automated as you're thinking, but the posted tool https://log4shell.huntress.com/ might be part of the tooling solution.
Will this be coming to the multiarch-latest tag? I tried to pull the latest image but it keeps saying it's on the newest one (which is dated 2 months ago)
Will this be coming to the
multiarch-latesttag? I tried to pull the latest image but it keeps saying it's on the newest one (which is dated 2 months ago)
That tag is deprecated. Please switch to one of the active tags
https://github.com/itzg/docker-minecraft-server#deprecated-image-tags
Edit: Apparently Fabric released loader version 0.12.9 with a fix.
How do I force the fabric loader to be 1.12.9?
My workflow:
- Downloaded the newest image:
docker pull itzg/minecraft-server - Set the loader to be version
FABRIC_LOADER_VERSION=0.12.9in my docker-compose.yml - Restarted my server:
docker-compose up -d
In my log I still see fabricloader@0.11.6:
minecraft_1 | [15:11:58] [main/INFO]: [FabricLoader] Loading 4 mods:
minecraft_1 | - fabricloader@0.11.6
minecraft_1 | - java@17
minecraft_1 | - lithium@0.7.5
minecraft_1 | - minecraft@1.17.1
Any hint will be appreciated.
How do I force the fabric loader to be 1.12.9?
We've made a new issue for this. See #1191
@StoneLabs and @itzg , can you explain this?
In the meantime, I would encourage everyone to do the following: 1.18.1: You are safe.
Hmβ¦? Have a look at my latest itzg/minecraft-server container with 1.18.1 - and still an unsafe log4j-core-2.14.1.jar π’
π©πͺ β ~/bin [master|β]
23:54 # ll /var/lib/docker/volumes/137f5aa71947c350ec198ff1e9f4b8ebb27bb4d98e950ea6bab40d36a41319ef/_data/
.rw-rw-r-- 1000 1000 2 B 2 days ago ξ banned-ips.json
.rw-rw-r-- 1000 1000 2 B 2 days ago ξ banned-players.json
.rw-rw-r-- 1000 1000 68 B 2 days ago ο
eula.txt
drwxrwxr-x 1000 1000 4.0 KB 2 days ago ο libraries
drwxrwxr-x 1000 1000 4.0 KB 2 days ago ο logs
.rw-rw-r-- 1000 1000 44 MB 2 days ago ξ minecraft_server.1.18.1.jar
.rw-rw-r-- 1000 1000 2 B 2 days ago ξ ops.json
.rw-r--r-- 1000 1000 1.2 KB 2 days ago ξ server.properties
.rw-rw-r-- 1000 1000 2 B 2 days ago ξ usercache.json
drwxrwxr-x 1000 1000 4.0 KB 2 days ago ο versions
.rw-rw-r-- 1000 1000 2 B 2 days ago ξ whitelist.json
drwxrwxr-x 1000 1000 4.0 KB a day ago ο world
π©πͺ β ~/bin [master|β]
23:54 # ll /var/lib/docker/volumes/137f5aa71947c350ec198ff1e9f4b8ebb27bb4d98e950ea6bab40d36a41319ef/_data/libraries/org/apache/logging/log4j/log4j-core/2.14.1/log4j-core-2.14.1.jar
.rw-rw-r-- 1000 1000 1.7 MB 2 days ago ξ /var/lib/docker/volumes/137f5aa71947c350ec198ff1e9f4b8ebb27bb4d98e950ea6bab40d36a41319ef/_data/libraries/org/apache/logging/log4j/log4j-core/2.14.1/log4j-core-2.14.1.jar
π©πͺ β ~/bin [master|β]
23:54 # docker images | grep itzg
itzg/minecraft-server latest c6b43dc544b1 2 days ago 874MB
I also downloaded latest 1.18.1 Minecraft server jar directly from https://feedback.minecraft.net/hc/en-us/articles/4416161161101-Minecraft-Java-Edition-1-18-1-
And still found this unsafe log4j-core-2.14.1.jar π’
π β [63%] β
π Thomas.Merz@dm-C02CGH01MD6M:~/Downloads [ttys006/2824]
00:09 $ zip -sf server.jar | grep log4j-core
META-INF/libraries/org/apache/logging/log4j/log4j-core/
META-INF/libraries/org/apache/logging/log4j/log4j-core/2.14.1/
META-INF/libraries/org/apache/logging/log4j/log4j-core/2.14.1/log4j-core-2.14.1.jar
I opened https://bugs.mojang.com/browse/MC-245526 for this issue in 1.18.1
This is an interesting point. I assume that, despite the JAR having the unsafe version in its filename, either the JAR is patched (e.g. by removing or stubbing JdniLookup.class) or mitigations are applied at runtime.
I'm not sure how to make use of the fix. I did a docker-compose pull && docker-compose up -d and then I attached to my container to see what things looked like. I did a px faux | grep java to see what the java pid was and got 526. Then I did sudo -u minecraft cat /proc/526/environ which gave me:
root@9b4cbb397fe5:/data# sudo -u minecraft cat /proc/526/environ
MOTD=Endeavouring to EndeavourHOSTNAME=9b4cbb397fe5LANGUAGE=en_US:enLEVEL=ftb-endeavourJAVA_HOME=/opt/java/openjdkRCON_PASSWORD=minecraftORIGINAL_TYPE=FTBAAUTOPAUSE_PERIOD=10RCON_PORT=25575EULA=trueTYPE=FTBAPWD=/dataFTB_MODPACK_ID=80ONLINE_MODE=trueJVM_DD_OPTS=fml.readTimeout:240 fml.loginTimeout:900FAMILY=FORGEHOME=/dataLANG=en_US.UTF-8FTB_MODPACK_VERSION_ID=2094MODE=survivalWHITELIST=<REDACTED>PVP=falseALLOW_NETHER=trueALLOW_FLIGHT=trueTERM=xtermAUTOPAUSE_TIMEOUT_INIT=600ENABLE_RCON=trueGID=1000AUTOPAUSE_TIMEOUT_KN=120VERSIONS_JSON=https://launchermeta.mojang.com/mc/game/version_manifest.jsonSHLVL=0INIT_MEMORY=6GSERVER=forge-1.16.5-36.2.9.jarDIFFICULTY=easyAUTOPAUSE_KNOCK_INTERFACE=eth0LC_ALL=en_US.UTF-8PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binVERSION=LATESTMEMORY=1GVANILLA_VERSION=1.18.1UID=1000ENABLE_AUTOPAUSE=falseOLDPWD=/dataJAVA_VERSION=jdk8u312-b07MAX_MEMORY=10GAUTOPAUSE_TIMEOUT_EST=3600
There is no JVM_OPTS in the environment variables. Also the only log4j xml file I see on the system is /tmp/log4j2.xml
This is an FTB server (FAMILY=FORGE according to the latest patches). Any advice?
β¦ either the JAR is patched (e.g. by removing or stubbing
JdniLookup.class) or mitigations are applied at runtime.
Nope: JdniLookup.class is definitely NOT removed as my query showed already and once again:
for fs in $(mount|grep -E "ext4|overlay|xfs"|awk '{print $3}'); do
echo $fs:
find $fs/ -xdev -iname "*.jar" -or -iname "*.war" | while read line; do
if [ $( zip -sf "${line}" | grep org/apache/logging/log4j/core/lookup/**JndiLookup.class** ) ]; then
echo "${line}" | tee -a /var/log/CVE-2021-44228.log
fi
done
done
This shows /var/lib/docker/volumes/137f5aa71947c350ec198ff1e9f4b8ebb27bb4d98e950ea6bab40d36a41319ef/_data/libraries/org/apache/logging/log4j/log4j-core/2.14.1/log4j-core-2.14.1.jar which is my latest MC server docker container from @itzg π€·π»ββοΈ
And I don't see any mitigation, too, for my running MC server:
π©πͺ β ~ [master|β]
08:54 # pgrep -fla minecraft_server.1.18.1.jar
15711 mc-server-runner --stop-duration 60s --named-pipe /tmp/minecraft-console-in java -Xmx1G -Xms1G -jar minecraft_server.1.18.1.jar
16417 java -Xmx1G -Xms1G -jar minecraft_server.1.18.1.jar
π©πͺ β ~ [master|β]
08:54 # for pid in $(pgrep -fla minecraft_server.1.18.1.jar|awk '{print $1}'); do cat /proc/$pid/environ; done
HOSTNAME=763340d15a15LANGUAGE=en_US:enJAVA_HOME=/opt/java/openjdkRCON_PASSWORD=secretORIGINAL_TYPE=VANILLAAUTOPAUSE_PERIOD=10RCON_PORT=12345EULA=TRUETYPE=VANILLAPWD=/dataFAMILY=VANILLAHOME=/dataLANG=en_US.UTF-8AUTOPAUSE_TIMEOUT_INIT=600ENABLE_RCON=trueGID=1000AUTOPAUSE_TIMEOUT_KN=120VERSIONS_JSON=https://launchermeta.mojang.com/mc/game/version_manifest.jsonSHLVL=0MCRAM=1GSERVER=minecraft_server.1.18.1.jarAUTOPAUSE_KNOCK_INTERFACE=eth0LC_ALL=en_US.UTF-8PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binVERSION=LATESTMEMORY=1GVANILLA_VERSION=1.18.1UID=1000ENABLE_AUTOPAUSE=falseOLDPWD=/dataJAVA_VERSION=jdk-17.0.1+12AUTOPAUSE_TIMEOUT_EST=3600HOSTNAME=763340d15a15LANGUAGE=en_US:enJAVA_HOME=/opt/java/openjdkRCON_PASSWORD=secretORIGINAL_TYPE=VANILLAAUTOPAUSE_PERIOD=10RCON_PORT=12345EULA=TRUETYPE=VANILLAPWD=/dataFAMILY=VANILLAHOME=/dataLANG=en_US.UTF-8AUTOPAUSE_TIMEOUT_INIT=600ENABLE_RCON=trueGID=1000AUTOPAUSE_TIMEOUT_KN=120VERSIONS_JSON=https://launchermeta.mojang.com/mc/game/version_manifest.jsonSHLVL=0MCRAM=1GSERVER=minecraft_server.1.18.1.jarAUTOPAUSE_KNOCK_INTERFACE=eth0LC_ALL=en_US.UTF-8PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binVERSION=LATESTMEMORY=1GVANILLA_VERSION=1.18.1UID=1000ENABLE_AUTOPAUSE=falseOLDPWD=/dataJAVA_VERSION=jdk-17.0.1+12AUTOPAUSE_TIMEOUT_EST=3600π©πͺ β ~ [master|β]
08:55 #
I also downloaded latest 1.18.1 Minecraft server jar directly from https://feedback.minecraft.net/hc/en-us/articles/4416161161101-Minecraft-Java-Edition-1-18-1-
And still found this unsafe log4j-core-2.14.1.jar π’
Just because the jar is present, does not mean that the server itself is vulnerable to CVE-2021-44228. There are other methods beyond removing JndiLookup.class that mitigate the exploit. These can be applied at runtime.
Until a statement from Mojang proves us wrong, we should assume such mitigations were taken as explicitly stated in the blog post.
You can further verify this by simply attempting the exploit on an 1.18.1 vanilla server as seen below. Note here, that the server did at no time attempt to establish a connection to the dummy server. I've additionally verified this using packet analyzer tools.
On the topic of CVE-2021-4504:
It is, to my knowledge not known whether Minecraft servers are affected or not. However, as stated by apache:
the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations.
we should therefore assume it is not affected until Mojang says otherwise. Additionally, CVE-2021-4504 does not allow for RCE, only DoS.
I'm not sure how to make use of the fix.
Tip for next time: Use strings /proc/.../environ instead of cat /proc/.../environ. The output is much easier to read :).
But if I'm seeing this correctly, you are running mc version 1.18.1? If so, you are good. There is no need for arguments.
If not, please open a new issue, so we can solve this and keep things a little more organized.
Thank you @StoneLabs
If there are further concerns about the mitigations please contact Mojang or Apache.
I am considering locking this issue from further comments, so please respect what is in the power of this image to address and what is not.
Since I didn't see it listed as a potential mitigation in this thread, I'd like to propose one of the other mitigations which would be effective against CVE-2021-45046 (even if it is just a DoS) as well as the original 44228.
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
(This was an example, likely would have to be tweaked with a path)
Removing the entire affected class would ofc cause errors should it be used, but in all reality Minecraft, and indeed most software products, shouldn't be using it at all.
Alternatively, the new Log4j 2.16 has blocked use of JNDI entirely.
@itzg Appreciate the work you do here. I've tested all vanilla/snapshot/forge/spigot/bukkit/spigot/paper/etc jars and if you want I can share the logic for making most versions working securely, snapshots for example does work for the official xml files. forge has special xml files.
@itzg Appreciate the work you do here. I've tested all vanilla/snapshot/forge/spigot/bukkit/spigot/paper/etc jars and if you want I can share the logic for making most versions working securely, snapshots for example does work for the official xml files. forge has special xml files.
@sheepa I would prefer to rely on upstream fixes and not get in the business of custom mitigations; however, I would be interested in seeing a PR if it's not too much effort.
Has there been any triage attempted regarding the latest vulnerability found in log4j v2.16 that is patched in v2.17?
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105
Has there been any triage attempted regarding the latest vulnerability found in log4j v2.16 that is patched in v2.17? https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105
As far as my understanding goes, Minecraft servers are not affected by CVE-2021-45105.
This issue is stale because it has been open 30 days with no activity. Please add a comment describing the reason to keep this issue open.