latest 11-jdk on ubuntu jammy breaks keytool -importcert

Question

latest 11-jdk on ubuntu jammy breaks keytool -importcert

gmolkvk opened this issue 2 years ago · 13 comments

I am using your images by using maven:3.8.5-eclipse-temurin-11 - see https://github.com/carlossg/docker-maven/blob/master/eclipse-temurin-11/Dockerfile#L1

Adding trusted certificates to the java truststore is failing on the new ubuntu jammy image

Using :11-jdk-focal

FROM eclipse-temurin:11-jdk-focal

RUN keytool -importcert -noprompt -file /usr/local/share/ca-certificates/extra/rootca.crt -cacerts -storepass changeit -alias 'custom root ca'
Certificate was added to keystore

Using :11-jdk

FROM eclipse-temurin:11-jdk

RUN keytool -importcert -noprompt -file /usr/local/share/ca-certificates/extra/rootca.crt -cacerts -storepass changeit -alias 'custom root ca'
[0.004s][warning][os,thread] Failed to start thread - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# //hs_err_pid6.log
The command '/bin/sh -c keytool -importcert -noprompt -file /usr/local/share/ca-certificates/extra/rootca.crt -cacerts -storepass changeit -alias 'custom root ca'' returned a non-zero code: 1

The only statement changed between builds is the Dockerfile FROM statement

Answer 1 · 2022-05-27T13:38:29.000Z

eclipse-temurin:17-jdk same problem

Answer 2 · 2022-05-27T15:25:26.000Z

the change to jammy was done in docker-library/official-images#12516

Answer 3 · 2022-05-30T11:34:13.000Z

I think if your docker engine is upgraded to the latest that resolves this issue. Can folks give that a try?

Answer 4 · 2022-05-30T12:33:07.000Z

Will try and let you know how it goes

Answer 5 · 2022-05-30T14:21:23.000Z

I can confirm that updating to docker-engine 20.10.16 resolved the issue for me.

Answer 6 · 2022-05-31T12:05:17.000Z

Hi folks! I had a similar issue, and your discussions here and there helped me to investigate. So I figured I could share my findings:

the change to jammy was done in docker-library/official-images#12516

Yes. All eclipse-temurin:<version>-jdk became based on the latest ubuntu LTS 22.04 Jammy.
You have to specify eclipse-temurin:<version>-jdk-focal if you want to use the image with the previous ubuntu LTS 20.04 Focal as a base image.

I can confirm that updating to docker-engine 20.10.16 resolved the issue for me.

This is probably due to this PR on docker.

Latest glibc will attempt to use clone3(). As a result, most newer distro (ubuntu Jammy 22.04, but probably others) will fail unless we allow the syscall (in docker, in systemd...).

To reproduce:

$ wget "https://raw.githubusercontent.com/moby/moby/c7cd1b9436ac381747a5c52dddac5a66f97c61f8/profiles/seccomp/default.json" -O before_clone3.json
 # the seccomp profile right before PR https://github.com/moby/moby/commit/9f6b562dd12ef7b1f9e2f8e6f2ab6477790a6594

docker run --security-opt seccomp=before_clone3.json -it --entrypoint /bin/bash eclipse-temurin:17-jdk # (should also fail with any ubuntu 22.04 really...)
root@e2a511de29d4:/# curl google.com
curl: (6) getaddrinfo() thread failed to start
root@e2a511de29d4:/#

Permanent fix:

upgrade docker, or expect all newer, up-to-date base images to fail in the future

Mitigation/workaround:

docker run --security-opt seccomp=your_policy_which_allows_clone3.json .... (docs)
downgrade your (base) images (the -focal should stay available for some 2years, IF the maintainers of docker's official images wait for the next ubuntu LTS to drop ubuntu Focal/20.04)

Answer 7 · 2022-05-31T14:49:30.000Z

Can someone clarify why updating docker fixes this? Thanks!

Answer 8 · 2022-06-01T09:20:09.000Z

Can someone clarify why updating docker fixes this? Thanks!

#215 (comment) explains it.

Answer 9 · 2022-10-14T19:03:40.000Z

Hi folks! I had a similar issue, and your discussions here and there helped me to investigate. So I figured I could share my findings:

the change to jammy was done in docker-library/official-images#12516

Yes. All eclipse-temurin:<version>-jdk became based on the latest ubuntu LTS 22.04 Jammy. You have to specify eclipse-temurin:<version>-jdk-focal if you want to use the image with the previous ubuntu LTS 20.04 Focal as a base image.

I can confirm that updating to docker-engine 20.10.16 resolved the issue for me.

This is probably due to this PR on docker.

Latest glibc will attempt to use clone3(). As a result, most newer distro (ubuntu Jammy 22.04, but probably others) will fail unless we allow the syscall (in docker, in systemd...).

To reproduce:
$ wget "https://raw.githubusercontent.com/moby/moby/c7cd1b9436ac381747a5c52dddac5a66f97c61f8/profiles/seccomp/default.json" -O before_clone3.json
 # the seccomp profile right before PR https://github.com/moby/moby/commit/9f6b562dd12ef7b1f9e2f8e6f2ab6477790a6594

docker run --security-opt seccomp=before_clone3.json -it --entrypoint /bin/bash eclipse-temurin:17-jdk # (should also fail with any ubuntu 22.04 really...)
root@e2a511de29d4:/# curl google.com
curl: (6) getaddrinfo() thread failed to start
root@e2a511de29d4:/#
Permanent fix:

upgrade docker, or expect all newer, up-to-date base images to fail in the future

Mitigation/workaround:

docker run --security-opt seccomp=your_policy_which_allows_clone3.json .... (docs)

downgrade your (base) images (the -focal should stay available for some 2years, IF the maintainers of docker's official images wait for the next ubuntu LTS to drop ubuntu Focal/20.04)

This is excellent research! To expand on this just a little bit:

glibc 2.34 and newer contain this commit, which defaults to using clone3. Ubuntu Jammy ships with 2.35, and thus contains this change in behavior. This was not available for Docker until PR 42681 (fix for 42680). Which is available starting with version 20.10.10.

Answer 10 · 2022-10-14T19:21:42.000Z

Just out of curiosity, has anyone had success with the seccomp workaround?

Answer 11 · 2022-11-07T15:25:41.000Z

@keeganwitt I have tried it, but it seems that you cannot override the base seccom profile (this is hard-coded). Here is the relevant change in Docker moby/moby#42681

Answer 12 · 2023-08-25T07:28:00.000Z

@keeganwitt I have tried it, but it seems that you cannot override the base seccom profile (this is hard-coded). Here is the relevant change in Docker moby/moby#42681

I also tried it.
Using the lastest default.json or the default.json after the commit, both of them failed.

# the latest file
wget https://raw.githubusercontent.com/moby/moby/master/profiles/seccomp/default.json -O clone3_default.json

# or the fixed issue committed file
wget https://raw.githubusercontent.com/berrange/moby/9f6b562dd12ef7b1f9e2f8e6f2ab6477790a6594/profiles/seccomp/default.json -O clone3_default.json

cat clone3_default.json | grep clone3

docker run --security-opt seccomp=clone3_default.json -it --entrypoint /bin/bash eclipse-temurin:8u362-b09-jdk-jammy@sha256:8f16a677c2270ba982f998b8eb3869a4bfbe0aa385ebf786d33ce23d3e4ff3bc

curl www.baidu.com

My docker is binary linux 20.10.9

Answer 13 · 2024-01-30T18:11:16.000Z

If you cannot update Docker a workaround is described here (Option 3): https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0#faccessat2