Bug: JBoss Keycloack migrated from Dockerhub to Quay.io causing problems
matheusvnm opened this issue ยท 2 comments
Describe the bug
Recently, JBoss removed the Keycloak repository in DockerHub. The future Keycloack versions will be released on Quay.io, which can be seen here. Below stands the logs with errors related to the inexistent jboss/keycloack repository.
pytest -s
===================================================================================================== test session starts =====================================================================================================
platform darwin -- Python 3.9.6, pytest-7.4.3, pluggy-1.4.0
rootdir: /Users/smmarques/Repositorios/testcontainers-python
configfile: pyproject.toml
plugins: anyio-4.3.0, cov-4.1.0
collected 1 item
tests/test_keycloak.py::test_docker_run_keycloak[16.1.1] Pulling image jboss/keycloak:16.1.1
-------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------
INFO testcontainers.core.container:container.py:62 Pulling image jboss/keycloak:16.1.1
FAILED
========================================================================================================== FAILURES ===========================================================================================================
______________________________________________________________________________________________ test_docker_run_keycloak[16.1.1] _______________________________________________________________________________________________
../../venv/lib/python3.9/site-packages/docker/api/client.py:265: in _raise_for_status
response.raise_for_status()
../../venv/lib/python3.9/site-packages/requests/models.py:1021: in raise_for_status
raise HTTPError(http_error_msg, response=self)
E requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/create
The above exception was the direct cause of the following exception:
../../venv/lib/python3.9/site-packages/docker/models/containers.py:873: in run
container = self.create(image=image, command=command,
../../venv/lib/python3.9/site-packages/docker/models/containers.py:932: in create
resp = self.client.api.create_container(**create_kwargs)
../../venv/lib/python3.9/site-packages/docker/api/container.py:439: in create_container
return self.create_container_from_config(config, name, platform)
../../venv/lib/python3.9/site-packages/docker/api/container.py:456: in create_container_from_config
return self._result(res, True)
../../venv/lib/python3.9/site-packages/docker/api/client.py:271: in _result
self._raise_for_status(response)
../../venv/lib/python3.9/site-packages/docker/api/client.py:267: in _raise_for_status
raise create_api_error_from_http_exception(e) from e
../../venv/lib/python3.9/site-packages/docker/errors.py:39: in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
E docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/containers/create: Not Found ("No such image: jboss/keycloak:16.1.1")
During handling of the above exception, another exception occurred:
../../venv/lib/python3.9/site-packages/docker/api/client.py:265: in _raise_for_status
response.raise_for_status()
../../venv/lib/python3.9/site-packages/requests/models.py:1021: in raise_for_status
raise HTTPError(http_error_msg, response=self)
E requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/create?tag=16.1.1&fromImage=jboss%2Fkeycloak
The above exception was the direct cause of the following exception:
tests/test_keycloak.py:8: in test_docker_run_keycloak
with KeycloakContainer(f"jboss/keycloak:{version}") as kc:
../../venv/lib/python3.9/site-packages/testcontainers/core/container.py:82: in __enter__
return self.start()
../../venv/lib/python3.9/site-packages/testcontainers/keycloak/__init__.py:67: in start
super().start()
../../venv/lib/python3.9/site-packages/testcontainers/core/container.py:64: in start
self._container = docker_client.run(
../../venv/lib/python3.9/site-packages/testcontainers/core/docker_client.py:59: in run
container = self.client.containers.run(
../../venv/lib/python3.9/site-packages/docker/models/containers.py:876: in run
self.client.images.pull(image, platform=platform)
../../venv/lib/python3.9/site-packages/docker/models/images.py:464: in pull
pull_log = self.client.api.pull(
../../venv/lib/python3.9/site-packages/docker/api/image.py:429: in pull
self._raise_for_status(response)
../../venv/lib/python3.9/site-packages/docker/api/client.py:267: in _raise_for_status
raise create_api_error_from_http_exception(e) from e
../../venv/lib/python3.9/site-packages/docker/errors.py:39: in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
E docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=16.1.1&fromImage=jboss%2Fkeycloak: Not Found ("pull access denied for jboss/keycloak, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
------------------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------------------
INFO testcontainers.core.container:container.py:62 Pulling image jboss/keycloak:16.1.1
=================================================================================================== short test summary info ===================================================================================================
FAILED tests/test_keycloak.py::test_docker_run_keycloak[16.1.1] - docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=16.1.1&fromImage=jboss%2Fkeycloak: Not Found ("pull access denied for jboss/keycloak, repository does not exist or may r...
====================================================================================================== 1 failed in 3.84s ======================================================================================================
Problem with new Keycloak versions
Even though we can set a different image/tag in the KeycloakContainer
, the problem remains as that the newer versions of Keycloak differ in API structure when it comes to health checks compared to older versions. In other words, the /auth
is not used anymore for readiness/startup checks (as can be seen here and here). Hence, when one first calls the start()
function, the KeycloackContainer
component keeps waiting for an endpoint that does not exist anymore. After a while, the KeycloackContainer
crashes as it reaches the maximum number of trials or times out.
pytest -s
=================================================================================== test session starts ====================================================================================
platform darwin -- Python 3.9.6, pytest-7.4.3, pluggy-1.4.0
rootdir: /Users/smmarques/Repositorios/testcontainers-python
configfile: pyproject.toml
plugins: anyio-4.3.0, cov-4.1.0
collected 1 item
tests/test_keycloak.py::test_docker_run_keycloak[latest] Pulling image quay.io/keycloak/keycloak:latest
========================================================================================= FAILURES =========================================================================================
_____________________________________________________________________________ test_docker_run_keycloak[latest] _____________________________________________________________________________
tests/test_keycloak.py:8: in test_docker_run_keycloak
with KeycloakContainer(f"quay.io/keycloak/keycloak:{version}") as kc:
../../venv/lib/python3.9/site-packages/testcontainers/core/container.py:82: in __enter__
return self.start()
../../venv/lib/python3.9/site-packages/testcontainers/keycloak/__init__.py:68: in start
self._connect()
../../venv/lib/python3.9/site-packages/testcontainers/core/waiting_utils.py:67: in wrapper
raise TimeoutError(
E TimeoutError: Wait time (120s) exceeded for _connect(args: (), kwargs: {}). Exception: Wait time (120s) exceeded for get_exposed_port(args: (8080,), kwargs: {}). Exception: Port mapping for container 4b998280f780cc1161433e68d566385c1585f88e3b731d7c9bc51bd866015ea1 and port 8080 is not available
------------------------------------------------------------------------------------ Captured log call -------------------------------------------------------------------------------------
INFO testcontainers.core.container:container.py:62 Pulling image quay.io/keycloak/keycloak:latest
INFO testcontainers.core.container:container.py:74 Container started: 4b998280f780
INFO testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 4b998280f780> with image quay.io/keycloak/keycloak:latest to be ready ...
INFO testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 4b998280f780> with image quay.io/keycloak/keycloak:latest to be ready ...
INFO testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 4b998280f780> with image quay.io/keycloak/keycloak:latest to be ready ...
INFO testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 4b998280f780> with image quay.io/keycloak/keycloak:latest to be ready ...
INFO testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 4b998280f780> with image quay.io/keycloak/keycloak:latest to be ready ...
INFO testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 4b998280f780> with image quay.io/keycloak/keycloak:latest to be ready ...
INFO testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 4b998280f780> with image quay.io/keycloak/keycloak:latest to be ready ...
INFO testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 4b998280f780> with image quay.io/keycloak/keycloak:latest to be ready ...
INFO testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 4b998280f780> with image quay.io/keycloak/keycloak:latest to be ready ...
INFO testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 4b998280f780> with image quay.io/keycloak/keycloak:latest to be ready ...
INFO testcontainers.core.waiting_utils:waiting_utils.py:52 Waiting for container <Container: 4b998280f780> with image quay.io/keycloak/keycloak:latest to be ready ...
================================================================================= short test summary info ==================================================================================
FAILED tests/test_keycloak.py::test_docker_run_keycloak[latest] - TimeoutError: Wait time (120s) exceeded for _connect(args: (), kwargs: {}). Exception: Wait time (120s) exceeded for get_exposed_port(args: (8080,), kwargs: {}). Exception: Port mapping...
To Reproduce
To reproduce the first error (missing jboss/keycloak
repository): You must first have all the dependencies installed with poetry. After that just run pytest -s
in the keycloak/tests
directory to see the first error occur.
To reproduce the second error (connection timeout): : You must first have all the dependencies installed with poetry. Then you must change the test function (on keycloak/tests/test_keycloak.py
) to pull the image from the new Keycloak repository, as described below.
import pytest
from testcontainers.keycloak import KeycloakContainer
@pytest.mark.parametrize("version", ["latest"])
def test_docker_run_keycloak(version: str):
with KeycloakContainer(f"quay.io/keycloak/keycloak:{version}") as kc:
kc.get_client().users_count()
Finally, execute pytest -s
in the keycloak/tests
directory.
Runtime environment
The setup I'm currently using for testing is described below
testcontainers-python/modules/keycloak on ๎ main [!] via ๐ v3.9.6 (venv) on โ๏ธ (us-east-1) took 1m29s
โฏ uname -a
Darwin MacBook-Air-de-Sandro.local 23.1.0 Darwin Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8103 arm64
testcontainers-python/modules/keycloak on ๎ main [!] via ๐ v3.9.6 (venv) on โ๏ธ (us-east-1)
โ python --version
Python 3.9.6
testcontainers-python/modules/keycloak on ๎ main [!] via ๐ v3.9.6 (venv) on โ๏ธ (us-east-1)
โ docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.10.0)
compose: Docker Compose (Docker Inc., v2.15.1)
dev: Docker Dev Environments (Docker Inc., v0.0.5)
extension: Manages Docker extensions (Docker Inc., v0.2.17)
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
scan: Docker Scan (Docker Inc., v0.23.0)
Server:
Containers: 16
Running: 2
Paused: 0
Stopped: 14
Images: 28
Server Version: 20.10.22
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.15.49-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 2
Total Memory: 3.841GiB
Name: docker-desktop
ID: ASI4:QNGX:DKJH:GV73:BYZC:RQZ3:2GWP:A7FE:WGFM:2AC4:SX3O:M5XN
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5000
127.0.0.0/8
Live Restore Enabled: false
testcontainers-python/modules/keycloak on ๎ main [!] via ๐ v3.9.6 (venv) on โ๏ธ (us-east-1)
โ pip freeze
alabaster==0.7.16
anyio==4.3.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
asn1crypto==1.5.1
async-timeout==4.0.3
attrs==23.2.0
azure-core==1.29.7
azure-storage-blob==12.19.0
Babel==2.14.0
boto3==1.34.28
botocore==1.34.28
build==1.1.1
CacheControl==0.14.0
cachetools==5.3.2
certifi==2023.11.17
cffi==1.16.0
cfgv==3.4.0
charset-normalizer==3.3.2
cleo==2.1.0
clickhouse-driver==0.2.6
coverage==7.4.0
crashtest==0.4.1
cryptography==42.0.1
cx_Oracle==8.3.0
deprecation==2.1.0
distlib==0.3.8
dnspython==2.5.0
docker==7.0.0
docutils==0.20.1
dulwich==0.21.7
ecdsa==0.18.0
exceptiongroup==1.2.0
fastjsonschema==2.19.1
filelock==3.13.1
google-api-core==2.15.0
google-auth==2.27.0
google-cloud-pubsub==2.19.0
googleapis-common-protos==1.62.0
grpc-google-iam-v1==0.13.0
grpcio==1.60.0
grpcio-status==1.60.0
h11==0.14.0
identify==2.5.35
idna==3.6
imagesize==1.4.1
importlib-metadata==7.0.1
iniconfig==2.0.0
installer==0.7.0
isodate==0.6.1
jaraco.classes==3.3.1
Jinja2==3.1.3
jmespath==1.0.1
kafka-python==2.0.2
keyring==24.3.0
kubernetes==29.0.0
markdown-it-py==3.0.0
MarkupSafe==2.1.4
mdurl==0.1.2
minio==7.2.3
more-itertools==10.2.0
msgpack==1.0.8
mypy==1.7.1
mypy-extensions==1.0.0
neo4j==5.16.0
nh3==0.2.15
nodeenv==1.8.0
oauthlib==3.2.2
opensearch-py==2.4.2
outcome==1.3.0.post0
packaging==23.2
pexpect==4.9.0
pg8000==1.30.4
pika==1.3.2
pkginfo==1.9.6
platformdirs==4.2.0
pluggy==1.4.0
poetry==1.8.2
poetry-core==1.9.0
poetry-plugin-export==1.7.0
pre-commit==3.6.2
proto-plus==1.23.0
protobuf==4.25.2
psycopg2-binary==2.9.9
ptyprocess==0.7.0
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser==2.21
pycryptodome==3.20.0
Pygments==2.17.2
PyJWT==2.8.0
pymongo==4.6.1
PyMySQL==1.1.0
pyproject_hooks==1.0.0
PySocks==1.7.1
pytest==7.4.3
pytest-cov==4.1.0
python-arango==7.9.1
python-dateutil==2.8.2
python-jose==3.3.0
python-keycloak==3.7.0
pytz==2023.3.post1
PyYAML==6.0.1
rapidfuzz==3.6.2
readme-renderer==42.0
redis==5.0.1
requests==2.31.0
requests-oauthlib==1.3.1
requests-toolbelt==1.0.0
rfc3986==2.0.0
rich==13.7.0
rsa==4.9
s3transfer==0.10.0
scramp==1.4.4
selenium==4.17.2
shellingham==1.5.4
six==1.16.0
sniffio==1.3.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
Sphinx==7.2.6
sphinxcontrib-applehelp==1.0.8
sphinxcontrib-devhelp==1.0.6
sphinxcontrib-htmlhelp==2.0.5
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.7
sphinxcontrib-serializinghtml==1.1.10
SQLAlchemy==2.0.25
testcontainers==4.0.1
tomli==2.0.1
tomlkit==0.12.4
trio==0.24.0
trio-websocket==0.11.1
trove-classifiers==2024.3.3
twine==4.0.2
typing_extensions==4.9.0
tzlocal==5.2
urllib3==1.26.18
virtualenv==20.25.1
websocket-client==1.7.0
wrapt==1.16.0
wsproto==1.2.0
xattr==1.1.0
zipp==3.17.0
@matheusvnm thanks for reporting that issue. I already wrote a fix and linked your issue.
@matheusvnm thanks for reporting that issue. I already wrote a fix and linked your issue.
Thanks for the quick reply. Your implementation is very similar to the one I did in my personal project. I gave my 2 cents about your solution in the PR. Do you know when the changes will be released, @max-pfeiffer?