urllib3 v2 incompatibility
zachmullen opened this issue · 40 comments
Minimal repro:
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
/opt/worker_venv/lib/python3.8/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (2.0.0) or chardet (None)/charset_normalizer (3.1.0) doesn't match a supported version!
warnings.warn(
>>> client = docker.from_env(version='auto')
Traceback (most recent call last):
File "/opt/worker_venv/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/opt/worker_venv/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/opt/worker_venv/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/opt/worker_venv/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/opt/worker_venv/lib/python3.8/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/opt/worker_venv/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/opt/worker_venv/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/opt/worker_venv/lib/python3.8/site-packages/requests/adapters.py", line 487, in send
resp = conn.urlopen(
File "/opt/worker_venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/opt/worker_venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 496, in _make_request
conn.request(
TypeError: request() got an unexpected keyword argument 'chunked'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/worker_venv/lib/python3.8/site-packages/docker/client.py", line 96, in from_env
return cls(
File "/opt/worker_venv/lib/python3.8/site-packages/docker/client.py", line 45, in __init__
self.api = APIClient(*args, **kwargs)
File "/opt/worker_venv/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
self._version = self._retrieve_server_version()
File "/opt/worker_venv/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: request() got an unexpected keyword argument 'chunked'urllib3 2.0.0 just released today. A quick fix would be to pin to urllib3<2.
I think also the request version needs to be pinned, as there was a change?
Works with requests 2.28.1, but fails with 2.29.x
/tmp/ansible_docker_swarm_payload_315b1r1d/ansible_docker_swarm_payload.zip/ansible_collections/community/docker/plugins/module_utils/common.py", line 222, in _init_
super(AnsibleDockerClientBase, self)._init_(**self._connect_params)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 197, in _init_
self._version = self._retrieve_server_version()
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(...
....
},
"msg": "Error connecting: Error while fetching server API version: request() got an unexpected keyword argument 'chunked'"
}
I can confirm just pinning requests<2.2.29 is solving this for me
That's because older versions of requests vendored in their own urllib3. Modern requests uses the distributed version.
I think urllib3 2.0.0 (or vendored urllib3 verions in requests) is a red herring; the problem is requests 2.29.0 itself because of a new feature: https://github.com/psf/requests/pull/6226/files
Docker SDK for Python uses requests with its own HTTP adapters for Unix connections by default (if you don't talk to the Docker daemon over TCP connections). For that, the code in Docker SDK for Python that provides a Unix connection HTTP adapter needs to be adjusted to support the same chunking support that urllib3 (already < 2.0.0) provides, and that requests 2.29.0 uses. (The other HTTP adapters might have the same problems.)
Wow this post is really helpful as I've just encounter an issue with Ansible managing Docker.
I created a PR for making Docker SDK for Python compatible with requests 2.29.0: #3116. It fixes the problems for me (as long as I stick to urllib3 < 2.0, what request has been requiring for a long time now). It seems to fix the problems I had with requests 2.29.0 and urllib3 < 2.0; would be nice to have more real life testing (especially for all the transports I don't use).
Hello, urllib3 2.0 maintainer here 👋
First, let me mention that right now, if you use pip install docker with a recent enough pip that can do dependency resolution you will end up with requests 2.29.0 and urllib3 1.26.15. With those versions, the docker.from_env reproducer above works. I'm surprised that @felixfontein sees issues with those versions, can you please explain how to reproduce them?
I can confirm however that docker-py is not compatible with urllib3 2.0. Since requests will soon allow it, docker-py should modify setup.py to use urllib3 >= 1.26.0, < 2.0.0 or fix the bug.
So what is the issue? Here is how docker-py supports Unix domain sockets:
docker-py/docker/transport/unixconn.py
Lines 17 to 37 in a02ba74
It assumes that urllib3.connectionpool.HTTPConnectionPool will call conn.request in a way that is compatible with http.client, but this has changed in urllib3 2.0, and HTTPConnectionPool has its own request() function with its own signature.
I've opened #3117 for a fix.
@pquentin as you mention installing docker with a recent pip will give you requests 2.29.0 and urllib3 < 2.0, so urllib3 2.0 shouldn't be a problem in that case. But the stack trace printed above isn't the only thing that can go wrong with requests 2.29.0, I found some problems in CI with requests 2.29.0 and urllib3 1.26.15 as well. These happen because requests 2.29.0 includes https://github.com/psf/requests/pull/6226/files, which ends up calling UnixHTTPConnection.urlopen() with some parameters that httplib.HTTPConnection's version does not provide. In my case this happened with APIClient.put_archive() when streaming data in. (I'm not using Docker SDK for Python directly, but some code vendored from it - the first stack trace of ansible-collections/community.docker#611 (comment) happens with requests 2.29.0 and urllib3 1.26.15.)
BTW, your fix only fixes one of the transports; my similar PR (#3116) catches some more.
Docker is installed over pip with the dependencies:
"Requirement already up-to-date: docker in /usr/local/lib/python3.8/dist-packages (6.0.1)",
"Requirement already satisfied, skipping upgrade: requests>=2.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.29.0)",
"Requirement already satisfied, skipping upgrade: packaging>=14.0 in /usr/local/lib/python3.8/dist-packages (from docker) (23.1)",
"Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /usr/local/lib/python3.8/dist-packages (from docker) (1.5.1)",
"Requirement already satisfied, skipping upgrade: urllib3>=1.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.0.1)",
"Requirement already satisfied, skipping upgrade: charset-normalizer<4,>=2 in /usr/local/lib/python3.8/dist-packages (from requests>=2.26.0->docker) (3.1.0)",
"Requirement already satisfied, skipping upgrade: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2.8)",
"Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2019.11.28)"
And we have the same problem:
"msg": "Error connecting: Error while fetching server API version: request() got an unexpected keyword argument 'chunked'"
The only solution for us was:
pip3 remove requests
pip3 install requests=2.28.1
Hello from downstream consumer of docker-py from ue4-docker! So, what is the current workaround? Lock requests to <2.29, or urllib3 to <2, or both?
And what do you mean by "recent enough" pip? We hit this issue on Ubuntu 20.04 with pip-22.0.4.
And what do you mean by "recent enough" pip? We hit this issue on Ubuntu 20.04 with pip-22.0.4.
I meant at least pip 20.3, and ideally 21.3 or above, to get the new resolver and all its fixes. Unfortunately the recent enough pip is no longer enough as requests 2.30.0 was just released with urllib3 2.0 support.
Until docker-py pins urllib3 or merges #3116, you’ll have to pin to urllib3<2.0 yourself. As far as I can tell this has nothing to do with requests, so you can use the latest here.
You can run the following with requests==0.29.0 and urllib3==1.26.13 to trigger an error:
from docker import APIClient
from docker.utils import kwargs_from_env
kwargs = kwargs_from_env()
client = APIClient(**kwargs)
def generate_data():
yield b'123'
container = client.create_container(image='debian:bullseye', command=['/bin/sh', '-c', 'ls -la /'], stdin_open=True, detach=True, name='test')
client.put_archive(container['Id'], '/', generate_data())Stacktrace:
Traceback (most recent call last):
File "/path/to/my-script.py", line 11, in <module>
client.put_archive(container['Id'], '/', generate_data())
File "/path/to/docker-py/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/path/to/docker-py/docker/api/container.py", line 980, in put_archive
res = self._put(url, params=params, data=data)
File "/path/to/docker-py/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/path/to/docker-py/docker/api/client.py", line 241, in _put
return self.put(url, **self._set_request_timeout(kwargs))
File "/home/felix/.local/lib/python3.10/site-packages/requests/sessions.py", line 647, in put
return self.request("PUT", url, data=data, **kwargs)
File "/home/felix/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/felix/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/felix/.local/lib/python3.10/site-packages/requests/adapters.py", line 487, in send
resp = conn.urlopen(
File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 396, in _make_request
conn.request_chunked(method, url, **httplib_request_kw)
AttributeError: 'UnixHTTPConnection' object has no attribute 'request_chunked'
So yes, you need requests < 2.29.0 as well with the currently released Docker SDK for Python.
I can confirm just pinning
requests<2.2.29is solving this for me
Sadly this does not work in python 3.10+, I had to downgrade my container to 3.9.
I can confirm just pinning
requests<2.2.29is solving this for meSadly this does not work in python 3.10+, I had to downgrade my container to 3.9.
@zwgtdev pinning to requests==2.29.0 worked for me with Python 3.10.6.
I simply added requests==2.29.0 to requirements.txt file containing, in my case, docker==6.0.1.
requests version 2.30.0 released on May 3rd causes issues.
I can confirm just pinning requests<2.2.29 is solving this for me
The digits are transposed here. Pinning requests<2.29.2 does the trick for me as requests 2.29.0 still pins urllib3 < 2.
A huge thank you to everyone here for the debugging, discussion, and PR(s) to fix the issue! ❤️
A new release is available with this fix as well as several other queued up improvements: https://github.com/docker/docker-py/releases/tag/6.1.0
I can confirm just pinning
requests<2.2.29is solving this for me
For folks using the workaround, looks like there's a small typo here. Don't need to go back to requests 2.2.29, 2.29.0 will do just fine.
requests<2.29.0
Running ansible on an Ubuntu 18.04 box, targeting a 22.04 box with docker, seeing the error
'Error connecting: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument ''chunked'''
The 18.04 workstation has
ansible 7.5.0
ansible-core 2.14.5
requests 2.28.2
requests-unixsocket 0.1.5
urllib3 1.26.15
The 22.04 box is docker 23.0.6
$ docker --version
Docker version 23.0.6, build ef23cbc
$ dpkg --no-pager -l docker\* | fgrep ii
ii docker-buildx-plugin 0.10.4-1~ubuntu.22.04~jammy amd64 Docker Buildx cli plugin.
ii docker-ce 5:23.0.6-1~ubuntu.22.04~jammy amd64 Docker: the open-source application container engine
ii docker-ce-cli 5:23.0.6-1~ubuntu.22.04~jammy amd64 Docker CLI: the open-source application container engine
ii docker-ce-rootless-extras 5:23.0.6-1~ubuntu.22.04~jammy amd64 Rootless support for Docker.
The requests module is at 2.28.22, so definitely less than 2.2.29
Any hints ?
Any hints ?
In our case the problem was the requests/urllib3 version on the target machine. That's where docker-py was executed with docker access via socket.
Also note that Ansible 7.5.0 includes community.docker 3.4.3; if you use modules that use the vendored Docker SDK for Python code instead of Docker SDK for Python directly, you need to make sure that you have community.docker 3.4.5 installed instead if you have requests >= 2.29.0 or urllib3 >= 2.0 on the target machine. It will be part of Ansible 7.6.0 (which should get released in ~two weeks).
The target 22.04 box has
❯ pip list | egrep 'request|urllib'
requests 2.30.0
urllib3 2.0.2
I downgraded requests to 2.29.0, which also brought urllib down
❯ pip install requests==2.29.0 --force-reinstall
❯ pip list | egrep 'request|urllib'
requests 2.29.0
urllib3 1.26.15
Unfortunately, still get the same error
@Halfwalker which version of community.docker are you using, and which modules of it? And are you sure that you are checking the same Python interpreter on the target box that Ansible is using to execute the module with? Also I think this is a discussion that doesn't really belong in this repository (this is about Docker SDK for Python, not about third-party code using that SDK); https://github.com/ansible-collections/community.docker/issues would be a better place.
How the heck it is possible, that this issue has status fixed by a PR done in totally unrelated project microsoft/Olive?!
How?
Because microsoft/Olive#239 contains "Skip docker system test for bert example until bug is fixed: #3113" which triggered this GitHub feature that reacts to the "fixed" word.
But the issue was actually closed after a release with the actual fix: #3113 (comment)
I faced the same issue when was tying to set up server with ansible , two work around which helped me:
1)Do not change versions of requests(latest) and urllib3(latest) and install docker-py instead of docker
2)remove urlib3 , install urllib3==1.26.15 and then install docker with pip3
Hope it will help anyone )
Do NOT install https://pypi.org/project/docker-py/ instead of https://pypi.org/project/docker/. docker-py should only be used if you need Python 2.6 support.
msg: 'Error connecting: Error while fetching server API version: request() got an unexpected keyword argument ''chunked'''
I faced the same issue with that configuration:
Ubuntu 20.04
Docker version 24.0.2, build cb74dfc
docker py is 6.1.2 and its latest at the moment.
requests 2.31.0
urllib3 2.0.2
pip3 install docker
Requirement already satisfied: **docker in /usr/local/lib/python3.8/dist-packages (6.1.2)**
Requirement already satisfied: packaging>=14.0 in /usr/local/lib/python3.8/dist-packages (from docker) (23.1)
Requirement already satisfied: **requests>=2.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.31.0)**
Requirement already satisfied: **urllib3>=1.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.0.2)**
Requirement already satisfied: websocket-client>=0.32.0 in /usr/local/lib/python3.8/dist-packages (from docker) (0.59.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.8/dist-packages (from requests>=2.26.0->docker) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2019.11.28)
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from websocket-client>=0.32.0->docker) (1.14.0)
After fix from this comment, which downgrade requests from 2.31.0 to 2.28.1, problem with pulling images with docker_image ansible plugin works fine.
Ansible version [core 2.13.6]
python version = 3.8.10
community.docker-3.2.1 from ansible galaxy
Docker is installed over pip with the dependencies:
"Requirement already up-to-date: docker in /usr/local/lib/python3.8/dist-packages (6.0.1)", "Requirement already satisfied, skipping upgrade: requests>=2.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.29.0)", "Requirement already satisfied, skipping upgrade: packaging>=14.0 in /usr/local/lib/python3.8/dist-packages (from docker) (23.1)", "Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /usr/local/lib/python3.8/dist-packages (from docker) (1.5.1)", "Requirement already satisfied, skipping upgrade: urllib3>=1.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.0.1)", "Requirement already satisfied, skipping upgrade: charset-normalizer<4,>=2 in /usr/local/lib/python3.8/dist-packages (from requests>=2.26.0->docker) (3.1.0)", "Requirement already satisfied, skipping upgrade: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2.8)", "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2019.11.28)"And we have the same problem: "msg": "Error connecting: Error while fetching server API version: request() got an unexpected keyword argument 'chunked'"
The only solution for us was:
pip3 remove requests pip3 install requests=2.28.1
this fixed the issue for me
pip uninstall requests
pip install requests==2.28.1
@komkomissarov you need to upgrade community.docker. You are using an outdated version (7 months old!) which still has this bug (which only got fixed a ~month ago). You need version 3.4.5 (released 24 days ago) or later.
urllib3<2
Help please! 🙏🏼
How do I pin to urllib3<2 ?
Help please! 🙏🏼
How do I pin to urllib3<2 ?
@MaritzaB pip install 'urllib3<2' or add urllib3<2 in your requirements list.
Errors:
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 496, in _make_request
conn.request(
TypeError: request() got an unexpected keyword argument 'chunked'
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 496, in _make_request
conn.request(
TypeError: request() got an unexpected keyword argument 'preload_content'
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 496, in _make_request
conn.request(
TypeError: request() got an unexpected keyword argument 'decode_content'
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 522, in _make_request
if not conn.is_closed:
AttributeError: 'UnixHTTPConnection' object has no attribute 'is_closed'
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 556, in _make_request
response.length_remaining, # type: ignore[attr-defined]
AttributeError: 'UnixHTTPResponse' object has no attribute 'length_remaining'
Why: It returns UnixHTTPResponse and get appropriace request object instead of objects from urllib. And that objects did not have these properties/methods. The worst thing: It's mentioned in comments inside the code (for me line 493).
Fast fix: Comment appropriate lines and blocks in "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py"
Correct fix: Detect existence of these params/methods and use them only if they are available.
@alex-kalanis What version of urllib3, requests and docker-py are you using? docker-py is compatible with urllib3 2.x since version 6.1.0, released in May 2023.
Installed this module for use with Ansible, via the geerlingguy.docker and geerlingguy.pip roles. I now have on the target system:
- docker 7.0.0
- requests 2.31.0
- urllib3 2.2.1
If I try to use Ansible to manage Docker configuration this fails with:
"msg": "Error connecting: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument 'chunked'"
Can you please share the full stack traceback? It's unlikely to come from the versions you mentioned.
The full traceback is:
File "/tmp/ansible_docker_network_payload_8ki4gfeh/ansible_docker_network_payload.zip/ansible_collections/community/docker/plugins/module_utils/common_api.py", line 118, in __init__
super(AnsibleDockerClientBase, self).__init__(**self._connect_params)
File "/tmp/ansible_docker_network_payload_8ki4gfeh/ansible_docker_network_payload.zip/ansible_collections/community/docker/plugins/module_utils/_api/api/client.py", line 188, in __init__
self._version = self._retrieve_server_version()
File "/tmp/ansible_docker_network_payload_8ki4gfeh/ansible_docker_network_payload.zip/ansible_collections/community/docker/plugins/module_utils/_api/api/client.py", line 212, in _retrieve_server_version
raise DockerException(
fatal: [XX.XXX.com]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"api_version": "auto",
"appends": false,
"attachable": null,
"ca_cert": null,
"client_cert": null,
"client_key": null,
"connected": [],
"debug": false,
"docker_host": "unix://var/run/docker.sock",
"driver": "bridge",
"driver_options": {},
"enable_ipv6": null,
"force": false,
"internal": null,
"ipam_config": null,
"ipam_driver": null,
"ipam_driver_options": null,
"labels": {},
"name": "reverse-proxy",
"scope": null,
"ssl_version": null,
"state": "present",
"timeout": 60,
"tls": false,
"tls_hostname": null,
"use_ssh_client": false,
"validate_certs": false
}
},
"msg": "Error connecting: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument 'chunked'"
}
This was fixed in community.docker 3.4.5, are you using an older version? Please report any further issues you have to https://github.com/ansible-collections/community.docker instead.
I was, yes. I'd also tried switching between the O/S (Ubuntu 22.04) package and PIP installs of docker-py on both host and target, and getting nowhere, but hadn't through to update the Ansible collection, which has resolved the issue!
In case someone lands on this issue like I did trying to use ansible to manage Docker on a Ubuntu 24 LTS server, a quick solution. Make sure to use the new "community.docker.docker_compose_v2" task instead of the v1 "community.docker.docker_compose" task. You get this exact same error with v1.
In case someone lands on this issue like I did trying to use ansible to manage Docker on a Ubuntu 24 LTS server, a quick solution. Make sure to use the new "community.docker.docker_compose_v2" task instead of the v1 "community.docker.docker_compose" task. You get this exact same error with v1.
Thanks. I spend hours lookinng for a solution for installing Octoprint docker and your comment was the only good suggestion.