Cannot connect to webservice behind Cloudflare
Closed this issue · 7 comments
Hello,
When I try connect to my zeronet web server behind Cloudflare I got error:
Traceback (most recent call last):
(...)
File "/home/ko/dev/ZeroFramePy/zeroframe_ws_client/__init__.py", line 81, in __init__
self._connect()
File "/home/ko/dev/ZeroFramePy/zeroframe_ws_client/__init__.py", line 115, in _connect
self.wrapper_key = self._get_wrapper_key()
File "/home/ko/dev/ZeroFramePy/zeroframe_ws_client/__init__.py", line 140, in _get_wrapper_key
wrapper_body = urllib.request.urlopen(wrapper_request).read()
File "/usr/lib64/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python3.7/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib64/python3.7/urllib/request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python3.7/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib64/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/lib64/python3.7/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
Request has been blocked by Cloudflare because User-Agent is empty.
Fix
Original file:
ZeroFramePy/zeroframe_ws_client/__init__.py
Line 139 in dc1c5a7
I added my user agent to the request and now request to get wrapper_key is correct. Everything is working.
wrapper_request = urllib.request.Request(site_url, headers={'Accept': 'text/html', 'User-Agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36 (pl-PL,pl;q=0.9,en-US;q=0.8,en;q=0.7)'})In my opinion User-Agent also should contains version of ZeroFramePy library
Do you know if Cloudflare only accepts user-agent from real browsers or could we just use something like ZeroFramePy/1.0.0? Note that default user-agent is not empty (but Python-urllib/3.7) so it looks like Cloudflare only accepts real browsers.
We should use one of the following user-agents (by priority, probably first that works):
ZeroFramePy/1.0.0Mozilla/5.0 ZeroFramePy/1.0.0Mozilla/5.0 Gecko/20000000 Firefox/70.0 ZeroFramePy/1.0.0Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0 ZeroFramePy/1.0.0Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0
@krzotr Can you check which of this user-agents work for you?
This should also be fixed in https://github.com/filips123/ZeroFrameJS.
I wrote script to test User-Agents. Looks like the User-Agent header is required. Can be empty but must be defined.
Does not matter if you try to test on cloudflare.com or zeronet reverse proxied by cloudflare - the same results
Result:
code: 200 - UA: ZeroFramePy/1.0.0
code: 200 - UA: Mozilla/5.0 ZeroFramePy/1.0.0
code: 200 - UA: Mozilla/5.0 Gecko/20000000 Firefox/70.0 ZeroFramePy/1.0.0
code: 200 - UA: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0 ZeroFramePy/1.0.0
code: 200 - UA: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0
code: 403 - UA:
import urllib.request
site_url = "https://cloudflare.com/"
user_agents = [
'ZeroFramePy/1.0.0',
'Mozilla/5.0 ZeroFramePy/1.0.0',
'Mozilla/5.0 Gecko/20000000 Firefox/70.0 ZeroFramePy/1.0.0',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0 ZeroFramePy/1.0.0',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0',
'' # Empty for testing
]
for user_agent in user_agents:
code = 200
headers = {
"Accept": "text/html"
}
if user_agent != "":
headers["User-Agent"] = user_agent
try:
wrapper_request = urllib.request.Request(site_url, headers=headers)
wrapper_body = urllib.request.urlopen(wrapper_request).read()
except Exception as e:
code = e.code
print("code: %d - UA: %s " % (code, user_agent))It looks like they block empty user-agent and also default Python user-agent (Python-urllib). So I will just use ZeroFramePy/1.0.0.
I will fix this tomorrow.
@krzotr I pushed new commits to use ZeroFramePy/ + __version__ as user-agent.
Does this work for you? Also, which version do you get from zeroframe_ws_client.__version__ (you should probably get 1.0.0 if you installed this with python setup.py install)?
If all things work correctly, I will publish new PyPI version.
I have not run python setup.py install. I cloned repository and installed all requirements.
Now I get:
Traceback (most recent call last):
File "/home/ko/dev/ZeroFramePy/test.py", line 5, in <module>
from zeroframe_ws_client import ZeroFrame
File "/home/ko/dev/ZeroFramePy/zeroframe_ws_client/__init__.py", line 18, in <module>
__version__ = pkg_resources.require('zeroframe-ws-client')[0].version
File "/home/ko/dev/ZeroFramePy/venv/lib64/python3.7/site-packages/setuptools-40.8.0-py3.7.egg/pkg_resources/__init__.py", line 900, in require
File "/home/ko/dev/ZeroFramePy/venv/lib64/python3.7/site-packages/setuptools-40.8.0-py3.7.egg/pkg_resources/__init__.py", line 786, in resolve
pkg_resources.DistributionNotFound: The 'zeroframe-ws-client' distribution was not found and is required by the application
I think better way is put correct version to __version__ variable and modify setup() to get version of ZeroFrame from __init__ file. Here is example - single-sourcing-package-version. The different is you do not need install package to work correctly.
After I changed __version__ = "1.0.0" i can connect to my server behind Cloudflare.
I want to automatically get the package version from the Git tag, so I use setuptools-git-version. Because of this, I don't want to hard-code version into the file and manually update it on every release.
Do you think this would be OK:
from pkg_resources import get_distribution, DistributionNotFound
try:
__version__ = get_distribution(__name__).version
except DistributionNotFound:
__version__ = '0.0.0'