[BUG] Occasional (but disruptive) connection failure errors
pradh opened this issue · 1 comments
pradh commented
The code snippet below runs into connection failures on my linux machine (and Fiona's too), but runs fine on colab. Not yet sure if its a lib/network issue.
import json
import requests
resp = requests.get('https://api.datacommons.org/node/places-in?dcids=geoId/06&placeType=CensusTract')
ans = json.loads(json.loads(resp.content)['payload'])
for pair in ans:
resp = requests.get('https://api.datacommons.org/node/property-values?dcids=' + pair['place'] + '&property=geoJsonCoordinates')
ans = json.loads(json.loads(resp.content)['payload'])
if pair['place'] not in ans or 'out' not in ans[pair['place']]: continue
for val in ans[pair['place']]['out']:
print(val['value'])
pradh commented
Error message:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 978, in _validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 308, in connect
conn = self._new_conn()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 171, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f8107a4fd90>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.datacommons.org', port=443): Max retries exceeded with url: /node/property-values?dcids=geoId/06001406201&property=geoJsonCoordinates (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8107a4fd90>: Failed to establish a new connection: [Errno 110] Connection timed out'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "dc_scrape.py", line 7, in <module>
resp = requests.get('https://api.datacommons.org/node/property-values?dcids=' + pair['place'] + '&property=geoJsonCoordinates')
File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.datacommons.org', port=443): Max retries exceeded with url: /node/property-values?dcids=geoId/06001406201&property=geoJsonCoordinates (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8107a4fd90>: Failed to establish a new connection: [Errno 110] Connection timed out'))