Timeout during image upload
freedge opened this issue · 4 comments
freedge commented
While uploading an image, I get a timeout as below. It looks like the connection timeout (default of 10s) is used instead of the request timeout (default of 30s).
After further checking, in fact it's caused by this issue psf/requests#5263
For POST requests, is it possible to set something higher than 10s as default timeout?
Cheers
uploading new EOS
/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'XXX. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
Traceback (most recent call last):
File "/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/app-root/lib/python3.6/site-packages/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/lib64/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1075, in _send_output
self.send(chunk)
File "/usr/lib64/python3.6/http/client.py", line 996, in send
self.sock.sendall(data)
File "/usr/lib64/python3.6/ssl.py", line 934, in sendall
v = self.send(byte_view[count:])
File "/usr/lib64/python3.6/ssl.py", line 903, in send
return self._sslobj.write(data)
File "/usr/lib64/python3.6/ssl.py", line 601, in write
return self._sslobj.write(data)
socket.timeout: The write operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/app-root/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/app-root/lib/python3.6/site-packages/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/app-root/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/app-root/lib/python3.6/site-packages/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/lib64/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1075, in _send_output
self.send(chunk)
File "/usr/lib64/python3.6/http/client.py", line 996, in send
self.sock.sendall(data)
File "/usr/lib64/python3.6/ssl.py", line 934, in sendall
v = self.send(byte_view[count:])
File "/usr/lib64/python3.6/ssl.py", line 903, in send
return self._sslobj.write(data)
File "/usr/lib64/python3.6/ssl.py", line 601, in write
return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "upgrade.py", line 112, in <module>
resp = clnt.api.add_image(targetImage)
File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_api.py", line 1870, in add_image
files={'file': image_data})
File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_client.py", line 898, in post
return self._make_request('POST', url, timeout, data=data, files=files)
File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_client.py", line 664, in _make_request
raise error
File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_client.py", line 642, in _make_request
data, files)
File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_client.py", line 774, in _send_request
raise error
File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_client.py", line 763, in _send_request
files=files)
File "/opt/app-root/lib/python3.6/site-packages/requests/sessions.py", line 590, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/opt/app-root/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/opt/app-root/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/opt/app-root/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out',))
mharista commented
Hi @freedge
There are two timeouts you can set via the connect method in the cvprac client. Have you tried increasing the connect_timeout and the request_timeout via the connect method? Also the client post method does allow a timeout parameter to be sent.
Connect method info:
def connect(self, nodes, username, password, connect_timeout=10,
request_timeout=30, protocol='https', port=None, cert=False,
is_cvaas=False, tenant=None, api_token=None, cvaas_token=None):
Login to CVP and get a session ID and cookie. Currently
certificates are not verified if the https protocol is specified. A
warning may be printed out from the requests module for this case.
Args:
nodes (list): A list of hostname/IP addresses for CVP nodes
username (str): The CVP username
password (str): The CVP password
connect_timeout (int): The number of seconds to wait for a
connection.
request_timeout (int): The default number of seconds to allow
api requests to complete before timing out.
protocol (str): The protocol to use to connect to CVP.
THIS PARAMETER IS NOT USED AND WILL BE DEPRECATED.
ONLY INCLUDED TO NOT BREAK EXISTING CODE THAT HAS PROTOCOL
SPECIFIED IN CONNECTION.
port (int): The TCP port of the endpoint for the connection.
If this keyword is not specified, the default value is
automatically determined by the transport type.
(http=80, https=443)
cert (str or boolean): Path to a cert file used for a https
connection or boolean with default False. If a cert is
provided then the connection will not attempt to fallback
to http. The False default sets the request to not verify
the servers TLS certificate.
is_cvaas (boolean): Flag for enabling connection to CVaaS.
tenant: (string): Tenant/Org within CVaaS to connect to.
Required if is_cvaas is enabled.
cvaas_token (string): API Token to use in place of UN/PW login
for CVaaS.
api_token (string): API Token to use in place of UN/PW login
for CVP 2020.3.0 and beyond.
Post method info:
def post(self, url, data=None, files=None, timeout=30):
Make a POST request to CVP. If the request call raises an error
or if the JSON response contains a CVP session related error then
retry the request on another CVP node.
Args:
url (str): Portion of request URL that comes after the host.
data (dict): Dict of key/value pairs to pass as parameters into
the request. Default is None.
files (dict): Dict of file name to files for upload. Currently
only used for adding images to CVP. Default is None.
timeout (int): Number of seconds the client will wait between
bytes sent from the server. Default value is 30 seconds.
freedge commented
My point is that the timeout=30 option on the post method is useless (in the context of pushing a large image), as it will timeout after 10 seconds anyway. My understanding is that the code is done with sane defaults of 10s to connect to a service and 30seconds to perform requests such as POST, but practically the wrong timeout value is used.
freedge commented
(also I will try to play with the connect_timeout, thanks for the suggestion :) )
freedge commented
we added a connect_timeout in this fashion
-clnt.connect([os.environ['CVP_HOST']], 'cvpadmin', os.environ['CVP_PASS'])
+clnt.connect([os.environ['CVP_HOST']], 'cvpadmin', os.environ['CVP_PASS'], connect_timeout=120)
and it works great. Thanks