set_http_retries() does not have effect
greatvovan opened this issue · 1 comments
I have a script that simulates real-time data, for which it ingests some small numbers of records inline.
from azure.kusto.data import KustoClient, KustoConnectionStringBuilder
from azure.kusto.data.exceptions import KustoThrottlingError
cluster = "https://myadx.centralus.kusto.windows.net"
client_id = "..."
client_secret = "..."
authority_id = "..."
db = "db"
kcsb = KustoConnectionStringBuilder.with_aad_application_key_authentication(cluster, client_id, client_secret, authority_id)
client = KustoClient(kcsb)
while True:
client.execute(db, ".append t <| print now(), expression")
sleep(30)There is also a logic to handle HTTP 429 (throttling) and generate meaningful data (^ expression in ADX), but I don't think it is relevant here.
Recently the script started to fail too often with error
http.client.RemoteDisconnected: Remote end closed connection without response
causing
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
causing
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
I probably need to mention that there are two copies of the script sending data with 30 s and 10 s interval. The one that is more likely to fail is 30 s one.
I am sure there are no internet problems on my end. If restarted, the script continues to work normally.
It is not a big problem to handle this error outside of the library, but I decided to try set_http_retries() method of KustoClient, which I added right after initialization of the client. I tried it both with an integer value and requests.adapters.Retry object. Both seem to have no effect in my situation.
Enabling debug logging level in my program I can see:
2022-11-07 18:12:00,766: DEBUG connectionpool urllib3.connectionpool Starting new HTTPS connection (1): myadx.centralus.kusto.windows.net:443
2022-11-07 18:12:01,187: DEBUG connectionpool urllib3.connectionpool https://myadx.centralus.kusto.windows.net:443 "GET /v1/rest/auth/metadata HTTP/1.1" 200 None
2022-11-07 18:12:01,192: DEBUG retry urllib3.util.retry Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
2022-11-07 18:12:01,197: DEBUG connectionpool urllib3.connectionpool Starting new HTTPS connection (1): login.microsoftonline.com:443
2022-11-07 18:12:01,466: DEBUG connectionpool urllib3.connectionpool https://login.microsoftonline.com:443 "GET /.../v2.0/.well-known/openid-configuration HTTP/1.1" 200 1753
2022-11-07 18:12:01,470: DEBUG authority msal.authority openid_config = {...}
2022-11-07 18:12:01,470: DEBUG application msal.application Region to be used: None
2022-11-07 18:12:01,470: DEBUG application msal.application Found 0 RTs matching {'environment': 'login.microsoftonline.com', 'home_account_id': None, 'family_id': '1'}
2022-11-07 18:12:01,470: DEBUG application msal.application Found 0 RTs matching {'environment': 'login.microsoftonline.com', 'home_account_id': None, 'client_id': '...'}
2022-11-07 18:12:01,570: DEBUG connectionpool urllib3.connectionpool https://login.microsoftonline.com:443 "GET /common/discovery/instance?api-version=1.1&authorization_endpoint=https://login.microsoftonline.com/common/oauth2/authorize HTTP/1.1" 200 945
2022-11-07 18:12:01,574: DEBUG telemetry msal.telemetry Generate or reuse correlation_id: 7c5aeac9-cd46-4940-96fe-db1ed9ae99ab
2022-11-07 18:12:01,727: DEBUG connectionpool urllib3.connectionpool https://login.microsoftonline.com:443 "POST /.../oauth2/v2.0/token HTTP/1.1" 200 1343
2022-11-07 18:12:01,731: DEBUG token_cache msal.token_cache event={
...
}
2022-11-07 18:12:01,737: DEBUG connectionpool urllib3.connectionpool Starting new HTTPS connection (1): myadx.centralus.kusto.windows.net:443
2022-11-07 18:12:02,303: DEBUG connectionpool urllib3.connectionpool https://myadx.centralus.kusto.windows.net:443 "POST /v1/rest/mgmt HTTP/1.1" 200 None
When run in debugger, the output begins at the first call of client.execute(). Note at the line 3. It looks like it disregards the set retry policy, and uses a Retry with total=1.
Is it a bug or my expectation of retry is off in this case?
Output of pip freeze
[paste the output of pip freeze here below this line]
azure-core==1.25.1
azure-identity==1.10.0
azure-kusto-data==3.1.3
certifi==2022.6.15.1
cffi==1.15.1
charset-normalizer==2.1.1
cryptography==38.0.1
idna==3.3
ijson==3.1.4
msal==1.18.0
msal-extensions==1.0.0
portalocker==2.5.1
pycparser==2.21
PyJWT==2.4.0
python-dateutil==2.8.2
pywin32==304
PyYAML==6.0
requests==2.28.1
six==1.16.0
typing_extensions==4.3.0
urllib3==1.26.12
Solved in version 4.0.0
Re-open if you encounter further issues