Azure/azure-storage-python

What's the correct way to recover from an Azure Storage “connection reset” 10054 error, when called from Python SDK?

rfernand2 opened this issue · 6 comments

rfernand2 commented

We have a python app that uses Azure Storage, calling the BlockBlobService and AppendBlobService modules of the azure-storage-blob==2.1.0 package. When running a large number of simultaneous instances of our python app, we start seeing many occurrences of the following error:
ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
What is the best practice for recovering from these types of errors? Our normal retry function just fails on all 25 of its retries when this error is encountered. Is there a way to request that the connection to Azure is reset within a retry function?

mitchdenny commented

@rfernand2 I am transferring this issue over to the https://github.com/azure/azure-sdk-for-python repository since it is Python specific.

rfernand2 commented

OK, thanks.

amishra-dev commented

Hi Roland (@rfernand2 ), are you still hitting this issue?

tasherif-msft commented

Hi @rfernand2, you have the ability to use a response hook (essentially a callback function) that would enable you to do something with the response you get before the retry is kicked off once again.
Using a raw_response_hook looks like the following:

counter = 0
def fail_response(response):
    if response.http_response.status_code == 404:
        counter += 1
bsc = BlobServiceClient(url, credential=your_credential, retry_total=4)
container = bsc.get_container_client(container_name)
with self.assertRaises(Exception):
    container.create_container(raw_response_hook=fail_response)

In this case the counter will keep incrementing before retrying.
Let me know if you need further more specific help on your use case!

rfernand2 commented

Hi @tasherif-msft, thanks for the response. I'm not sure I understand how the above code would do anything to cause the RETRY to succeed. As I mentioned above, once I see this "connection reset" error, it always fails the 25 normal retries. Is there something special about a raw_response_hook that would automatically reestablish the connection?

tasherif-msft commented

Hi @rfernand2 I just noticed that you're using version 2.1.0 which has been deprecated for some time now.
This repository has the track 2 SDK which I highly recommend you upgrade to. Let me know if this is a possibility for you.
@xiafu-msft he is using track 1 SDK if you have any input