Upload files to storage failing with 403 TooManyDevicesError

[L-PDS]

Context

• OS and version used: Embedded Linux on an IoT Device
• Python version: 3.10.4

I have a python app running on an embedded linux device. The app runs async code that connects an iothub device client. I have a class called device_client_wrapper that uses a semaphore to ensure that only one function calls device client methods at a time, as that was causing me issues before.

I just added a new function to this class which is used to upload files from the device to azure blob storage. It was working fine for a while, and often works the first time, but then if I execute it too many times I get the following error:

ERROR upload_file Exception: Unexpected failure
ERROR Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 33, in handle_result
    return await callback.completion()
  File "/usr/lib/python3.10/site-packages/azure/iot/device/common/async_adapter.py", line 91, in completion
    return await self.future
azure.iot.device.exceptions.ServiceError: HTTP operation returned: 403 TooManyDevicesError(Error: Forbidden)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/cogo_ig60/cloud/device_client_wrapper.py", line 99, in upload_file
    blob_info = await self.device_client.get_storage_info_for_blob(blob_name)
  File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 571, in get_storage_info_for_blob
    storage_info = await handle_result(callback)
  File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 57, in handle_result
    raise exceptions.ClientError("Unexpected failure") from e
azure.iot.device.exceptions.ClientError: Unexpected failure

This is the function causing the error:

async def upload_file(self, path_to_file: str, blob_name: str):
        """Upload a file to Azure Blob Storage."""
        try:
            await self.semaphore.acquire()
            try:
                blob_info = await self.device_client.get_storage_info_for_blob(blob_name)
                sas_url = "https://{}/{}/{}{}".format(
                    blob_info["hostName"],
                    blob_info["containerName"], 
                    blob_info["blobName"],
                    blob_info["sasToken"]
                )
                logger.info(
                    f"Uploading file {path_to_file} to blob storage as {blob_name}")
                with BlobClient.from_blob_url(sas_url) as blob_client:
                    with open(path_to_file, "rb") as f:
                        result = blob_client.upload_blob(f, overwrite=True)

                logger.info(f"Upload complete for {path_to_file}")
                self.device_client.notify_blob_upload_status(blob_info['correlationId'], True, 200, f'OK: {path_to_file}')
                return (True, result)
            finally:
                self.semaphore.release()
        except FileNotFoundError as e:
            # catch file not found and add an HTTP status code to return in notification to IoT Hub
            logger.error(f'upload_file FileNotFoundError: {e}')
            logger.error(traceback.format_exc())
            e.status_code = 404
            self.device_client.notify_blob_upload_status(blob_info['correlationId'], False, e.status_code, f'{e}')
            return (False, e)
        except AzureError as e:
            # catch Azure errors that might result from the upload operation
            logger.error(f'upload_file AzureError: {e}')
            logger.error(traceback.format_exc())
            self.device_client.notify_blob_upload_status(blob_info['correlationId'], False, e.status_code, f'{e}')
            return (False, e)
        except Exception as e:
            # catch all other errors
            logger.error(f'upload_file Exception: {e}')
            logger.error(traceback.format_exc())
            self.device_client.notify_blob_upload_status(blob_info['correlationId'], False, e.status_code, f'{e}')
            return (False, e)

It seems to be an issue with device_client.get_storage_info_for_blob. I know there are a max of 10 concurrent file uploads per device, but I am uploading 7 files, then they finish uploading successfully before I try again. Even so, with the semaphore it should prevent any concurrent file uploads. That is unless I'm missing something about how my code is working.

Is something not being released properly, or not able to complete properly?

[cartertinney]

@L-PDS
This sounds like an issue with the Storage SDK (BlobClient) rather than the IoT Device SDK. The Device SDK is simply reported the error it is receiving from IoTHub - that there are too many devices (presumably, too many devices uploading files based on what you've said). As such, if there's a discrepancy between how many devices you think there are uploading, and how many are reported to be uploading, that seems like an issue with the client that actually does the uploading - the Storage SDK.

[L-PDS]

That makes sense, I'll move this issue to a storage sdk issue

[fred-overflow]

That makes sense, I'll move this issue to a storage sdk issue

@L-PDS Did you actually move it to the storage sdk? I couldn't find it there, but would be interested to see if you got a response there as I am facing the same issue.

[mikkelti]

I am also experiencing an issue with this. It seems my program is allowed to call the get_storage_info_for_blob function 10 times, but the 11th time it fails with the TooManyDevicesError.