Slow parallel upload using blob_client::upload_block_blob_from_buffer
BenjaminLanfer opened this issue · 1 comments
When trying to upload especially smaller files using blob_client::upload_block_blob_from_buffer I find that upload speed is very slow (~500kbit/s). It appears that only a single parallel request is used although the maximum number of concurrent requests was set to 64.
From a cursory glance at the code, a rather large block size is chosen. That is, the minimum block size is set to azure::storage_lite::constants::default_block_size = 8 * 1024 * 1024, which is 8MB. Also, the grain size is set to around 4MB, where it is only 64kB in blob_client::download_blob_to_buffer.
Would it be feasible to set a smaller block size to allow for more blocks being uploaded in parallel even for smaller files?
Thanks for looking into this.
The block size was originally chosen to enable LBB feature that can ultimately boost performance at the service side. This is considered a feature request to expose the option to specify the block size to a smaller value and is put to our backlog.
In the meantime, you can check the track 2 Storage SDK which has similar dependencies with CPPLite here and will be more actively maintained: https://github.com/Azure/azure-sdk-for-cpp/tree/master/sdk/storage
The block size can be customized in the mentioned track 2 SDK.