codingjoe/django-s3file

Large file greater than 5GB

Closed this issue · 4 comments

iiji-ez commented

Hi,
Thanks for awesome project.

Basically, I can upload file to s3 properly. :)
But, I am facing bad request error when I upload large file >5GB.
I checked developer mode on browser. PUT request to s3 was only one.
It seems django-storages supports multipart uploading.

Can I upload grater than 5GB file with django-s3file?

codingjoe commented

Hi @iiji-ez,

Thank you for reaching out. Yes, you can upload extremely large files. In this case you need to use a different storage backend tho. Sadly, the default django-strages backend will load the file into your app server. Even if your app server is inside an AWS server, this might cause problems. To avoid that we just recently added our own storage backend, that prevents that problem from happening, see also https://github.com/codingjoe/django-s3file#using-optimized-s3boto3storage

I hope this solves your Problem!
Best
Joe

iiji-ez commented

Hi @codingjoe, thanks for your comment.
My question isn't regarding issue #125.
The storages_optimized backend as you mentioned is working fine less than 5GB.

Yes, you can upload extremely large files.

Let me ask again.

  • Does django-s3file supports greater than 5GB? Please let me know configuration if it is supported.
  • Should I need to improve taking care multipart uploading?

Just FYI)
https://aws.amazon.com/s3/faqs/

The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.

So, I thought taking care multipart uploading is needed.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

codingjoe commented

Hi @iiji-ez,

Thank you for your patience, I really did misunderstand you question. As I see it, the AWS SDK for JavaScript does come with a simple integration for multipart uploads, see also @aws-sdk/lib-storage
.
However, the question is, if we should go ahead and include the SDK here to tackle this problem. Honestly, most people doesn't seem to be an issue. Besides, as I understand it, each part would require a presigned URL.

I think this exceeds the purpose of this library. I would maybe note this as a known limitation.

Best,
Joe

iiji-ez commented

Hi @codingjoe,

However, the question is, if we should go ahead and include the SDK here to tackle this problem. Honestly, most people doesn't seem to be an issue.

I understand.
Thank you for your kindness comment.