skoczen/django-ajax-uploader

Memory usage with bigger files

Closed this issue · 16 comments

deschler commented

Uploading huge files seems to be problematic. I've tried to upload a 3GB file and the apache wsgi process slowly took up the entire 12GB RAM of the server. Using the cancel link didn't stop it and i had to restart apache in order to prevent further swapping.

Now i understand that http was never designed to handle huge uploads, but i'm wondering why the process eats up that much of memory. While the file is being uploaded, the RAM usage slowly increases by the amount of the filesize, which is fine i think. After it reaches 100% upload process however, the memory usage goes up further and the file is not written.

Using the django dev server, this behaviour is even more visible. Here a file of about 200MB is enough to make my machine with 4GB RAM swap.

I override upload_complete in my custom Backend which moves the uploaded file to another location using shutil.move. I first thought this was the problem, but it's the same when i use the included LocalUploadBackend.

skoczen commented

Hey,

Which backend is this with? I've uploaded 2GB files to S3 without trouble, on both dev server and live using Gunicorn.

Any more information you can provide on setup would be great.

deschler commented

It's the included LocalUploadBackend, resp. a subclass of it. But it happens with both.

Setup is Python 2.6, Django-1.3.1, Apache with mod_wsgi in daemon mode on Debian Squeeze.

sultaniman commented

The same happened today in my project served by gunicorn and nginx when tried to upload file 2 gigs

skoczen commented

imanhodjaev: what backend?

deschler: Thanks for the info. Will try to reproduce here when I get time. Pull requests are always welcome if you figure it out before I do :)

skoczen commented

If I had to guess on this, I'd say that it sounds like memory is being reserved for each chunk, iteratively, (5, 10, 15, 20, 25, etc) and not freed. I don't see that issue directly in the code, and haven't found time to do a deep dive.

If this is still bugging you and you have time to take a look, your ideas are appreciated!

sultaniman commented

I guess upload issues were due to the FILE_UPLOAD_MAX_MEMORY_SIZE option.
We know Django handles uploads with MemoryUploadHandler which fallbacks to TemporaryFileUploadHandler in case if file size is out of memory. So I just set FILE_UPLOAD_MAX_MEMORY_SIZE to 350M.

sultaniman commented

For some reason I don't know for a while AjaxFileUploader works very slowly with default upload backend (LocalUploadBackend) I just follow default tutorial described here

davelowe commented

When I try using this, memory gets eaten up like bananas in a monkey pit (assuming monkeys live in pits, that is). In Activity Monitor on my Mac I watch the CPU jump up and then drop after the upload is completed, but the Real Memory number just keeps going up. Before upload, it's at 30.1MB. After uploading a 40MB file, it's now sitting at 327.6MB.

I tried a 600MB file earlier and it froze up my computer for a time. This is with the default_storage backend, with Django's MemoryFileUploadHandler in and out of the FILE_UPLOAD_HANDLERS setting, and setting to FILE_UPLOAD_MAX_MEMORY_SIZE a small and a large number.

skoczen commented

Dave:

Two questions:

  1. What server are you running? The built-in test server, or a proper one (gunicorn, etc)?
  2. What ajax-upload backend?

My main production use of ajax-uploader is on an EC2 micro (512 MB ram), storing to s3. I'm able to upload multi-gig files, no problem, so I'd guess it's either the test env you're running (server, etc), or something in Django's localstorage backend.

davelowe commented
  1. Both locally on the built-in Django server and on a proper nginx server.
  2. I tried two but most recently I used default_storage. I have to use the local filesystem for the project I'm using this for.
skoczen commented

Hey Dave,

  1. nginx - via mod_wsgi, or are you proxying to a django server?
  2. default_storage falls back to django's default storage. What's django's storage backend set to?
davelowe commented

Hey Steven,

  1. Via mod_wsgi
  2. The storage is the default
skoczen commented

Hey Dave,

Thanks for the info. I haven't tested mod_wsgi (I typically proxy_pass), but I'd bet it's fine. I'll try to replicate your issue when I get a chance - as a warning, it may be a bit, things are a little crazy right now.

Thanks,
Steven

davelowe commented

I appreciate it, Steven, thanks for your help!

ericmarcos commented

Same problem here, with both django server and apache+mod_wsgi with local storage. Changing FILE_UPLOAD_MAX_MEMORY_SIZE doesn't make any difference. My setup is Ubuntu 12.04, Python 2.7 and Django 1.4.5.
Last comment is from a year ago, did anyone find a solution to this problem?

skoczen commented

Doing a catch-up clean-out of issues on ajaxuploader. Looks like this one stalled, so I'm closing it. Please re-open if that's not right!