codingjoe/django-s3file

Adding exception to certain files?

Closed this issue · 7 comments

dearandrewkim commented

Hello Joe,

I have been testing this in my staging production and it is working very nicely for most of the files. However, I have some ImageFields which is also using another package (django-image-cropping) and it is not playing nicely with django-s3file. It is being uploaded to tmp folder but not being moved across to the proper folder and therefore django is unable to link to the image

I was wondering whether if it's possible to provide exception to certain models, can we add like a decorator?

Thanks.

codingjoe commented

Hi @dearandrewkim,

Thanks for reaching out. I am not 100% sure that I can solve the issue alone, as this seems to be an issue with django-image-cropping. Let's get from help from @anrie (ping) on this.

So all this package does, is bypass the upload. This is done in the frontend and via a Django middleware.
The main difference is, that the request.FILES now holds a File-object instead of a UploadedFile-object.

I don't really know why django-image-cropping, doesn't move the files. I quickly browsed through the repository and couldn't find any obvious reason, why it should behave any different from the build in file field.
I am also the maintainer of django-stdimage and I have honestly never experienced the problem you are describing.

Anyhow, maybe @anrie can illuminate the discussion further. Meanwhile, I could create a patch, that converts the S3 file back into a UploadedFile-object, see if that makes any difference? What do you think?

Best,
Joe

dearandrewkim commented

Thanks for your reply @codingjoe. I will have a look with my team.

Meanwhile, it would be good to have simple decorator or a very simple way to ignore direct uploads for certain models.

I also use a package called "django-cleanup" which automatically deletes files when the file field is changed. They have a simple decorator @cleanup.ignore which works very well for me.

For my project, the ones which are using django-image-cropping are profile images and banner images that people can upload so it's no big deal if it doesn't directly upload to s3.

Thank you

codingjoe commented

Hi @dearandrewkim,

What you are looking for is already possible. See the README only shows the easy "all-in" method. But you can also enable the functionality individually. You will need to remove s3file from your INSTALLED_APPS setting and create your own widget:

from django import forms

from s3file.forms import S3FileInputMixin


class S3FileInput(S3FileInputMixin, ClearableFileInput):
    """File input widget that uploads files directly to AWS S3."""

The ignore or opt-out idea, hold too many pitfalls IMHO. It is less explicit and maybe not so zen.

For my project, the ones which are using django-image-cropping are profile images and banner images that people can upload so it's no big deal if it doesn't directly upload to s3.

That highly depends on your server setup. Ideally your proxy would deny any large upload body to protect your application servers.

All that being said, I'd really like to fix the problem itself. I would be super cool, if you could provide me with a small sample app, that experiences the error for me to try to fix it.

Best,
Joe

  • 👍1
dearandrewkim commented

Thanks for the tips above! I will try creating my own widget.
I will work on a sample app this weekend and try to replicate the bug. I'll report back.

dearandrewkim commented

I created an app using django-image-cropping and s3 file but unfortunately I was unable to recreate the issue. They actually work fine together. I can share this with you if you want.

On my side, I've tried to replicate this issue and in the terminal it says the following:

Internal Server Error: /__s3_mock__/
Traceback (most recent call last):
  File "/home/andrew/projects/konsta2/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/andrew/projects/konsta2/venv/lib/python3.8/site-packages/s3file/middleware.py", line 23, in __call__
    return views.S3MockView.as_view()(request)
  File "/home/andrew/projects/konsta2/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/andrew/projects/konsta2/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/andrew/projects/konsta2/venv/lib/python3.8/site-packages/s3file/views.py", line 47, in post
    '<?xml version="1.0" encoding="UTF-8"?>'
  File "/home/andrew/projects/konsta2/venv/lib/python3.8/site-packages/django/conf/__init__.py", line 84, in __getattr__
    val = getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'AWS_STORAGE_BUCKET_NAME'
[26/Sep/2020 10:34:24] "POST /__s3_mock__/ HTTP/1.1" 500 97442

I don't know if this is of any help.

I'm happy to share my actual project repo with you if you want but may be irrelevant to this issue and package.
I'll provide update here if I am able to recreate this issue on the smaller app.

I have yet to create my own widget. I'll update here how I go.

anrie commented

AttributeError: 'Settings' object has no attribute 'AWS_STORAGE_BUCKET_NAME'

What happens if you define this setting?

And although the issue seems resolved I would point to these issues, especially the THUMBNAIL_DEFAULT_STORAGE setting.

Cheers and good luck!
Andreas

dearandrewkim commented

Hi @anrie , thanks for this, I'll try implementing this.
@codingjoe , we created the widgets and it works beautifully now for the files we want to direct upload. It might be good if you could add this to the README.md. Thanks all

  • ❤️1