DRF return empty list in results randomly, but count still correct

[yothinix]

When I'm trying to use the custom filter it's working all the time in local development, but when we deploy to Production (Docker Swarm) We found an issue that sometimes the API response is returned empty results randomly, but the count is correct. Below is the example results from the API

API Response

{
    'count': 1, 
    'next': 'http://localhost:8000/api/somethings/?email=test%40example.com&limit=0&offset=0', 
    'previous': None, 
    'results': []
}

Right now we need to restart a uWSGI service (By restarting docker swarm for this service) and the problem is fixed for a moment and randomly happen again.

Here is our DRF view

class SomeView(ListCreateAPIView):
    queryset = SomeModel.objects.all()
    serializer_class = SomeModelSerializer
    filter_backends = (OrderingFilter, DjangoFilterBackend)
    filter_class = CustomFilter
    ordering = ('id',)

    def list(self, request, *args, **kwargs):
        if request.GET.get('all', None):
            # Do something
            serializer = self.get_serializer(queryset, many=True)
            return Response(serializer.data)
        else:
            return super(SomeView, self).list(self, request, *args, **kwargs)

Here is our CustomFilter

from django_filters.rest_framework.filters import CharFilter
import rest_framework_filters as filters


class CustomFilter(filters.FilterSet):
    json_field_separator = '___'

    json_field_is = CharFilter(name='json_field', method='filter_json_field')
    json_field_is_not = CharFilter(name='json_field', method='exclude_json_field')

    def split_lookup_field(self, value):
        return dict(field.split(':') for field in value.split(self.json_field_separator))

    def filter_json_field(self, queryset, name, value):
        try:
            lookup_field = self.split_lookup_field(value)
            return queryset.filter(**lookup_field)
        except (ValueError, FieldError):
            return queryset.none()

    def exclude_json_field(self, queryset, name, value):
        try:
            lookup_field = self.split_lookup_field(value)
        except (ValueError, FieldError):
            return queryset.none()

        for query_arg, query_value in lookup_field.items():
            queryset = queryset.exclude(**{query_arg: query_value})

        return queryset

    class Meta:
        model = SomeModel
        exclude = ['image', 'json_field']

Here is a version of the Package we use for this project

Django==1.10.8
djangorestframework==3.6.4
django-filter==1.0.4
djangorestframework-filters==0.10.2

We suspect that this is possible related to djangorestframework-filters since the only change before this error occured is when we implement CustomFilter above, Any idea?

UPDATE: The problem seems to be reduce, but still occured once we using DjangoFilterBackend from rest_framework_filters.backends package instead of django_filters

[rpkilby]

I believe this is related to #134 (comment). Please upgrade to django-filter 1.1.0, and let me know if that fixes your issue.

[yothinix]

@rpkilby We just update django-filter to 1.1.0 and the problem still happen and seems to be more than before we upgrade. Is it possible that when API received load of requests would cause this problem?

[rpkilby]

It doesn't look like you're using any features specific to this package. I would see if you can reproduce this with just django-filter. e.g., inherit from django_filters.FilterSet and use the django_filters.rest_framework.DjangoFilterBackend.

[yothinix]

@rpkilby Sorry for missing for 3 days, but we want to verify the results. It looks like the problem is not related to this package at all as you suggest. It look like that when we override pagination limit the pagination value spread across the request which is causing the problem above and I can reproduce it via unit test.

class SomeView(ListCreateAPIView):
    queryset = SomeModel.objects.all()
    serializer_class = SomeSerializer
    filter_backends = (OrderingFilter, DjangoFilterBackend)
    filter_class = CustomFilter
    ordering = ('id',)

    def list(self, request, *args, **kwargs):
        if request.GET.get('all', None):
            queryset = self.filter_queryset(self.get_queryset())
            self.pagination_class.default_limit = queryset.count()  # The problem is this line

            page = self.paginate_queryset(queryset)
            if page is not None:
                serializer = self.get_serializer(page, many=True)
                return self.get_paginated_response(serializer.data)

            serializer = self.get_serializer(queryset, many=True)

            return Response(serializer.data)
        else:
            return super(SomeView, self).list(self, request, *args, **kwargs)

I may close this issue now since it now solved, Thanks for helping.