Getting/Setting occasionally extremely slow

[siovene]

Hello,

I run a Django website that gets about 3 million requests per day, and I recently started using Sentry to monitor the site's performance and find optimization opportunities.

I noticed that when a certain route was being slow, it was because of a Redis GET or SET taking several hundred milliseconds or even whole seconds, which I found very odd.

My website is hosted on AWS, and the latency between my EC2 instances and my Elasticache cluster is in the order of a few microseconds. I contacted AWS Support to see if they could help, and we examined the Redis logs and saw that there were no slow requests logged. We also conducted a latency test that showed everything was in order.

My first thought was that this could be a serialization/deserialization issue, but this happens also when getting/setting a value with only a few bytes in it, or something atomic like a boolean.

According to Sentry, 50% of my redis requests take less than 0.4 ms (great) but the slowest 1% are slower than 177ms (not great). And I can easily see many that take several seconds.

Now I don't know exactly how Sentry is measuring this, but the same is confirmed by using a second APM tool: Scout APM. This makes me think that something is indeed wrong.

These are the relevant parts of my django-redis configuration:

    CACHES = {
        'default': {
            'BACKEND': 'django_redis.cache.RedisCache',
            'LOCATION': os.environ.get('CACHE_URL', 'redis://redis:6379/1').strip(), # <-- this is my AWS Elastiache Redis endpoint
            'OPTIONS': {
                'CLIENT_CLASS': 'django_redis.client.DefaultClient',
                'PICKLE_VERSION': 2,
                'SERIALIZER': 'astrobin.cache.CustomPickleSerializer',
                'PARSER_CLASS': 'redis.connection._HiredisParser',
                'CONNECTION_POOL_KWARGS': {
                    'max_connections': 100,
                },
            },
            'KEY_PREFIX': 'astrobin',
            'TIMEOUT': 3600,
        },
    }

import six
from django.utils.encoding import force_bytes
from django_redis.serializers.pickle import PickleSerializer

try:
    import cPickle as pickle
except ImportError:
    import pickle

class CustomPickleSerializer(PickleSerializer):
    def loads(self, value):
        if six.PY3:
            return self._loads_py3(value)
        return super().loads(force_bytes(value))

    def _loads_py3(self, value):
        return pickle.loads(
            force_bytes(value),
            fix_imports=True,
            encoding='latin1'
        )

Am I doing something obviously wrong? Is there something you can suggest I experiment with to try and figure this out?

Here's my versions:

django-redis==5.2.0
hiredis==2.3.2
redis[hiredis]==5.0.1

Thanks!
Salvatore

[WisdomPill]

hello @siovene, if you're looking for performance I would strongly recommend to move away from pickle, maybe plain json would be a good. But if you want to take it further then my suggestion would be to try orjson, not only for redis but also for rendering if you're using django-rest-framework it would be fairly easy.

In some cases you could also decide to not serialise at all.

I do not know how Scout APM works, I have never used it, but it would be nicer to get more insights into which method is taking long.

Imho sentry does do a good job when it comes to performance measurements, I prefer using new relic for that.
Sentry can be misleading because of the number of traces it analyses and it does not do a good job when trying to understand the big picture especially with multiple services.

[siovene]

Hi @WisdomPill,

I appreciate the tip, I will try JSON or orson. However, do you really think that pickling would be so slow that occasionally it takes 5 seconds to deserialize a 20 byte strings?

[WisdomPill]

it depends on the complexity of the class if you're serialising a class... if you're serialising basic objects (int, string and so on) I would maybe even go so far to not even serialise in order to see if there are any differences.

[siovene]

Thanks @WisdomPill.

Is there a way to specify this on a per-key basis? I just looked around my code base and I cache some simple values, some python dictionaries, but also some entire Django models or even querysets. And I also cache template fragments.

[WisdomPill]

then I would definitely drop pickle and write serializers for your querysets. you could give a try to CacheRouter or use caches directly