Cleanup completed job in redis
minhpq331 opened this issue · 5 comments
Thank you for great library.
I can see only Purge function to clean ALL job in redis queue. What should I do if I need to clean ONLY completed job?
Is there any expire, time to live,... function available for redis job?
AFAIK you don't need any cleanup for completed jobs - they should be removed automatically.
If this is still a problem - please add more details.
Hmm, seems not. What do you mean by "AFAIK", it is handled by your (nice, btw) lib or by redis itself ?
AFAIK means that I am not using Redis backend myself - so I am not sure how it behaves in production.
lib or by redis itself
It should be handled by the lib. Do you have an idea what is not properly cleaned up?
redis streams, for performance trade-off reasons, does not delete messages from the stream automatically. The function used in this library to "Delete" messages from a redis stream is XACK. XACK does not delete a message, as in free the memory from the stream. It instead only marks the message as processed preventing it from being served to any more consumer groups. Freeing messages from the stream is done using XTRIM, the MAXLEN argument of XADD, and XDEL for single messages.
XDEL is used in this library when re-enqueuing a message as part of a "Release" action. XTRIM is currently only used to remove all messages in a "Purge" action. And so, I expect without any other intervention, a redis backed taskq will grow to fill all available memory and storage without explict action by the user to purge messages.
Using the library as is, the user will need to manually have either the producer or consumer call
XTRIM taskq:{queuename}:stream MAXLEN ~ 1000
on some regular basis, like after every x inserts, etc. The max length and interval between calls should be chosen based on the queue's usage pattern.
@james-storey thanks, I am able to reproduce the problem. Looks like just replacing XACK with XDEL is enough...