lazywalker/DigiSkimmer

decoding queue overflow

Opened this issue · 2 comments

Seeing the following errors on a fairly constant basis:

2022-07-24 09:00:00,130 ERROR 1764 [KIWIC-20+] decoding queue overflow; dropping one file 2022-07-24 09:00:00,130 ERROR 1764 [FLYDOG-60~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,141 ERROR 1764 [RASP-160~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,141 ERROR 1764 [RASP-160!] decoding queue overflow; dropping one file 2022-07-24 09:00:00,142 ERROR 1764 [KIWIC-15!] decoding queue overflow; dropping one file 2022-07-24 09:00:00,143 ERROR 1764 [RASP-630!] decoding queue overflow; dropping one file 2022-07-24 09:00:00,144 ERROR 1764 [KIWIA-80~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,145 ERROR 1764 [KIWIC-10~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,146 ERROR 1764 [KIWIB-30+] decoding queue overflow; dropping one file 2022-07-24 09:00:00,147 ERROR 1764 [KIWIA-80!] decoding queue overflow; dropping one file 2022-07-24 09:00:00,147 ERROR 1764 [FLYDOG-40~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,148 ERROR 1764 [KIWIC-20~] decoding queue overflow; dropping one file 2022-07-24 09:00:00,148 ERROR 1764 [KIWIB-12!] decoding queue overflow; dropping one file

What I'd like to understand is how the queue works and how it decides there is an overflow. I suspect the logic is based on if it can decode everything within the cycle {i.e. 15s for FT8). In other words is the CPU going to fail to decode in time.

Is this easily tweakable? I would be willing to bet that it wouldn't run away and build up an excessive queue as it's not unsual to have a a very busy cycle(s) followed by less busy cycle(s), thus the queue would catch up within, say, two minutes or less.

Try to tweak these two values within settings.py

DECODER_QUEUE = {
    'maxsize': 10,
    'workers': 3
}

the value maxsize can set the size of queue, workers means how many decoders working at the sometime, more workers, faster queue consuming and more cpu resources will be taken.

for example, tune the maxsize to 100

Perfect thanks