stdbuf -oL tee
Opened this issue · 6 comments
Hello,
I noticed that tee is missing stdbuf in the output. Then, the pipeline stucks with no error after some time. Sometimes shortly after beginning, sometimes later.
I'm attaching 2 pipeline.sh scripts:
FIXED.sh.txt -- this one is fixed. I edited it to insert stdbufs. It never stucks.
This is a wrong, current piplener output. Stucks after some time.
WRONG.pipeline.sh.txt
Also, there there has to be stdbuf -o0 if the commands input and output is binary (it means 0 bytes buffer), and stdbuf -oL if it is text (1 full line buffer).
Maybe it would work with -o0 anytime, but I'm not sure.
./auto_switcher.py is an empty py script for now, I'm going to implement it.
my.py.txt
This script is generating it.
@Gldkslfmsd Hi, sorry for the late response. I have pushed a fix to the main branch, can you please confirm it works for you now?
Hi,
it's better, but there are still some tees without stdbuf:
stdbuf -oL nc -lk 5001 2> >(ts '[%Y-%m-%d %H:%M:%S]' > /pwd/logs/$DATE/00-cs_audio_rec.err) | tee >(while ! ss -ltn |
I can't re-run the pipeline, it takes too much time to pull, restart the workers etc. I only looked at the code.
You can check cruise-control/pipeliner-setups/202109-dominik whether it runs for couple of minutes without stuck. If yes, then stdbuf might be the problem.
(Don't forget to remove the committed cruise-control/pipeliner-setups/202109-dominik/pipeliner.py with my simple temporary fix.)
Pushed another commit