Question: integrating/averaging?

[PenelopeFudd]

I'm running one of the examples (plotting how much data is received on the wlan0 network interface), and unfortunately the graph looks like grass, and it's impossible to figure out what the average bandwidth is (for example). When I've had this problem on AWS CloudWatch graphs, the solution has been to increase the period between samples, and the grass turns into a nice smooth line. In this example, changing the 'sleep 1' to 'sleep 60' or something higher will do that, but this only works because the /proc/net/dev data is a running total.

        $ while true; do sleep 1; cat /proc/net/dev; done |
          gawk '/wlan0/ {if(b) {print $2-b; fflush()} b=$2}' |
          feedgnuplot --lines --stream --xlen 10 --ylabel 'Bytes/sec' --xlabel seconds

I've seen a Gnuplot demo for getting a running average, can we use that with feedgnuplot somehow?

Thanks!

[dkogan]

Hi. feedgnuplot currently doesn't have direct control over the "using" directive, which is what you'd need here. I can implement it, but I've been hesitant to do it (and usually you don't NEED it). Probably I will do that in the near future, though.

In this case, I'm not convinced that you want feedgnuplot to compute the averages. I think feedgnuplot should do the plotting, and the data processing should happen before; with something like the "datamash" tool for instance. datamash doesn't currently support running averages, it looks like, but it would be great if it did. In the meantime, you can do moving averages by placing perl (for instance) in the loop:

while true; do sleep 1; cat /proc/net/dev; done |
  gawk '/wlp3s0/ {if(b) {print $2-b; fflush()} b=$2}' |
  perl -M'List::Util sum' \
       -nE 'BEGIN {$N=5; $|=1; $,=" ";} chomp; push @l, $_; shift @l if @l > $N; say $_,sum(@l)/scalar(@l);' |
  feedgnuplot --lines --stream --xlen 20 --ylabel 'Bytes/sec' --xlabel seconds \
              --legend 0 "Throughput (KB/s)" \
              --legend 1 "Running average over 5 samples"

I'll leave this open since I probably will just add direct "using" support in the not-so-distant future.

[dkogan]

I just implemented generic --using expressions in feedgnuplot, so you can use the logic in that demo directly:

while true; do sleep 1; cat /proc/net/dev; done |
gawk '/wlp3s0/ {if(b) {x=$2-b; print x,x,x; fflush()} b=$2}' | \
feedgnuplot \
    --cmds 'samples(x) = $0 > 4 ? 5 : ($0+1)' \
    --cmds 'avg5(x) = (shift5(x), (back1+back2+back3+back4+back5)/samples($1))' \
    --cmds 'shift5(x) = (back5 = back4, back4 = back3, back3 = back2, back2 = back1, back1 = x)' \
    --cmds 'back1 = back2 = back3 = back4 = back5 = sum = 0' \
    --legend 0 'input data' \
    --using  1 '1:(avg5($2))' \
    --legend 1 "running mean over previous 5 points"  \
    --using  2 '1:(sum = sum + $2, sum/($1))' \
    --legend 2 "cumulative mean" \
    --lines --stream --ylabel 'Bytes/sec' --xlabel seconds

This is still ugly, and I wouldn't do it this way, but you now can. The using expressions are generically useful in all sorts of other ways, though. A note about the sample above: I removed the --xlen. Otherwise that was interfering with the running averages. I can fix it, but you get the point