Query timing changes data reported
Opened this issue · 4 comments
The 5-min offset for data retrieval, combined with only hourly updates causes RPi-Monitor to interpolate the Status values, which makes decoding the flags very difficult.
Not sure how long the EBAM takes to generate an hourly report, but probably <5min.
To complicate data retrieval timing, the EBAM clock drifts independently of the Pi and needs to be accomodated.
Timing is hard here: to obtain "correct" timestamps, data must be obtained from EBAM, written to file, and queried by RPi-Monitor within the first minute of each hour.
To consider:
- unknown synchronicity between EBAM & RPi clocks
- unknown data record generation time of EBAM
- query rate/offset of the RPi-Monitor service
Current plan:
- establish new cron-based service to keep EBAM clock synced to Pi
- change hourly DAQ script to query for new records, not just the last record
- launch at top of each hour (hopefully to finish within first minute)
- retry several times until new record obtained
- (possibly) change RPi-Monitor query rate for EBAM data
10s
→1min
?- may require 10s query rate to prevent 0:00-data from getting 0:01 timestamp
- may require 60s query rate to prevent inter-minute averaging of values (which then become nonsense values)
Examples of the timing mismatch producing nonsense values: -32.195 in this case, where it should be either 4 or -99. (XREF XavierBerger/RPi-Monitor#276)
unclear why status isn't updated until after first minute of hour (EBAM or python script?)- cron currently triggered at minute 1, not minute 0 - will fix!
- interpolation at point 10:01:10 is not linear between adjacent ports (why?)
- note RPi-Monitor query timing is aligned to zero-seconds (e.g. :00, :10, :20, :30, ...)
- assuming EBAM & computer have same clock time, it's not possible to keep RPi-Monitor from including previous data in new minute's average (:00 will always hold old data)
- unclear why status isn't updated until after first minute of hour (EBAM or python script?)
- cron currently triggered at minute 1, not minute 0 - will fix!
Calling data script at minute 0 instead of 1, with 5 second query delay, does not produce latest data record because of EBAM clock offset (10:59:49 vs 11:00:05 = 16sec behind):
lar@ebam-plus:~/ebam-plus $ ls -l /tmp
total 16
-rw-r--r-- 1 root root 199 Jul 12 11:00 ebam.csv
tail -n 10 /home/lar/ebam-plus/src/getbam.log
Data Report
2019-07-12 10:59:49
Location, 3, W15096
Time,ConcRT(ug/m3),ConcHR(ug/m3),Flow(lpm),WS(m/s),WD(Deg),AT(C),RH(%),BP(mmHg),FT(C),FRH(%),BV(V),PM,Status
2019-07-12 10:00:00,+031663,+099999,+11.6,00.0,000,+024.5,041,694,+033.2,017,12.5,0,00512
4
gooddata = 1
New hour saved successfully.
Best (?) solution: set EBAM clock to intentionally run ~10sec ahead of RPi, so that data record is ready by start of new minute (xref #9)
- prevent :00 values from holding-over old data
- maintains sufficient time-synchronicity (i.e. constant 10s lead time is totally OK for hourly data sets)
- does not pollute :40 or :50 values because data queried at most minutely