Send the data in batches
Opened this issue · 13 comments
It would be a better way to get all the data from the bGeigeiNano in batches to the api and not average (as now) calculated over 30 seconds or 5 minutes.
Maybe worth mentioning that this came up due to finding some odd gps points from @jamoross ’s geigiecast.
I can post a kibana link later but it looks like the device averaged some valid and invalid gps coordinates and sent us 0.003,0.009 as the location.
It’d be good if we can only send the same points that show up in the log, perhaps filtered to only points with reasonable gps confidence.
Here's one of the points in question http://kibana.safecast.cc/app/kibana#/doc/ingest-measurements-*/ingest-measurements-2021-02-20?id=UFd7vXcBj2uUTIA-1wb7&_g=(refreshInterval:(pause:!t,value:0),time:(from:'2021-02-12T12:28:03.937Z',to:'2021-02-24T05:23:17.620Z')), see https://api.safecast.org/en-US/bgeigie_imports/51022 for the SD log.
{
"device_urn": "geigiecast:62025",
"device_class": "geigiecast",
"device": 62025,
"when_captured": "2021-02-20T03:28:47Z",
"loc_lat": 0.00253,
"loc_lon": 0.00944,
"loc_olc": "6FG22235+2Q8",
"lnd_7318u": 104,
"dev_test": true,
"service_uploaded": "2021-02-20T03:29:05Z",
"service_transport": "geigiecast:172.58.27.225",
"service_handler": "i-051cab8ec0fe30bcd",
"@timestamp": "2021-02-20T03:28:47Z",
"ingest": {
"location": {
"lat": 0.00253,
"lon": 0.00944
}
}Which would be here, way of the actual drive path.
That is probally a rounding error for "NO GPS" (meaning 0,0) and a valid location.
$BNRDD,2025,2021-02-20T03:28:17Z,106,16,10479,A,3742.1517,N,14044.9053,E,468.20,A,7,124*77
$BNRDD,2025,2021-02-20T03:28:22Z,107,8,10487,A,3742.1082,N,14044.9270,E,467.50,A,7,124*4C
$BNRDD,2025,2021-02-20T03:28:27Z,104,4,10491,A,3742.0719,N,14044.9486,E,465.50,A,7,124*48
$BNRDD,2025,2021-02-20T03:28:32Z,101,10,10501,A,3742.0514,N,14044.9806,E,460.50,A,8,107*74
$BNRDD,2025,2021-02-20T03:28:37Z,106,9,10510,A,3742.0405,N,14045.0310,E,457.40,A,8,114*4C
$BNRDD,2025,2021-02-20T03:28:42Z,104,7,10517,A,3742.0237,N,14045.0877,E,454.70,A,9,97*73
$BNRDD,2025,2021-02-20T03:28:47Z,105,5,10522,A,3742.0035,N,14045.1421,E,452.50,A,9,107*41
It's interesting looking at the log leading up to 2021-02-20T03:28:47Z though. I don't see any invalid gps markers (should be "V") according to https://github.com/Safecast/bGeigieNanoKit/wiki/Operations-Manual#data-log
Any idea how the 0 might have reached the geigiecast for averaging?
There are some V in the log, but not quite around that point and not with 0 GPS either
$BNRDD,2025,2021-02-20T03:22:26Z,67,9,9855,A,3744.1093,N,14044.1753,E,403.30,V,8,108*57
$BNRDD,2025,2021-02-20T03:34:17Z,66,3,10932,A,3741.1886,N,14047.7545,E,488.60,V,5,177*6B
$BNRDD,2025,2021-02-20T03:34:22Z,62,2,10934,A,3741.1886,N,14047.7545,E,488.60,V,5,177*6E
$BNRDD,2025,2021-02-20T03:34:27Z,62,5,10939,A,3741.1886,N,14047.7545,E,488.60,V,5,177*61
$BNRDD,2025,2021-02-20T03:34:32Z,58,2,10941,A,3741.1886,N,14047.7545,E,488.60,V,5,177*64
$BNRDD,2025,2021-02-20T03:35:12Z,61,0,10990,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6D
$BNRDD,2025,2021-02-20T03:35:17Z,60,2,10992,A,3741.3162,N,14048.3816,E,489.10,V,3,482*69
$BNRDD,2025,2021-02-20T03:35:22Z,61,3,10995,A,3741.3162,N,14048.3816,E,489.10,V,3,482*68
$BNRDD,2025,2021-02-20T03:35:27Z,57,1,10996,A,3741.3162,N,14048.3816,E,489.10,V,3,482*69
$BNRDD,2025,2021-02-20T03:35:32Z,56,1,10997,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6D
$BNRDD,2025,2021-02-20T03:35:37Z,49,2,10999,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6B
$BNRDD,2025,2021-02-20T03:35:42Z,42,2,11001,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6B
$BNRDD,2025,2021-02-20T03:35:47Z,36,3,11004,A,3741.3162,N,14048.3816,E,489.10,V,3,482*69
$BNRDD,2025,2021-02-20T03:35:52Z,29,3,11007,A,3741.3162,N,14048.3816,E,489.10,V,3,482*60
$BNRDD,2025,2021-02-20T03:35:57Z,29,3,11010,A,3741.3162,N,14048.3816,E,489.10,V,3,482*63
$BNRDD,2025,2021-02-20T03:36:02Z,25,1,11011,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6F
$BNRDD,2025,2021-02-20T03:36:07Z,23,2,11013,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6D
$BNRDD,2025,2021-02-20T03:36:12Z,24,1,11014,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6A
$BNRDD,2025,2021-02-20T03:36:17Z,24,2,11016,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6E
$BNRDD,2025,2021-02-20T03:36:22Z,27,6,11022,A,3741.3162,N,14048.3816,E,489.10,V,3,482*68
$BNRDD,2025,2021-02-20T03:36:27Z,29,3,11025,A,3741.3162,N,14048.3816,E,489.10,V,3,482*61
$BNRDD,2025,2021-02-20T03:36:32Z,33,5,11030,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6C
$BNRDD,2025,2021-02-20T03:36:37Z,31,0,11030,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6E
$BNRDD,2025,2021-02-20T03:36:42Z,32,3,11033,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6F
$BNRDD,2025,2021-02-20T03:36:47Z,30,1,11034,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6D
$BNRDD,2025,2021-02-20T03:36:52Z,29,2,11036,A,3741.3162,N,14048.3816,E,489.10,V,3,482*60
$BNRDD,2025,2021-02-20T03:36:57Z,29,3,11039,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6B
$BNRDD,2025,2021-02-20T03:37:02Z,30,2,11041,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6C
$BNRDD,2025,2021-02-20T03:37:07Z,29,1,11042,A,3741.3162,N,14048.3816,E,489.10,V,3,482*61
$BNRDD,2025,2021-02-20T03:37:12Z,34,6,11048,A,3741.3162,N,14048.3816,E,489.10,V,3,482*64
$BNRDD,2025,2021-02-20T03:37:17Z,36,4,11052,A,3741.3162,N,14048.3816,E,489.10,V,3,482*6A
$BNRDD,2025,2021-02-20T03:37:22Z,32,2,11054,A,3741.3162,N,14048.3816,E,489.10,V,3,482*68
$BNRDD,2025,2021-02-20T03:39:41Z,102,4,11246,A,3741.3228,N,14050.2430,E,265.00,V,6,138*5F
If it's possible it'd be great to have the geigiecast post the same data we'd find in a log. If every 5s will drain too much battery, then perhaps we could send the data in batches.
For now I think we can probably just ignore stuff stuff with lat/lon in the 0-0.01 range when making graphs and maps.
Hrm... also curious is that we seemed to get basically no data from the geigiecast around that time into ingest:
I must have just had something weird going on in my browser for that last screenshot. Data is there, here's the section with the weird point that reached ingest.
Here's a CSV incase it's helpful (zipped for github)
The bGeigieCast was not designed to be a life streaming thing. It was made to be equivalent to the pointcast. That is why the 5 minutes come in. The 30 seconds were made for testing only, but I used it as a kind of live streaming.
Yeah, that's understandable. I think it still works pretty well for both purposes and filtering after the fact isn't hard.