GPSU Invalid Timestamp information
time4tea opened this issue · 18 comments
I am trying to parse data stream from HERO7 Black.
However, in the data stream there appears to be a corrupt data item:
GPSU type 'U' size 16 data: "2225525525425525",
It seems unlikely that this is due to storage device data corruption, as the rest of the stream is totally fine. I guess I'm asking if there is a known issue with GPSU timestamps being emitted incorrectly from the device, or if this is a known value to indicate some kind of error condition.
I do know that in some cases the first GPS5 with a GPSF of 2 or 3 is actually invalid - seems that if the lock is acquired part way through a packet, the entrie packet is marked with a lock, but some samples correspond to a non-locked period, but here the GPSF is 0
The previous and subsequent timestamps in the data stream are just fine, however the previous GPS5 sample looks odd:
STRM nest size 264
EMPT type 'L' size 4 data: 4,
TSMP type 'L' size 4 data: 58,
STNM type 'c' size 44 data: "GPS (Lat., Long., Alt., 2D speed, 3D speed)"
GPSF type 'L' size 4 data: 3,
GPSU type 'U' size 16 data: "220617220042.050",
GPSP type 'S' size 2 data: 128,
UNIT type 'c' samplesize 3 repeat 5 data: "deg", "deg", "m", ...
SCAL type 'l' samplesize 4 repeat 5 data: 10000000, 10000000, 1000, 1000, 100,
GPS5 type 'l' samplesize 20 repeat 4 data: (redacted valid looking data)
STRM nest size 184
EMPT type 'L' size 4 data: 5,
TSMP type 'L' size 4 data: 58,
STNM type 'c' size 44 data: "GPS (Lat., Long., Alt., 2D speed, 3D speed)"
GPSF type 'L' size 4 data: 3,
GPSU type 'U' size 16 data: "220617220042.050",
GPSP type 'S' size 2 data: 128,
UNIT type 'c' samplesize 3 repeat 5 data: "deg", "deg", "m", ...
SCAL type 'l' samplesize 4 repeat 5 data: 10000000, 10000000, 1000, 1000, 100,
GPS5 type 'l' samplesize 20 repeat 0
STRM nest size 244
EMPT type 'L' size 4 data: 5,
TSMP type 'L' size 4 data: 61,
STNM type 'c' size 44 data: "GPS (Lat., Long., Alt., 2D speed, 3D speed)"
GPSF type 'L' size 4 data: 0,
GPSU type 'U' size 16 data: "2225525525425525",
GPSP type 'S' size 2 data: 0,
UNIT type 'c' samplesize 3 repeat 5 data: "deg", "deg", "m", ...
SCAL type 'l' samplesize 4 repeat 5 data: 10000000, 10000000, 1000, 1000, 100,
GPS5 type 'l' samplesize 20 repeat 3 data: 13,17788664,629855,-134217724,81, (redacted valid looking data)
STRM nest size 284
EMPT type 'L' size 4 data: 5,
TSMP type 'L' size 4 data: 66,
STNM type 'c' size 44 data: "GPS (Lat., Long., Alt., 2D speed, 3D speed)"
GPSF type 'L' size 4 data: 3,
GPSU type 'U' size 16 data: "220617220044.415",
GPSP type 'S' size 2 data: 125,
UNIT type 'c' samplesize 3 repeat 5 data: "deg", "deg", "m", ...
SCAL type 'l' samplesize 4 repeat 5 data: 10000000, 10000000, 1000, 1000, 100,
GPS5 type 'l' samplesize 20 repeat 5 data: (redacted valid looking data)
Really appreciate any insights! - Thank you!
Is there any significance that the string (spacing for clarity) is "22 255 255 254 255 25" - looks like some kind of uninitialised buffer write?
I have never seen this error. I expect you are correct in that there was some invalid data delivered from the GPS chip, or empty payload. We do have any error signalling for this, as we didn't know it could happen. Have you seen this error more than once?
Hiya - Yes - this is the second time that it has been reported to me (although it is a defect reported against my parsing software, its not my actual device) - The end user has supplied the MP4 file in the linked issue above if you'd like the whole thing.
Interestingly, the invalid GPSU data was the same in both cases.
Yes please to the sample clip.
Sure. The MP4 is in the linked issue above, along with a bit of narrative, but I'll duplicate it here: https://s3.amazonaws.com/s3.voneicken.com/GH010359.MP4
This seems to happen when I use my Hero 7 black without having the date set, i.e., it's set to some date in 2016. This in turn happens 'cause I run the camera with a USB-C power source and no internal battery: it looses the date after some hours without having any power when I remove the USB-C power to recharge the power pack.
Basically: remove battery from Hero 7 black, wait long enough for it to loose date&time (a few hours?), plug in USB-C power, switch to prefs and see that date is in 2016, start recording, observe the issue described here.
That is surprising as that time and date field is not from real time clock, it is the value reported from GPS.
It's not unbelievable though. This capture is from right at the start of the trace. It could be that after a few packets with a GPS time after a hard boot with loss of RTC, the firmware decides to do some extra work, which misses a deadline, so the output isn't captured for that timeslice. I have no idea how the gopro firmware is written, so this is pure conjecture!!
In the look at the number of samples per payload of GPS5, the GPS chip is behaving eratically. Instead of 17-19 samples per payload (averaging around 18.1Hz), the stream (linked above) has payloads of 13, 9, 0, 4, 5, 0, 3.., averaging 4.6Hz. The GPS chip is not behaving ideally. This camera likely needs a factory reset.
I'm not sure the problem is reproducible. I just took a look at the recordings I have since the first one without date&time set:
- 2022/4/16 - date&time not set, GPS track is fine
- 2022/6/17 - date&time not set, GPS track messed-up, sample linked 4 comments up
- 2022/6/18 - date&time sync'd before recording, GPS track is fine
- 2022/6/23 - date&time not set, GPS track is fine
- 2022/6/24 - date&time not set, GPS track has issues with data points out of order (by <1sec)
- 2022/6/25 - date&time sync'd, GPS track is fine
I'm concluding that if the date&time are lost there is a chance that the GPS track ends up messed-up, but it's not systematic. Presumably some race condition. I'm not sure this can be pursued productively at this point.
We also don't have any firmware releases planned for HERO7. This should not come up on more recent cameras.
Hi @dnewman-gpsw - Just FYI - I had another user report of invalid data in GPSU. time4tea/gopro-dashboard-overlay#141
I look forward to knowing the camera model this was.
Hi - The Camera was Hero 9 Black - HD9.01.01.52.00
I can see the same sort of behaviour as before, where the number of samples from the GPS chip is erratic for the whole file.
Here is some debug from my software (the GPS5 values are redacted)
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 31 35 37 2e 37 32 39] [b'230625112157.729']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 31 35 37 2e 37 32 39] [b'230625112157.729']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 4, Length Bytes=88/80
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 30 2e 35 33 34] [b'230625112200.534']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 30 2e 35 33 34] [b'230625112200.534']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 5, Length Bytes=108/100
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 31 2e 36 33 34] [b'230625112201.634']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 32 2e 35 36 39] [b'230625112202.569']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 32 2e 35 36 39] [b'230625112202.569']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 34 2e 30 35 34] [b'230625112204.054']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 34 2e 30 35 34] [b'230625112204.054']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 2, Length Bytes=48/40
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 34 2e 30 35 34] [b'230625112204.054']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 37 2e 30 32 34] [b'230625112207.024']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 37 2e 37 33 39] [b'230625112207.739']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 38 2e 35 36 34] [b'230625112208.564']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 2, Length Bytes=48/40
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 30 2e 33 37 39] [b'230625112210.379']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 9, Length Bytes=188/180
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 30 2e 33 37 39] [b'230625112210.379']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 31 2e 38 30 39] [b'230625112211.809']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 31 2e 38 30 39] [b'230625112211.809']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 9, Length Bytes=188/180
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 33 2e 37 33 34] [b'230625112213.734']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 33 2e 37 33 34] [b'230625112213.734']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 36 2e 31 35 34] [b'230625112216.154']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 37 2e 37 34 39] [b'230625112217.749']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 37 2e 37 34 39] [b'230625112217.749']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 8, Length Bytes=168/160
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 39 2e 38 33 39] [b'230625112219.839']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 39 2e 38 33 39] [b'230625112219.839']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 32 2e 32 35 39] [b'230625112222.259']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 32 2e 32 35 39] [b'230625112222.259']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 34 2e 31 38 35] [b'230625112224.185']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 2, Length Bytes=48/40
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 34 2e 31 38 35] [b'230625112224.185']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 6, Length Bytes=128/120
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 35 2e 33 39 35] [b'230625112225.395']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [35 35 30 39 31 30 30 30 38 32 32 34 30 34 30 30] [b'5509100082240400']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [35 35 30 39 31 30 30 30 38 32 32 34 30 34 30 30] [b'5509100082240400']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 39 2e 30 32 35] [b'230625112229.025']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 10, Length Bytes=208/200
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 39 2e 30 32 35] [b'230625112229.025']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 6, Length Bytes=128/120
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 39 2e 36 38 35] [b'230625112229.685']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 39 2e 36 38 35] [b'230625112229.685']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 4, Length Bytes=88/80
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 31 2e 36 36 35] [b'230625112231.665']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 33 2e 37 30 30] [b'230625112233.700']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 33 2e 37 30 30] [b'230625112233.700']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 6, Length Bytes=128/120
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 35 2e 36 38 30] [b'230625112235.680']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 36 2e 35 30 35] [b'230625112236.505']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 36 2e 35 30 35] [b'230625112236.505']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 6, Length Bytes=128/120
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 38 2e 37 30 35] [b'230625112238.705']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 38 2e 37 30 35] [b'230625112238.705']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 2, Length Bytes=48/40
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 34 30 2e 32 34 35] [b'230625112240.245']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 34 31 2e 35 36 35] [b'230625112241.565']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 2, Length Bytes=48/40
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 34 31 2e 35 36 35] [b'230625112241.565']
As you can see, the GPSU values are incorrect for a few samples:
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [35 35 30 39 31 30 30 30 38 32 32 34 30 34 30 30] [b'5509100082240400']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0
GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [35 35 30 39 31 30 30 30 38 32 32 34 30 34 30 30] [b'5509100082240400']
GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20
I don't have permission to share the file - is there any more information that would be useful for you? I'm guessing that this is probably happening quite a lot, as my software doesn't have that many users, and we have seen a couple of incidences now.
Thanks!
I hadn't previously noticed, but the timings of the samples are also very erratic, sometimes multiple packets at exact same time (hence 0 size, i guess)
Filtering is likely the only solution for these older camera sources. HERO11 deprecated GPS5 and GSPU, adding GSP9 as cleaner way to sync time and location.
I can filter the invalid GPSU, but incorrect values are correlated with loss of sync with GPS chip or something, as can be seen by the erratic readings.
I take it that no further firmware will be created for these devices, so it is what it is...
Correct, new firmware is not likely.