How to process nonuniform time series?
Opened this issue · 5 comments
Sometimes my data stream is corrupted and I have to throw out some samples. In this case, I have a time series with some variation in the sampling period. Is there a way to use this package to compute some sort of allan deviation in this case?
It seems that many other use cases involve variable sampling periods (either due to corrupted data or event-triggered sampling) and in these cases measuring time-stability can still be of value.
If this is not supported in this package, I would also appreciate some literature references for this kind of data analysis. Allan deviation is something I usually take for granted, so I'm not familiar with the theory.
Thanks!
Hi !
I am new to the field but I have spent the last month on such questions. I hope others correct me if I am wrong.
It does not seem that the package itself provides method to "fill in the gaps".
The standard program adopted by the scientific community is called "Stable32" and it can do that. You can find all the procedures an algorithms in "Handbook of Frequency Stability Analysis, William Riley, David A. Howe". Riley is the one who created Stable32 in first place.
I actually implemented the algorithm described in the handbook to fill the gaps (there are other important things such as removing outliers and removing drift).
I hope it helps.
take a look at gradev() - at least if your data-series is mostly uniform with a few gaps:
https://github.com/aewallin/allantools/blob/master/allantools/allantools.py#L1263
if you data is completely nonuniform then this may not help much.
The reference for gradev() should be added to the code, I think it is:
Ilaria Sesia and Patrizia Tavella, Estimating the Allan variance in the presence of long periods of missing data and outliers. 2008 Metrologia 45 S134 http://dx.doi.org/10.1088/0026-1394/45/6/S19
gap-filling algorithms based on published references would be a welcome addition to allantools. My impression is that they always make assumptions about the noise type of the valid data (for generating gap-filling synthetic data) - and therefore no optimal solution that works in all cases exists.
This is a good reference. I was not aware of it.
I had an algorithm that I started to write a year or so ago to do this, but I never finished it. It's on my rainy-day list of things to do for summer 2020. Are there other references for filling in frequency-data gaps?
The following reference explores how to handle different noises in frequency measurements, by correcting the definition of Allan Variance :
The Corrected Allan Variance: Stability Analysis of Frequency Measurements With Missing Data
Lorenzo Galleani ; Ilaria Sesia
IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control ( Volume: 66 , Issue: 10 , Oct. 2019 )
https://ieeexplore.ieee.org/document/8756280
or
https://doi.org/10.1109/TUFFC.2019.2927424
As Anders says, the problem is then to identify the type of noise, and this is the tricky part...
I also had in mind to implement this in AllanTools at some point, but please do not feel restrained from doing it yourself first ! ;-)
Probably won’t get to it until Summer if I do. Thanks for the reference.