Junk on adev plot
Closed this issue · 10 comments
I have a dataset. {lot it with Tom van Baak's adev5 and it looks clean. Plot it with allantools.oadev() and there is junk on the right edge of the data. I have seen this junk on other peoples adev plots, not sure what is happening.
The first one is using allantools.oadev(), and the second is adev5 + gnuplot.
My program is basically:
a = load_dataset(input_file)
a.compute("oadev")
plt.plot(a, errorbars=True, grid=True)
[repeated for all three data sets]
plt.show()
All data from a TAPR-TICC in TimeLab mode. Plot A is a plain GPS PPS, Plot B is a GPS OXCO, and plot C is B->A. B is hard to see as C pretty much overlays B.
Raw data on request.
How large is the largest tau-value you are calculating and plotting, compared to the total length of the time-series? A conservative rule might be tau_max = length/4. So if you have 1000 points collected at 1s interval the largest tau you should reasonable hope to extract an ADEV at is about tau=250s.
The overlapping allan-deviation is a bit better in this respect than the standard ADEV, and the TOTDEV in particular is designed to work around this limitation (it copies and reflects your original data to create a longer ('syntehtic') input dataset). That approach is valid if your clock really is a noise process and copying/reflecting the data produces more realistic data.
Error-bars are not working out-of-the-box correctly currently in allantools. There are examples on how to properly calculate them with first a power-law-noise estimator and then greenhall_edf(). If the errorbars work correctly you should get larger and larger errorbars at large tau - so the last points you see might not be anomalous they just have a very large confidence interval.
My data set is 15153 samples, at one second each. Good data, or bad data, tvb's adev5 (really oadev), and allantools oadev should be giving the same result, but they are not. Also, tvb took my data and plotted it with stable32. The stable32 result looked like the adev5 result. So allantools oadev is the odd man out.
Long time ago I learned that GPS signals don't look anything like standard noise. Many people try to model GPS errors as gaussian and the results are very misleading. The Skewness and Kurtosis are unique. So I'll just stick to the data I got.
With 15153, my tau_max would be 3788. It does look like all three methods agree to that tau. And all three have the same rightmost cutoff. So that still leaves the question of why they diverge at high tau. Especially since calc_adev_phase() specifically references tvb's adev_lib code and yet produces divergent results from it. First pass the adev code in both looks similar to me.
if you post the dataset somewhere, with a description of what it is, I can do a comparison.
if the codes output an "N" number, i.e. the number of frequency-pairs used for a particular tau then this is one more to check between the different codes.
From looking at the graphs, I disagree that adev5 and allantools share the same rightmost cutoff :
- adev5 plots end at tau =~ 5000 s
- allantools plots end at tau =~ 7500 s (i.e half the duration of the sample).
I suspect allantools and adev5 plots would be in very good agreement if allantools plots were truncated at 5000 s. I don’t know how adev5 what criterion adev5 uses to stop the curve. Of course the last points have a very small statistical significance : but I feel it should be left to the user to keep them or not.
Here is some additional discussion on this topic:
https://groups.google.com/d/msg/allantools/hL1JeBodaa8/atLSH94KAwAJ
I somehow forgot that the discussion on the mailing list was dealing with this, sorry…
Thanks for pointing it out !
closing this issue - no real bug just differences in how tau-values are selected or cut off based on the length of the time-series.