Issue with reading data dated to 31 Mar from IAGA-2002 files (Mac OS)
teraita opened this issue · 5 comments
teraita commented
When reading the data to the 1min values there is issue with handling of the data of 31 March. The file is not read correctly between 03-05UT. When manipulating the date information in filename, and data row (detestr and doy) in IAGA-2002 format to respond 31 Jan the file works. Might be some issue with epoch/ stream lengths?
Problem sees to hit only to period 03-05UT:
Same data file, date information manipulated:
When merging XYZ and F of the date and calculating deltaF the values are corrupted, so it this not only visualisation problem in MagPy.
Is this only Mac related issue?
Please, request data samples, if needed.
leonro commented
Hi Tero, can you please send me the data files? I will check what going on.
teraita commented
Hi Roman,
Here is the 1min SOD file with XYZ.
Cheers,
Tero
Ps. We still have “chance” to get trough the group stage by stealing point from Belgium. At least about 3 hours, when Lukaku strikes… ;-)
… On 21. Jun 2021, at 19.26, Roman Leonhardt ***@***.***> wrote:
Hi Tero,
can you please send me the data files? I will check what going on. Sorry for not replying earlier, but my notebook broke down and I needed some time to set up everything again...
Best wishes,
Roman
Dr. Roman Leonhardt
Conrad Observatorium
ZAMG - Zentralanstalt für Meteorologie und Geodynamik
1190 Wien, Hohe Warte 38
Tel.: +43 1 36026 2507
Fax: +43 1 36866 21
E-Mail: ***@***.***<https://webmail.zamg.ac.at/owa/redir.aspx?SURL=t9AS8jCcPggsx8oPf2D18nQbebRT-hDvZYGVU9uNLKVg80_NrjnSCG0AYQBpAGwAdABvADoAcgBvAG0AYQBuAC4AbABlAG8AbgBoAGEAcgBkAHQAQAB6AGEAbQBnAC4AYQBjAC4AYQB0AA..&URL=mailto%3aroman.leonhardt%40zamg.ac.at>
www.zamg.ac.at<https://webmail.zamg.ac.at/owa/redir.aspx?SURL=DgfuyOdHyWS5QXRRIyzOHmne5_jxCaCmQXd1H4B7AnJg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgB6AGEAbQBnAC4AYQBjAC4AYQB0AC8A&URL=http%3a%2f%2fwww.zamg.ac.at%2f>
Conrad Observatorium
2763 Muggendorf , Trafelberg 1-2
Tel.: +43 2632 73276 oder 7367221
www.conrad-observatory.at<https://webmail.zamg.ac.at/owa/redir.aspx?SURL=3GEn2GkM89BDePjqOqAYT0d3pi9ZpUlcz8Zli53n9uNg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgBjAG8AbgByAGEAZAAtAG8AYgBzAGUAcgB2AGEAdABvAHIAeQAuAGEAdAAvAA..&URL=http%3a%2f%2fwww.conrad-observatory.at%2f>
________________________________
Von: Tero Raita ***@***.***>
Gesendet: Donnerstag, 20. Mai 2021 14:15:28
An: geomagpy/magpy
Cc: Subscribed
Betreff: [geomagpy/magpy] Issue with reading data dated to 31 Mar from IAGA-2002 files (Mac OS) (#103)
When reading the data to the 1min values there is issue with handling of the data of 31 March. The file is not read correctly between 03-05UT. When manipulating the date information in filename, and data row (detestr and doy) in IAGA-2002 format to respond 31 Jan the file works. Might be some issue with epoch/ stream lengths?
Problem sees to hit only to period 03-05UT:
[SOD20190331_03-05UT]<https://user-images.githubusercontent.com/22149196/118975443-b0e7da00-b97c-11eb-8cda-9aa3c9af40f9.png>
[SOD20190331]<https://user-images.githubusercontent.com/22149196/118975410-a9283580-b97c-11eb-8229-ddf25757a31c.png>
Same data file, date information manipulated:
[SOD20190131]<https://user-images.githubusercontent.com/22149196/118975462-b6452480-b97c-11eb-883c-dcaa11e17398.png>
When merging XYZ and F of the date and calculating deltaF the values are corrupted, so it this not only visualisation problem in MagPy.
Is this only Mac related issue?
Please, request data samples, if needed.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#103>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AD7YDSL7DIYWKM7LPESA643TOT4OBANCNFSM45G35ZEA>.
EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#103 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFI7QTDMLP2Y2WDTNMHSTNTTT5R3BANCNFSM45G35ZEA>.
teraita commented
Hi,
2Hz IAGA-2002 files of 30-31Mar 2019.
Cheers,
Tero
… On 21. Jun 2021, at 19.32, Tero Raita ***@***.***> wrote:
Hi Roman,
Here is the 1min SOD file with XYZ.
Cheers,
Tero
Ps. We still have “chance” to get trough the group stage by stealing point from Belgium. At least about 3 hours, when Lukaku strikes… ;-)
<sod20190331vmin.min>
> On 21. Jun 2021, at 19.26, Roman Leonhardt ***@***.*** ***@***.***>> wrote:
>
>
> Hi Tero,
>
>
> can you please send me the data files? I will check what going on. Sorry for not replying earlier, but my notebook broke down and I needed some time to set up everything again...
>
>
> Best wishes,
> Roman
>
>
> Dr. Roman Leonhardt
> Conrad Observatorium
> ZAMG - Zentralanstalt für Meteorologie und Geodynamik
> 1190 Wien, Hohe Warte 38
> Tel.: +43 1 36026 2507
> Fax: +43 1 36866 21
> E-Mail: ***@***.***<https://webmail.zamg.ac.at/owa/redir.aspx?SURL=t9AS8jCcPggsx8oPf2D18nQbebRT-hDvZYGVU9uNLKVg80_NrjnSCG0AYQBpAGwAdABvADoAcgBvAG0AYQBuAC4AbABlAG8AbgBoAGEAcgBkAHQAQAB6AGEAbQBnAC4AYQBjAC4AYQB0AA..&URL=mailto%3aroman.leonhardt%40zamg.ac.at <https://webmail.zamg.ac.at/owa/redir.aspx?SURL=t9AS8jCcPggsx8oPf2D18nQbebRT-hDvZYGVU9uNLKVg80_NrjnSCG0AYQBpAGwAdABvADoAcgBvAG0AYQBuAC4AbABlAG8AbgBoAGEAcgBkAHQAQAB6AGEAbQBnAC4AYQBjAC4AYQB0AA..&URL=mailto%3aroman.leonhardt%40zamg.ac.at>>
> www.zamg.ac.at <http://www.zamg.ac.at/><https://webmail.zamg.ac.at/owa/redir.aspx?SURL=DgfuyOdHyWS5QXRRIyzOHmne5_jxCaCmQXd1H4B7AnJg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgB6AGEAbQBnAC4AYQBjAC4AYQB0AC8A&URL=http%3a%2f%2fwww.zamg.ac.at%2f <https://webmail.zamg.ac.at/owa/redir.aspx?SURL=DgfuyOdHyWS5QXRRIyzOHmne5_jxCaCmQXd1H4B7AnJg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgB6AGEAbQBnAC4AYQBjAC4AYQB0AC8A&URL=http%3a%2f%2fwww.zamg.ac.at%2f>>
>
>
> Conrad Observatorium
> 2763 Muggendorf , Trafelberg 1-2
> Tel.: +43 2632 73276 oder 7367221
> www.conrad-observatory.at <http://www.conrad-observatory.at/><https://webmail.zamg.ac.at/owa/redir.aspx?SURL=3GEn2GkM89BDePjqOqAYT0d3pi9ZpUlcz8Zli53n9uNg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgBjAG8AbgByAGEAZAAtAG8AYgBzAGUAcgB2AGEAdABvAHIAeQAuAGEAdAAvAA..&URL=http%3a%2f%2fwww.conrad-observatory.at%2f <https://webmail.zamg.ac.at/owa/redir.aspx?SURL=3GEn2GkM89BDePjqOqAYT0d3pi9ZpUlcz8Zli53n9uNg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgBjAG8AbgByAGEAZAAtAG8AYgBzAGUAcgB2AGEAdABvAHIAeQAuAGEAdAAvAA..&URL=http%3a%2f%2fwww.conrad-observatory.at%2f>>
> ________________________________
> Von: Tero Raita ***@***.***>
> Gesendet: Donnerstag, 20. Mai 2021 14:15:28
> An: geomagpy/magpy
> Cc: Subscribed
> Betreff: [geomagpy/magpy] Issue with reading data dated to 31 Mar from IAGA-2002 files (Mac OS) (#103)
>
>
> When reading the data to the 1min values there is issue with handling of the data of 31 March. The file is not read correctly between 03-05UT. When manipulating the date information in filename, and data row (detestr and doy) in IAGA-2002 format to respond 31 Jan the file works. Might be some issue with epoch/ stream lengths?
>
> Problem sees to hit only to period 03-05UT:
> [SOD20190331_03-05UT]<https://user-images.githubusercontent.com/22149196/118975443-b0e7da00-b97c-11eb-8cda-9aa3c9af40f9.png <https://user-images.githubusercontent.com/22149196/118975443-b0e7da00-b97c-11eb-8cda-9aa3c9af40f9.png>>
>
> [SOD20190331]<https://user-images.githubusercontent.com/22149196/118975410-a9283580-b97c-11eb-8229-ddf25757a31c.png <https://user-images.githubusercontent.com/22149196/118975410-a9283580-b97c-11eb-8229-ddf25757a31c.png>>
>
> Same data file, date information manipulated:
> [SOD20190131]<https://user-images.githubusercontent.com/22149196/118975462-b6452480-b97c-11eb-883c-dcaa11e17398.png <https://user-images.githubusercontent.com/22149196/118975462-b6452480-b97c-11eb-883c-dcaa11e17398.png>>
>
> When merging XYZ and F of the date and calculating deltaF the values are corrupted, so it this not only visualisation problem in MagPy.
>
> Is this only Mac related issue?
>
> Please, request data samples, if needed.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub<#103 <#103>>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AD7YDSL7DIYWKM7LPESA643TOT4OBANCNFSM45G35ZEA <https://github.com/notifications/unsubscribe-auth/AD7YDSL7DIYWKM7LPESA643TOT4OBANCNFSM45G35ZEA>>.
>
> EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub <#103 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFI7QTDMLP2Y2WDTNMHSTNTTT5R3BANCNFSM45G35ZEA>.
>
leonro commented
Well.... something I dont really understand: I am testing for leap seconds within the method LeapSeconds of stream.py. I am using the following conversion which actually should work fine in order to get a datetime compatible output:
t = "2019-03-31T02:00:00"
ti = time.strptime(t, "%Y-%m-%dT%H:%M:%S")
tim = datetime.fromtimestamp(time.mktime(ti))
tim, however, then results in 2019-03-31T03:00:00 which is obviously wrong. Wrong results are apparently only obtained on March 31, 2019 between 2:00 and 3:00.
Workaround:
tim = datetime(*ti[:5]+(min(ti[5], 59),))