geospace-code/georinex

Unable to open file (ValueError: could not broadcast input array...)

padok opened this issue · 5 comments

padok commented

I am currently working with the Rinex files from the esa, but unfortunately I cannot load any of the Rinex version 3 files in the daily directory.
Thank you very much for your help!

File:
ftp://gssc.esa.int/gnss/data/daily/2019/brdc/BRDC00IGS_R_20191330000_01D_MN.rnx.gz

Click to view exception 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
 in 
     29 test = 'data/gnss/data/daily/2019/brdc/BRDC00IGS_R_20191330000_01D_MN.rnx.gz'
     30 
---> 31 obs = gr.load(test, fast=False)
     32 
     33 

~/.local/lib/python3.7/site-packages/georinex/base.py in load(rinexfn, out, use, tlim, useindicators, meas, verbose, fast, interval)
     57 
     58     if info['rinextype'] == 'nav':
---> 59         return rinexnav(rinexfn, outfn, use=use, tlim=tlim)
     60     elif info['rinextype'] == 'obs':
     61         return rinexobs(rinexfn, outfn, use=use, tlim=tlim,

~/.local/lib/python3.7/site-packages/georinex/base.py in rinexnav(fn, outfn, use, group, tlim)
    130         nav = rinexnav2(fn, tlim=tlim)
    131     elif int(info['version']) == 3:
--> 132         nav = rinexnav3(fn, use=use, tlim=tlim)
    133     else:
    134         raise LookupError(f'unknown RINEX  {info}  {fn}')

~/.local/lib/python3.7/site-packages/georinex/nav3.py in rinexnav3(fn, use, tlim)
    101         for j, i in enumerate(svi):
    102             darr[j, :] = np.genfromtxt(io.BytesIO(raws[i].encode('ascii')),
--> 103                                        delimiter=Lf)
    104 
    105 # %% discard duplicated times

ValueError: could not broadcast input array from shape (29) into shape (28)
padok commented

I tried to open every file in that folder and some of them work, but with data missing.

Click to view log 
WARNING:root:duplicate times detected on SV E02, using first of duplicate times
WARNING:root:duplicate times detected on SV E03, using first of duplicate times
WARNING:root:duplicate times detected on SV E05, using first of duplicate times
WARNING:root:duplicate times detected on SV E07, using first of duplicate times
WARNING:root:duplicate times detected on SV E08, using first of duplicate times
WARNING:root:duplicate times detected on SV E13, using first of duplicate times
WARNING:root:duplicate times detected on SV E15, using first of duplicate times
WARNING:root:duplicate times detected on SV E19, using first of duplicate times
WARNING:root:duplicate times detected on SV E21, using first of duplicate times
WARNING:root:duplicate times detected on SV E24, using first of duplicate times
WARNING:root:duplicate times detected on SV E25, using first of duplicate times
WARNING:root:duplicate times detected on SV E26, using first of duplicate times
WARNING:root:duplicate times detected on SV E27, using first of duplicate times
WARNING:root:duplicate times detected on SV E30, using first of duplicate times
WARNING:root:duplicate times detected on SV E33, using first of duplicate times
WARNING:root:duplicate times detected on SV E36, using first of duplicate times
O: BRDC00IGS_R_20191250000_01D_MN.rnx.gz
WARNING:root:duplicate times detected on SV E01, using first of duplicate times
WARNING:root:duplicate times detected on SV E02, using first of duplicate times
WARNING:root:duplicate times detected on SV E03, using first of duplicate times
WARNING:root:duplicate times detected on SV E04, using first of duplicate times
WARNING:root:duplicate times detected on SV E05, using first of duplicate times
WARNING:root:duplicate times detected on SV E08, using first of duplicate times
WARNING:root:duplicate times detected on SV E09, using first of duplicate times
WARNING:root:duplicate times detected on SV E11, using first of duplicate times
WARNING:root:duplicate times detected on SV E12, using first of duplicate times
WARNING:root:duplicate times detected on SV E19, using first of duplicate times
WARNING:root:duplicate times detected on SV E21, using first of duplicate times
WARNING:root:duplicate times detected on SV E24, using first of duplicate times
WARNING:root:duplicate times detected on SV E25, using first of duplicate times
WARNING:root:duplicate times detected on SV E26, using first of duplicate times
WARNING:root:duplicate times detected on SV E27, using first of duplicate times
WARNING:root:duplicate times detected on SV E31, using first of duplicate times
WARNING:root:duplicate times detected on SV E33, using first of duplicate times
WARNING:root:duplicate times detected on SV E36, using first of duplicate times
O: BRDC00IGS_R_20191200000_01D_MN.rnx.gz
WARNING:root:duplicate times detected on SV E01, using first of duplicate times
WARNING:root:duplicate times detected on SV E02, using first of duplicate times
WARNING:root:duplicate times detected on SV E03, using first of duplicate times
WARNING:root:duplicate times detected on SV E04, using first of duplicate times
WARNING:root:duplicate times detected on SV E09, using first of duplicate times
WARNING:root:duplicate times detected on SV E11, using first of duplicate times
WARNING:root:duplicate times detected on SV E12, using first of duplicate times
WARNING:root:duplicate times detected on SV E19, using first of duplicate times
WARNING:root:duplicate times detected on SV E24, using first of duplicate times
WARNING:root:duplicate times detected on SV E25, using first of duplicate times
WARNING:root:duplicate times detected on SV E30, using first of duplicate times
WARNING:root:duplicate times detected on SV E31, using first of duplicate times
WARNING:root:duplicate times detected on SV E36, using first of duplicate times
O: BRDC00IGS_R_20191300000_01D_MN.rnx.gz
O: BRDC00IGS_R_20190770000_01D_MN.rnx.gz
WARNING:root:duplicate times detected on SV E01, using first of duplicate times
WARNING:root:duplicate times detected on SV E03, using first of duplicate times
WARNING:root:duplicate times detected on SV E04, using first of duplicate times
WARNING:root:duplicate times detected on SV E07, using first of duplicate times
WARNING:root:duplicate times detected on SV E08, using first of duplicate times
WARNING:root:duplicate times detected on SV E12, using first of duplicate times
WARNING:root:duplicate times detected on SV E19, using first of duplicate times
WARNING:root:duplicate times detected on SV E21, using first of duplicate times
WARNING:root:duplicate times detected on SV E24, using first of duplicate times
WARNING:root:duplicate times detected on SV E26, using first of duplicate times
WARNING:root:duplicate times detected on SV E31, using first of duplicate times
O: BRDC00IGS_R_20190100000_01D_MN.rnx.gz
WARNING:root:duplicate times detected on SV E02, using first of duplicate times
WARNING:root:duplicate times detected on SV E03, using first of duplicate times
WARNING:root:duplicate times detected on SV E04, using first of duplicate times
WARNING:root:duplicate times detected on SV E05, using first of duplicate times
WARNING:root:duplicate times detected on SV E08, using first of duplicate times
WARNING:root:duplicate times detected on SV E09, using first of duplicate times
WARNING:root:duplicate times detected on SV E11, using first of duplicate times
WARNING:root:duplicate times detected on SV E12, using first of duplicate times
WARNING:root:duplicate times detected on SV E19, using first of duplicate times
WARNING:root:duplicate times detected on SV E24, using first of duplicate times
WARNING:root:duplicate times detected on SV E25, using first of duplicate times
WARNING:root:duplicate times detected on SV E31, using first of duplicate times
O: BRDC00IGS_R_20190290000_01D_MN.rnx.gz
scivision commented

@padok thanks for letting us know. We have created GeoRinex v.1.10.1 to fix this issue of malformed entries in NAV3 files. Thanks again!

  • 👍1
padok commented

Is the data really malformed? NASA and ESA are the largest providers of satellite data and I would like to read data from these files.
I can't do much with NaN replaced data.

Thanks in advanced!

VOMAY commented

Dear Padok,
I would like to add some clarification here. Because what you are saying about ESA is not entirely true.

GSSC is only an IGS data centre, just like CDDIS. Only parts of the data is directly provided by ESA and NASA. In fact, there are dozens of institutions around the world sharing GNSS data and products via GSSC and CDDIS. This is the whole idea of the IGS.

Now about your BRDC* files: As you may know, these files are combined files, based on navigations messages received by various receivers around the world. There are several in institutions generating these files. ESA is not one of them.

BRDC00IGS* is generated by TU Munich (almost certainly)
BRDC00IGN* is generated by IGN, Paris
BRDC00WRD* is generated by BKG, Frankfurt

It happens from time to time that a single file corrupt. In this case you could use on of the others.

You mentioned one specific file. Do you only have issues with this one day or also with others?

Best Regards

padok commented

Excuse me, I know the origin of the files, I had expressed it badly. I wanted to say that this is one of the most used sources and I have problems with many of these files.

I have to apologize, though.
My problems originated from the list of times when you selected a satellite. The list also contains times for which there is no data in the Rinex file and therefore all values are nan, hence my confusion.

For example:

rinexfile = 'data/gnss/data/daily/2019/brdc/BRDC00IGS_R_20191330000_01D_MN.rnx.gz'

obs = gr.load(rinexfile, fast=False)

for time in obs.sel(sv='G02').time:
    print(time.values)

outputs for example

[...]
2019-05-13T03:00:00.000000000
[...]

but G02 2019 05 13 03 00 00 is not part of the rinex file.

the nearest time in that file would be the line
G02 2019 05 13 04 00 00-2.118018455803E-04-9.208633855451E-12 0.000000000000E+00

I'm new to xarrays and I guess I just don't use it right.