implement ODIM BUFR readers
Opened this issue · 21 comments
At the last OPERA ET meeting it emerged that there is a real interest for xradar to have and ODIM BUFR reader, in particular to read the French radar data.
I am fully aware that that is not an easy task but considering that there are plans to open the OPERA data archive and that, particularly at the early stages of the OPERA program, many files are in BUFR format this would be very welcomed by the European radar community.
@jfigui Thanks for bringing this to attention.
wradlib used to read BUFR back in the old days, but dropped support because of all kind of issues. The bufr related python ecosystem evolved much since then, so this might not be an issue any more.
If you can point to some resources on how to read french bufr data that would be great. Also docs on how to handle tables etc. would be nice to have.
you probably know that but there are resources linked to OPERA Bufr here:
https://www.eumetnet.eu/activities/observations-programme/current-activities/opera/
In particular:
https://www.eumetnet.eu/wp-content/uploads/2017/01/OPERA_bufr_description_2012.pdf
https://www.eumetnet.eu/wp-content/uploads/2017/03/bufr_3.2.zip
I can probably send you some examples of polar volume bufr files as well if needed.
Yeah, but good it's now linked here. I've thought there would be some low hanging fruit with new packages like pdbufr or even directly with eccodes. The code on that eumetnet-page is rather old, unfortunately.
It would be great if you could provide some bufr polar volumes. Do they need special tables or can they be read with the standard tables?
I think French data need special tables. What I positively know is that eccodes cannot open the polar volume bufr data directly (I tried). I have always tried to stay away from BUFR files so I am definitely not an expert but I will ask for support at MF.
I attach a copy of the tables that are used at Météo-France
tables_bufr_opera.zip
and here there is an example of so called PAM file:
- a file containing a single sweep (i.e. one elevation) of the fields uncorrected reflectivity, co-polar correlation coefficient, non-corrected differential phase and differential reflectivity in polar coordinates (240 m x 0.5°) plus the standard deviation of the reflectivity in Cartesian coordinates
PAMA59LFPW180755.gz
here there is an example of so called PAG file, these files are sent to the OPERA data center:
- a file containing a single sweep (i.e. one elevation) of the fields uncorrected reflectivity, de-aliased Doppler velocity and standard deviation of the reflectivity.
PAGA59EODC180755.gz
One thing to keep in mind is that as far as I understood those files are not a single bufr file with multiple fields but a concatenation of bufr files. To be able to decode them properly you have to first cut the file and then read the bufr.
Another problematic issue is that the field data is compressed using z lib so even when you manage to decode the bufr you still have to decompress the data.
I feel like I'm remembering now, why we've dropped bufr in wradlib 😬
I can fully understand your decision!
But if we could give it another try ... :)
If at first you don't succeed...
I am wondering if this might be of use here
https://github.com/pytroll/trollbufr
Last time I checked trollbufr was not able to decode the Météo France BUFR. Among other things it was not prepared to read BUFR version 2 or 3 but it can be starting point.
There's also pybufrkit, which might be worth a test.
Hey everyone, have we had any success in this field ? I recently received a load of bufr radar files from météofrance, and I'm having little to no success in reading them with pybufrkit or even the opera bufr linux software listed in one of the comments up there. For me it is quite important and necessary to read these files and I use a python environment, if someone could give me any clues of how to move forward, I'll gladly be implementing a reader to use in an xarray environment. If someone advanced on this we could work together. I've attached a data sample and the tables they sent me. Thanks in advance!
No, there haven't been any advances here. The major pain point is, and this is a show stopper, that the provided tables are in some ".csv" format. eccodes does not read these tables per default and they will have to be converted for that purpose to the eccodes table format.
Please ask your table provider to provide the tables in ECMWF eccodes format.
I am having a similar problem with some sample BUFR files I had from Météo-France.
See here
@Francesco-Uboldi Are your MF tables the same as linked above by @jfigui and @rzambranap?
And would you join a team to implement an eccodes based BUFR reader for xradar?
I have version 279 instead of 281... I will try the 281 from @rzambranap .
I can try and help, but I am quite new to both radar data in general and BUFR format. In particular, I never used eccodes: I met this software recently, of course... but the table problem stopped me, and I went towards OPERA instead.
And I have limited knowledge of python! And of C. I come from Fortran and R...
By the way... wouldn't it be nice if some BUFR user from Météo-France agreed to be involved?!? Anyone out there?
@Francesco-Uboldi As I see the situation it all depends on the tables. We would need those tables in the format accepted by eccodes. Or there would need to be some transformation tool to convert the tables. It's a pity that BUFR data is so unaccessible.