Issue reading map file: Mismatched byte error
Closed this issue · 3 comments
BUG
From pyDARN repo
author: @Yas979
Please can I check if anyone can help with the following error message? I am attempting to create map plots of the convection electric field, using pydarn.SuperDARNRead to extract the map data from the .map file. This has worked in earlier versions of pydarn. However, the latest version of PyDarn results in the following error:
mapfile='/data/sol-ionosphere/sd_map_potential/fitacf3_allrad/north/map/2011/20110101.fit.map'
SDarn_read = pydarn.SuperDARNRead(mapfile)
map_data = SDarn_read.read_map()
Error: /data/sol-ionosphere/sd_map_potential/fitacf3_allrad/north/map/2011/20110101.fit.map contains an block size 540160288 > remaining bytes 69250442 at record = 0.
Traceback (most recent call last):
File "", line 1, in
File "/alice-home/1/y/ys378/alice3_python_env/env/lib64/python3.9/site-packages/pydarnio/dmap/superdarn.py", line 517, in read_map
self._read_darn_records(file_struct_list)
File "/alice-home/1/y/ys378/alice3_python_env/env/lib64/python3.9/site-packages/pydarnio/dmap/superdarn.py", line 393, in _read_darn_records
self._read_darn_record(format_fields, optional_list)
File "/alice-home/1/y/ys378/alice3_python_env/env/lib64/python3.9/site-packages/pydarnio/dmap/superdarn.py", line 359, in _read_darn_record
record = self.read_record()
File "/alice-home/1/y/ys378/alice3_python_env/env/lib64/python3.9/site-packages/pydarnio/dmap/dmap.py", line 449, in read_record
self.bytes_check(block_size, "block size",
File "/alice-home/1/y/ys378/alice3_python_env/env/lib64/python3.9/site-packages/pydarnio/dmap/dmap.py", line 286, in bytes_check
raise dmap_exceptions.MismatchByteError(self.dmap_file,
pydarnio.exceptions.dmap_exceptions.MismatchByteError: Error: /data/sol-ionosphere/sd_map_potential/fitacf3_allrad/north/map/2011/20110101.fit.map contains an block size 540160288 > remaining bytes 69250442 at record = 0.
Notes
Follow up:
From @carleyjmartin
If you could email/transfer me the file you are trying to open at carley.martin@usask.ca? It looks like the file itself doesn't fit the format of what pydarnio is expecting in read_map for a map file, have you tried to dmapdump using RST to check the contents of the file (don't worry if not it'll just be the first thing I do when I have the file)?
You could also try read_dmap() or read_records() instead of read_map() but I suspect you might still get the same error - let me know if that is the case if you give that a go!
Also could you let me know what version of pydarn and pydarnio you have installed, you can find it using pip3 list. Updating pydarn does not always update the pydarnio version, but it should be >1.1.0, we haven't really updated the lines causing the error for quite a few years, so if you can read this file using an older version of either pydarn/pydarnio, is it possible to let me know which version so we can track back and find out?
Thanks, Carley
Hi @carleyjmartin,
I have emailed file over a file drop platform. Please let me know if you have not received details on how to download the file.
I haven't used dmapdump on the file and will have a look at doing this, as well as the other tools to read the map file.
I am currently using pydarn v3.1.1, pydarnio v1.2.1.
Thanks,
Yasir
Thanks for sending the file @Yas979, I've had a look and for now I would recommend remaking the map file for that day, dmapdump is unable to parse the contents and when converting to ascii to try to view the contents with cnvmaptomap the resulting ascii file is empty. I also cannot plot using RST either, giving an error opening the file. This along with the MismatchByteError from pydarnio leads me to believe that the file is possibly corrupted, or does not match the format dictated for map files and pydarnio is returning the correct error for that situation.
Sorry I can't help more as I can't physically get into the file to see what it look like, if you find more files that are having this issue let me know, but for now I think it might just be the file having an error rather than pydarnio.
I've attached below a zip file with a version of the map file for that date (north, all radars, fitacf3) that I produced for my own map file repo, it appears to be a lot smaller is size than the file sent (~27MB unzipped), feel free to use this file to make some convection maps if you would like.
20110101.n.map.zip
Hi @carleyjmartin, Thanks for looking into this further and sending another map file.
I realised the map file I had been attempting to view was not in the correct cnv format and have since been able to view the cnv map files in PyDarn.
Thanks again