spacetelescope/stdatamodels

Assigning `f8` 2d data to a table can lead to corrupt output

Opened this issue · 2 comments

Let's assume we have a schema for TableModel that contains

      one_column_table:
        fits_hdu: ONECOL
        datatype:
          - name: col
            datatype: float64

If we assign data to the model as follows, the produced file is corrupt:

fn = tmp_path / "test.fits"

write_model = TableModel()
arr = np.arange(16 * 32, dtype='f8').reshape((16, 32))
write_model.one_column_table = arr
np.testing.assert_equal(arr, write_model.one_column_table['col'])
write_model.save(fn)

with TableModel(fn) as read_model:
    np.testing.assert_equal(arr, read_model.one_column_table['col'])

The above code fails on TableModel(fn) with:

E           astropy.utils.exceptions.AstropyUserWarning: ERROR loading embedded ASDF: non-ASCII characters are present in the FITS file header and have been replaced by "?" characters

This can be further reduced to just astropy:

from astropy.io import fits
import numpy as np

hdulist = fits.HDUList()
hdulist.append(fits.PrimaryHDU())

arr = np.arange(32 * 16).reshape((32, 16)).astype(dtype=[('col', 'f8')])

hdulist.append(fits.BinTableHDU())
hdulist[-1].data = arr

hdulist.writeto('foo.fits', overwrite=True)

with fits.open('foo.fits') as ff:
    ff.info()

Running the above code results in:

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 2: ordinal not in range(128)
and
OSError: Header missing END card.

Note that in all cases the error doesn't occur until the data is read. No errors are issued on write.

Using f4 does not produce the same error but does produce a file with an incorrectly shaped array ((32) instead of (32, 16)).

This may be related to #215