Fixing AstropyDeprecationWarning by using asdf-astropy causes real failure
Closed this issue · 1 comments
jdavies-st commented
When running the stdatamodels unit tests (from master), one gets the following deprecation warning:
$ pytest
========================================= test session starts ==========================================
platform darwin -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/jdavies/dev/stdatamodels, configfile: setup.cfg, testpaths: tests, src
plugins: asdf-2.14.2, doctestplus-0.12.1, openfiles-0.5.0
collected 184 items
tests/test_filetype.py ............ [ 6%]
tests/test_fits.py ....................................... [ 27%]
tests/test_history.py ... [ 29%]
tests/test_models.py ................................. [ 47%]
tests/test_resources.py ........ [ 51%]
tests/test_s3_utils.py ..... [ 54%]
tests/test_schema.py ............... [ 62%]
tests/test_storage.py . [ 63%]
tests/test_util.py ........................... [ 77%]
tests/test_validation.py ....xx......x......x.x.................. [ 99%]
src/stdatamodels/util.py . [100%]
=========================================== warnings summary ===========================================
tests/test_history.py::test_history_from_model_to_fits
tests/test_history.py::test_history_from_model_to_fits
/Users/jdavies/miniconda3/envs/stdatamodels/lib/python3.10/site-packages/astropy/io/misc/asdf/types.py:72: AstropyDeprecationWarning: ASDF functionality for astropy is being moved out of the astropy package to the new asdf-astropy package. Please use this package instead. astropy.io.misc.asdf is deprecated since astropy 5.1 and will be removed in a future release.
warnings.warn(create_asdf_deprecation_warning())
tests/test_history.py::test_history_from_model_to_fits
/Users/jdavies/miniconda3/envs/stdatamodels/lib/python3.10/site-packages/astropy/io/misc/asdf/types.py:77: AstropyDeprecationWarning: ASDF functionality for astropy is being moved out of the astropy package to the new asdf-astropy package. Please use this package instead. astropy.io.misc.asdf is deprecated since astropy 5.1 and will be removed in a future release.
warnings.warn(create_asdf_deprecation_warning())
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================= 179 passed, 5 xfailed, 3 warnings in 11.20s ==============================
These deprecations come from astropy itself:
$ pytest tests/test_history.py -W error::astropy.utils.exceptions.AstropyDeprecationWarning
========================================= test session starts ==========================================
platform darwin -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/jdavies/dev/stdatamodels, configfile: setup.cfg
plugins: asdf-2.14.2, doctestplus-0.12.1, openfiles-0.5.0
collected 3 items
tests/test_history.py .F. [100%]
=============================================== FAILURES ===============================================
___________________________________ test_history_from_model_to_fits ____________________________________
tmpdir = local('/private/var/folders/t1/md4315cx5kl3zjv8fkgmtvl586jmhc/T/pytest-of-jdavies/pytest-17/test_history_from_model_to_fit0')
def test_history_from_model_to_fits(tmpdir):
tmpfits = str(tmpdir.join('tmp.fits'))
m = DataModel()
m.history = [HistoryEntry({
'description': 'First entry',
'time': Time(datetime.datetime.now())})]
m.history.append(HistoryEntry({
'description': 'Second entry',
'time': Time(datetime.datetime.now())
}))
> m.save(tmpfits)
tests/test_history.py:57:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/stdatamodels/model_base.py:523: in save
self.to_fits(output_path, *args, **kwargs)
src/stdatamodels/model_base.py:648: in to_fits
ff = AsdfInFits(hdulist, tree)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/fits_embed.py:150: in __init__
super().__init__(tree=tree, **kwargs)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:177: in __init__
self.tree = tree
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:589: in tree
self._validate(asdf_object, custom=bool(tree))
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:617: in _validate
tagged_tree = yamlutil.custom_tree_to_tagged_tree(tree, self)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/yamlutil.py:261: in custom_tree_to_tagged_tree
return treeutil.walk_and_modify(
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:420: in walk_and_modify
return _recurse(top)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:407: in _recurse
result = _handle_children(result, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:377: in _handle_children
return _handle_generator(result)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:276: in _handle_generator
result = next(generator)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:304: in _handle_mapping
value = _recurse(value, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:407: in _recurse
result = _handle_children(result, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:377: in _handle_children
return _handle_generator(result)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:276: in _handle_generator
result = next(generator)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:304: in _handle_mapping
value = _recurse(value, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:407: in _recurse
result = _handle_children(result, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:377: in _handle_children
return _handle_generator(result)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:276: in _handle_generator
result = next(generator)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:337: in _handle_mutable_sequence
result.append(_recurse(value, json_id))
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:407: in _recurse
result = _handle_children(result, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:377: in _handle_children
return _handle_generator(result)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:276: in _handle_generator
result = next(generator)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:304: in _handle_mapping
value = _recurse(value, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:406: in _recurse
result = _handle_callback(node, json_id)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/treeutil.py:283: in _handle_callback
result = callback(node)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/yamlutil.py:258: in _walker
return tag.to_tree_tagged(obj, ctx)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'astropy.io.misc.asdf.tags.time.time.TimeType'>
node = <Time object: scale='utc' format='datetime' value=2022-12-12 12:17:45.455596>
ctx = <asdf.fits_embed.AsdfInFits object at 0x7fdff4d03b80>
@classmethod
def to_tree_tagged(cls, node, ctx):
> warnings.warn(create_asdf_deprecation_warning())
E astropy.utils.exceptions.AstropyDeprecationWarning: ASDF functionality for astropy is being moved out of the astropy package to the new asdf-astropy package. Please use this package instead. astropy.io.misc.asdf is deprecated since astropy 5.1 and will be removed in a future release.
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/astropy/io/misc/asdf/types.py:72: AstropyDeprecationWarning
======================================= short test summary info ========================================
FAILED tests/test_history.py::test_history_from_model_to_fits - astropy.utils.exceptions.AstropyDeprecationWarning: ASDF functionality for astropy is being moved o...
===================================== 1 failed, 2 passed in 0.45s ======================================
But if I pip install asdf-astropy and rerun the test, I get an actual validation error:
$ pytest tests/test_history.py
========================================= test session starts ==========================================
platform darwin -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/jdavies/dev/stdatamodels, configfile: setup.cfg
plugins: asdf-2.14.2, doctestplus-0.12.1, openfiles-0.5.0
collected 3 items
tests/test_history.py .F. [100%]
=============================================== FAILURES ===============================================
___________________________________ test_history_from_model_to_fits ____________________________________
tmpdir = local('/private/var/folders/t1/md4315cx5kl3zjv8fkgmtvl586jmhc/T/pytest-of-jdavies/pytest-18/test_history_from_model_to_fit0')
def test_history_from_model_to_fits(tmpdir):
tmpfits = str(tmpdir.join('tmp.fits'))
m = DataModel()
m.history = [HistoryEntry({
'description': 'First entry',
'time': Time(datetime.datetime.now())})]
m.history.append(HistoryEntry({
'description': 'Second entry',
'time': Time(datetime.datetime.now())
}))
> m.save(tmpfits)
tests/test_history.py:57:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/stdatamodels/model_base.py:523: in save
self.to_fits(output_path, *args, **kwargs)
src/stdatamodels/model_base.py:648: in to_fits
ff = AsdfInFits(hdulist, tree)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/fits_embed.py:150: in __init__
super().__init__(tree=tree, **kwargs)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:177: in __init__
self.tree = tree
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:589: in tree
self._validate(asdf_object, custom=bool(tree))
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/asdf.py:618: in _validate
schema.validate(tagged_tree, self, reading=reading)
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/asdf/schema.py:682: in validate
validator.validate(instance)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Validator(schema={}, format_checker=None)
args = ({'history': {'entries': [{'description': 'First entry', 'time': {'value': '2022-12-12T12:20:17.336', 'base_format': '....fits', 'model_type': 'DataModel'}, '_fits_hash': 'b8bf102cca29ade468c8db8c7dab0c0a8f7e9110d5b398ba436b361789642b44'},)
kwargs = {}
error = <ValidationError: "{'entries': [{'description': 'First entry', 'time': {'value': '2022-12-12T12:20:17.336', 'base_form...time': {'value': '2022-12-12T12:20:17.339', 'base_format': 'datetime'}}]} is not valid under any of the given schemas">
def validate(self, *args, **kwargs):
for error in self.iter_errors(*args, **kwargs):
> raise error
E jsonschema.exceptions.ValidationError: {'entries': [{'description': 'First entry', 'time': {'value': '2022-12-12T12:20:17.336', 'base_format': 'datetime'}}, {'description': 'Second entry', 'time': {'value': '2022-12-12T12:20:17.339', 'base_format': 'datetime'}}]} is not valid under any of the given schemas
E
E Failed validating 'anyOf' in schema['properties']['history']:
E {'anyOf': [{'items': [{'$ref': 'history_entry-1.0.0'}],
E 'type': 'array'},
E {'$ref': '#/definitions/history-1.1.0'}],
E 'description': 'A log of transformations that have happened to the '
E 'file. May\n'
E 'include such things as data collection, data '
E 'calibration\n'
E 'pipelines, data analysis etc.\n'}
E
E On instance['history']:
E {'entries': [{'description': 'First entry',
E 'time': {'base_format': 'datetime',
E 'value': '2022-12-12T12:20:17.336'}},
E {'description': 'Second entry',
E 'time': {'base_format': 'datetime',
E 'value': '2022-12-12T12:20:17.339'}}]}
../../miniconda3/envs/stdatamodels/lib/python3.10/site-packages/jsonschema/validators.py:314: ValidationError
======================================= short test summary info ========================================
FAILED tests/test_history.py::test_history_from_model_to_fits - jsonschema.exceptions.ValidationError: {'entries': [{'description': 'First entry', 'time': {'value'...
===================================== 1 failed, 2 passed in 0.81s ======================================
Not sure if this is a bug here, or in stdatamodels.
WilliamJamieson commented
This should be fixed by spacetelescope/stdatamodels#104.
The issue is that the way ASDF was handling time objects (and still is) preventing round-tripping of many types of time objects. The fix for this means that stdatmodels's history entry needs to be more careful in how it constructs a time object.