ASDF swallows warnings raised by converters

[WilliamJamieson]

If I raise a warning inside the code of an ASDF converter, then that warning never makes it to the user.

I would like to issue DepreciationWarning from an ASDF converter, so that I can handle "deprecating" tags themselves. Roman is changing the schema organization structure, this is causing tag URIs to shift and I have a map from the old tag URIs to the new ones. Thus I would like to issue a DepreciationWarning during file reads informing the user that the old tag is being interpreted as the new tag and the complications that might cause. Unfortunately, my warnings never make it out to the user opening files.

I suggest that ASDF adds an AsdfConverterDepreciationWarning which can be issued inside a converter so that ASDF can avoid swallowing those warnings.

[braingram]

Thanks for opening the issue. Do you have an example of a warning that is swallowed? Testing with a simple converter I'm failing to reproduce this issue:

import asdf
import warnings


class Foo:
    pass

tag_uri = "asdf://example.com/tags/foo-1.0.0"


class FooConverter:
    types = [Foo]
    tags = [tag_uri]

    def to_yaml_tree(self, obj, tag, ctx):
        warnings.warn("to_yaml_tree", DeprecationWarning)
        return 'foo'

    def from_yaml_tree(self, obj, tag, ctx):
        warnings.warn("from_yaml_tree", DeprecationWarning)
        return Foo()


class FooExtension:
    converters=[FooConverter()]
    tags=[tag_uri]
    extension_uri = "asdf://example.com/extensions/foo-1.0.0"


with asdf.config_context() as cfg:
    cfg.add_extension(FooExtension())

    af = asdf.AsdfFile({"foo": Foo()})
    assert type(af["foo"]) == Foo
    af.write_to("test.asdf")

    with asdf.open("test.asdf") as af:
        assert type(af["foo"]) == Foo

I see 2 deprecation warnings:

DeprecationWarning: to_yaml_tree
  warnings.warn("to_yaml_tree", DeprecationWarning)
DeprecationWarning: from_yaml_tree
  warnings.warn("from_yaml_tree", DeprecationWarning)

[WilliamJamieson]

Its a slightly complicated setup, but I don't see the warning I am issuing in my code.

To setup, write a ReadnoiseRefModel to an asdf file using the main branch of roman_datamodels: https://github.com/spacetelescope/roman_datamodels.

import roman_datamodels.datamodels as rdm
from roman_datamodels.maker_utils import mk_datamodel

model = mk_datamodel(rdm.ReadnoiseRefModel)
model.to_asdf("readnoise.asdf")

Now install my dev branch of roman_datamodels: https://github.com/WilliamJamieson/roman_datamodels/tree/feature/autogen. Then try to read the file:

import asdf
from roman_datamodels.datamodels import _generated
with asdf.open("readnoise.asdf") as af:
    model = af.tree["roman"]
    assert isinstance(model, _generated.ReadnoiseRefModel)

I only get the warning:

/Users/wjamieson/.pyenv/versions/roman_datamodels/lib/python3.11/site-packages/asdf/asdf.py:326: AsdfWarning: File 'file:///Users/wjamieson/workspaces/roman_datamodels/ReadnoiseRef.asdf' was created with extension URI 'asdf://stsci.edu/datamodels/roman/extensions/datamodels-1.0' (from package roman-datamodels==0.18.1.dev8+g70f05e8), which is not currently installed
  warnings.warn(msg, AsdfWarning)

Which is due to the slightly different versions of RAD and RDM used.

I should also get an additional warning:

DeprecationWarning: The tag: asdf://stsci.edu/datamodels/roman/tags/origin-1.0.0 has been removed as a stand alone tag, if this data is written the tag will not be included

[WilliamJamieson]

That warning should be issued by: https://github.com/WilliamJamieson/roman_datamodels/blob/6643f5052f1994ac00a356563b8aae74e283ddf7/src/roman_datamodels/pydantic/converter.py#L90-L92.

I know this warning is run because if I turn it into an error then I get an exception during reading the file.

[WilliamJamieson]

Could the warning be swallowed by ASDF issuing a warning due to the library version mismatches?

[braingram]

Thanks for sharing the details.

I was able to get the autogen branch installed but had to make a few edits to get it to run (more on that below). I am able to see the warning but as expected (since this is a DeprecationWarning) it only appears if run with python development mode (or with appropriately set warnings filters). Would you try running your test again with python -X dev <your script>?

To get the autogen branch to run I first ran into a ClassVar error:

TypeError: Plain typing.ClassVar is not valid as type argument

Removing the ClassVar annotations from the jinja templates (like this one) silenced that error.

The second error was TypeError: unhashable type: 'ModelPrivateAttr'. I modified the failing line: https://github.com/WilliamJamieson/roman_datamodels/blob/feature/autogen/src/roman_datamodels/pydantic/extension.py#L19
to instead use _tag_uri.default.

I doubt that either of the fixes impacted the warning filters and instead might point to version/dependency issues with the branch.

[WilliamJamieson]

Those "fixes" unfortunately break other code. The issue you experienced was an oversight on my part. Python < 3.11 requires ClassVar to have a type argument. I was not using one for some of the autogenerated code. I have corrected this with ClassVar[str].

Note that the ClassVar annotation is necessary for Pydantic to correctly handle the field upon class creation in order to support some of my other functionality, so it cannot be removed.