Subclass registration with generic decorator subclass fails

Question

Subclass registration with generic decorator subclass fails

Closed this issue 4 days ago · 5 comments

cattrs version: 24.1.0
Python version: 3.11.9
Operating System: Ubuntu 22.04

Description

I am using the decorator pattern to generate a decorator class Decorated from an abstract base class Base.

from abc import abstractmethod
from attrs import define


@define
class Base:
    @abstractmethod
    def meth(self): ...


@define
class Decorated(Base):
    base: Base

    def meth(self):
        pass

Since Base has many more subclasses and is widely used as an annotation I want to register all subclasses of Base with my converter.

from cattrs import Converter
from cattrs.strategies import include_subclasses

converter = Converter()
include_subclasses(Base, converter)

This works fine if Decorated is not a generic class, but as soon as I make the Base typed field generic,

from typing import Generic, TypeVar

T = TypeVar("T", bound=Base)

@define
class Decorated(Base, Generic[T]):
    base: T

    def meth(self):
        pass

include_subclasses(Base, converter)

the subclass registration for Base fails with

Traceback (most recent call last):
  File "/home/valentin/python/pCloudPython/misc/cattrs_playground/decorator_pattern_subclass_registration.py", line 38, in <module>
    include_subclasses(Base, converter)
  File "/home/valentin/.pyenv/versions/3.11.9/envs/os11/lib/python3.11/site-packages/cattrs/strategies/_subclasses.py", line 75, in include_subclasses
    _include_subclasses_without_union_strategy(
  File "/home/valentin/.pyenv/versions/3.11.9/envs/os11/lib/python3.11/site-packages/cattrs/strategies/_subclasses.py", line 117, in _include_subclasses_without_union_strategy
    dis_fn = converter._get_dis_func(subclass_union, overrides=overrides)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/valentin/.pyenv/versions/3.11.9/envs/os11/lib/python3.11/site-packages/cattrs/converters.py", line 976, in _get_dis_func
    return create_default_dis_func(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/valentin/.pyenv/versions/3.11.9/envs/os11/lib/python3.11/site-packages/cattrs/disambiguators.py", line 61, in create_default_dis_func
    overrides = [
                ^
  File "/home/valentin/.pyenv/versions/3.11.9/envs/os11/lib/python3.11/site-packages/cattrs/disambiguators.py", line 62, in <listcomp>
    getattr(converter.get_structure_hook(c), "overrides", {}) for c in classes
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/valentin/.pyenv/versions/3.11.9/envs/os11/lib/python3.11/site-packages/cattrs/converters.py", line 574, in get_structure_hook
    self._structure_func.dispatch(type)
  File "/home/valentin/.pyenv/versions/3.11.9/envs/os11/lib/python3.11/site-packages/cattrs/dispatch.py", line 134, in dispatch_without_caching
    res = self._function_dispatch.dispatch(typ)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/valentin/.pyenv/versions/3.11.9/envs/os11/lib/python3.11/site-packages/cattrs/dispatch.py", line 76, in dispatch
    return handler(typ)
           ^^^^^^^^^^^^
  File "/home/valentin/.pyenv/versions/3.11.9/envs/os11/lib/python3.11/site-packages/cattrs/converters.py", line 1290, in gen_structure_attrs_fromdict
    return make_dict_structure_fn(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/valentin/.pyenv/versions/3.11.9/envs/os11/lib/python3.11/site-packages/cattrs/gen/__init__.py", line 772, in make_dict_structure_fn
    return make_dict_structure_fn_from_attrs(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/valentin/.pyenv/versions/3.11.9/envs/os11/lib/python3.11/site-packages/cattrs/gen/__init__.py", line 349, in make_dict_structure_fn_from_attrs
    raise StructureHandlerNotFoundError(
cattrs.errors.StructureHandlerNotFoundError: Missing type for generic argument T, specify it when structuring.

This happens at registration time, not at (un)structuring time.

Is this expected/intended? Can this be mitigated?

Answer 1 · 2024-11-05T22:51:53.000Z

Hello, sorry for the delay, I have a baby nowdays.

The problem is I don't think include_subclasses is a good fit for generic classes. cattrs can structure a Decorated[int] for you easily, but it can't really structure a Decorated[T] because it doesn't know what T is. (Even if I made provisions to treat it as Decorated[Any], the behavior wouldn't be all that useful.)

I don't really know how to improve the situation.

Answer 2 · 2024-11-13T10:25:45.000Z

Hi! First of all congratulations!! Thanks for the reply and sorry for my late reply. I understand very well, I have two little kids too 😁

🤔 I probably don't understand the mechanics enough, but why is it not possible to just register the generic class with unknown T and sort of register a fully specified unstructure hook upon unstructuring a concrete fully specified type, e.g. Decorated[int]? The general Decorated case could be caught by the existing handler (which doesn't know T) yet, which in turn dispatches a concrete version for Decorated[int].

Is the problem, that subclass registration should be closed in itself, i.e. there should be nothing unknown and left undone once it's called?

Answer 3 · 2024-11-23T21:35:16.000Z

Not sure what exactly you're suggesting. The problem here is structuring, unstructuring should work just fine (and this is the error you see - StructureHandlerNotFoundError). The strategy fails because it's attempting to set up both structuring and unstructuring. If you need just unstructuring, we can probably accomodate this.

When speaking about structuring, the problem is this: assume the strategy worked, what would converter.structure(payload, Base) return in the Decorated case? A Decorated[Any]?

You could do something like this:

converter = Converter()

converter.register_structure_hook(
    Decorated, make_dict_structure_fn(Decorated[Any], converter)
)
include_subclasses(Base, converter)

print(converter.structure({"base": 1}, Base))

This will perform no structuring/validation on the contents of the base field (that's the default Any behavior) but structure everything else.

Answer 4 · 2024-11-28T10:15:22.000Z

Hi! Yeah I'd have to think about it in a bit more detail. Yes indeed, it's always the structuring part that's tricky, unstructuring is (almost always) trivial.

What I'm thinking is that, well, the converter really only needs to know the fully parameterized type at structuring time, not at registration time. Perhaps it is possible to register the generic type as incomplete in the whole subclass registration dance with tagged union strategy (you know, generating concrete structure fns with e.g. make_dict_structure_fn and register_structure_hook_funcing that general structure function for any superclass that gets the concrete class name from a tag field and the concrete structuring fn from a separate tag_to_hook mapping).

I thought that you could register a generic in some incomplete format in this hierarchy and really only at structuring time (there you need the fully parameterized type to be passed to the converter of course) you generate the actual structuring fn with make_dict_structure_fn. Not sure if a second mapping is necessary for that.

I will think about it and try to make my own solution for it. If I manage I'd be happy to show you if you're interested :-)

Answer 5 · 2024-11-28T13:39:58.000Z

Sure, let me know what you come up with!