rnag/dataclass-wizard

[BUG]: Incomplete Support for Union & NotRequired

Closed this issue · 6 comments

adamcunnington-mlg commented

Issues

There are 2 issues I've stumbled across when combining some semi-complex types:

  1. Low-impact as can be workaround by Using Union[...] or Optional: You can't use X | Y syntax < python3.10 with from __future__ import annotations if the types are not simple. In my case, it was when I was using a TypedDict(...) | None. I got crazy syntax error with incomplete terminations. E.g.:
meta: TypedDict("Meta", {"superset": NotRequired[ColumnMetaSuperset]}, total=False) | None = None
SyntaxError: TypedDict('Meta',{'superset':NotRequired['ColumnMetaSuperset']},Union[total=False),None]

Root cause suggestion from community member who helped me debug was:

My guess is that third party library created an override for type and not _TypedDictMeta
(which allows types to be or'ed in its annotations, but only if they do not have a metaclass)
Probably the same issue as https://github.com/rnag/dataclass-wizard/issues/118 , since StrEnum also uses a metaclass IIRC
  1. High-impact as there is no workaround: It looks like NotRequired can't be used in conjunction with Union/Optional
meta: Union[TypedDict("Meta", {"superset": NotRequired[ColumnMetaSuperset]}, total=False), None] = None
TypeError: issubclass() arg 1 must be a class

Again, suggestion from community member:
"I think it has to due with the Union[..., None]; the Optional parser assumes the first argument does not need a custom parser"

adamcunnington-mlg commented

@rnag here's the traceback to help investigation of point 2. I'm trying to understand what fix is needed but this get_parser_for_annotation function is complex.

You can see from the traceback below that I am printing base type (I've inserted print(base_type) on L341 of loaders.py). It's the specific flow of an optional/union followed by a NotRequired (which is used within TypedDict). You handle NotRequired outside of optional context but not within. I'd be really grateful for a fix/workaround here.

typing.Union
typing_extensions.NotRequired
Traceback (most recent call last):
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/loaders.py", line 534, in fromdict
    load = _CLASS_TO_LOAD_FUNC[cls]
KeyError: <class '__main__.Common'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/loaders.py", line 273, in get_parser_for_annotation
    base_type = get_origin(ann_type, raise_=True)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/utils/typing_compat.py", line 345, in get_origin
    return _get_origin(cls, raise_=raise_)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/utils/typing_compat.py", line 177, in _get_origin
    return cls.__origin__
AttributeError: type object 'Metric' has no attribute '__origin__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/loaders.py", line 273, in get_parser_for_annotation
    base_type = get_origin(ann_type, raise_=True)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/utils/typing_compat.py", line 345, in get_origin
    return _get_origin(cls, raise_=raise_)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/utils/typing_compat.py", line 177, in _get_origin
    return cls.__origin__
AttributeError: type object 'TypeParams' has no attribute '__origin__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ac/projects/medialabgroup/misc/scripts/mart-linter.py", line 810, in <module>
    main()
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/ac/projects/medialabgroup/misc/scripts/mart-linter.py", line 799, in main
    LinterWorkflow(bq_client, bq_dataset_name, prompter=questionary.prompt).run(mart_yaml_file_paths, fix)
  File "/home/ac/projects/medialabgroup/misc/scripts/mart-linter.py", line 747, in run
    self.common = Common.from_yaml_file(self.COMMON_FILE_PATH)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/wizard_mixins.py", line 147, in from_yaml_file
    return cls.from_yaml(in_file, decoder=decoder,
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/wizard_mixins.py", line 136, in from_yaml
    return fromdict(cls, o) if isinstance(o, dict) else fromlist(cls, o)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/loaders.py", line 536, in fromdict
    load = load_func_for_dataclass(cls)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/loaders.py", line 583, in load_func_for_dataclass
    field_to_parser = dataclass_field_to_load_parser(cls_loader, cls, config)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/class_helper.py", line 120, in dataclass_field_to_load_parser
    return _setup_load_config_for_cls(cls_loader, cls, config, save)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/class_helper.py", line 189, in _setup_load_config_for_cls
    name_to_parser[f.name] = cls_loader.get_parser_for_annotation(
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/loaders.py", line 373, in get_parser_for_annotation
    return MappingParser(
  File "<string>", line 5, in __init__
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/parsers.py", line 502, in __post_init__
    self.val_parser = get_parser(val_type, cls, extras)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/loaders.py", line 287, in get_parser_for_annotation
    load_hook = load_func_for_dataclass(
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/loaders.py", line 583, in load_func_for_dataclass
    field_to_parser = dataclass_field_to_load_parser(cls_loader, cls, config)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/class_helper.py", line 120, in dataclass_field_to_load_parser
    return _setup_load_config_for_cls(cls_loader, cls, config, save)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/class_helper.py", line 189, in _setup_load_config_for_cls
    name_to_parser[f.name] = cls_loader.get_parser_for_annotation(
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/loaders.py", line 353, in get_parser_for_annotation
    return OptionalParser(
  File "<string>", line 4, in __init__
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/parsers.py", line 156, in __post_init__
    self.parser: AbstractParser = get_parser(self.base_type, cls, extras)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/loaders.py", line 319, in get_parser_for_annotation
    return TypedDictParser(
  File "<string>", line 5, in __init__
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/parsers.py", line 545, in __post_init__
    self.key_to_parser: FieldToParser = {
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/parsers.py", line 546, in <dictcomp>
    k: get_parser(v, cls, extras)
  File "/home/ac/projects/medialabgroup/misc/scripts/.venv/lib/python3.9/site-packages/dataclass_wizard/loaders.py", line 364, in get_parser_for_annotation
    elif issubclass(base_type, defaultdict):
TypeError: issubclass() arg 1 must be a class
adamcunnington-mlg commented

@rnag are you able to take a look at this? is this project still maintained?

rnag commented

@adamcunnington-mlg I need more info on the second part of your ask:

High-impact as there is no workaround: It looks like NotRequired can't be used in conjunction with Union/Optional

Can you provide an example of how you're using either in conjunction with Union/Optional?

The below is invalid. TypedDict just does not seem to like it at all, and the field becomes required by default.

my_int: Optional[NotRequired[int]]

Assuming you mean below, it should be solved thanks to #125, but I'm not 100% sure if that's what you meant.

my_int: NotRequired[Optional[int]]
rnag commented

FYI, I added release with updates from #125 to v0.24.0. Please check it out and let me know if you still notice issues. Thanks

adamcunnington-mlg commented

Thanks - yes I meant that - and I can confirm the merged PR fixes it - thanks for releasing. The former issue remains but it's not important as there is a workaround.

  • 👍2
  • ❤️1
rnag commented

Closing for now, if you still experience the issue feel free to re-create, or if you are able to create a PR with a workaround, that would be great as well. Thanks!