scrapy/itemadapter

Recursive conversion to a dictionary

kmike opened this issue · 0 comments

kmike commented

It would be cool to be able to convert an item to a nested dictionary. Both with dataclasses and with attr.s you can define nested data:

import attr

@attr.s(auto_attribs=True)
class Price:
    value: float   # fixme: Decimal?
    currency: str

@attr.s(auto_attribs=True)
class Product:
    name: str
    price: Price 

Such structures are common in real-world usage of these packages. For example, attr.s provides an utility to convert a nested structure to a dict: https://www.attrs.org/en/stable/api.html#attr.asdict.

That's fine to add this feature later, but my worry is that the currently suggested way to convert ItemAdapter to a dict (dict(adapter)) is not future-proof; I'm not sure how to add nesting support to it in future, if we decide to. Likely it would require updating of all examples.

So, what do you think about adding a method (something like ItemAdapter.asdict()), and recommending it? It may support recursive export from the day 1, or just call dict(self) - in this case recursive=True flag can be added later.