home-assistant/core

Pydantic v2 migration issue

joostlek opened this issue · 33 comments

joostlek commented

The problem

Pydantic released v2 of their library. Some libraries want to utilize the new features of Pydantic v2 so HA should update to Pydantic v2. To not burden all libraries to rework, Pydantic is providing v1 in a separate package of v2. This has been implemented in several libraries already made by some HA community members (https://github.com/AngellusMortis/pyunifiprotect/pull/297). This way the library supports both v1 and v2, this way of working is preferred to not burden the v2 upgrade with a lot of libraries to update at once.

After installing all dependencies and using pipdeptree we see that the current libraries use pydantic.

pydantic==1.10.12
├── aiolivisi==0.0.19 [requires: pydantic]
├── aionotion==2023.5.5 [requires: pydantic>=1.10.7,<2.0.0]
├── aioopenexchangerates==0.4.0 [requires: pydantic>=1.9,<2.0]
├── aiopurpleair==2022.12.1 [requires: pydantic>=1.10.2,<2.0.0]
├── aiowaqi==0.2.1 [requires: pydantic>=1.10.8]
├── demetriek==0.4.0 [requires: pydantic>=1.9.0,<2.0.0]
├── elgato==4.0.1 [requires: pydantic>=1.8.0,<2.0.0]
├── gcal-sync==4.1.4 [requires: pydantic>=1.9.0,<2.0a]
├── google-nest-sdm==2.2.5 [requires: pydantic>=1.10.4]
├── ical==5.0.1 [requires: pydantic>=1.9.1]
│   ├── gcal-sync==4.1.4 [requires: ical>=4.2.5]
│   └── pyrainbird==4.0.0 [requires: ical>=4.2.9]
├── inflect==6.0.4 [requires: pydantic>=1.9.1]
│   ├── jaraco.itertools==6.2.1 [requires: inflect]
│   │   └── jaraco.abode==3.3.0 [requires: jaraco.itertools]
│   └── jaraco.text==3.11.1 [requires: inflect]
│       ├── jaraco.collections==4.2.0 [requires: jaraco.text]
│       │   ├── jaraco.abode==3.3.0 [requires: jaraco.collections]
│       │   ├── jaraco.email==3.1.0 [requires: jaraco.collections]
│       │   │   └── jaraco.net==9.3.1 [requires: jaraco.email]
│       │   │       └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
│       │   └── jaraco.net==9.3.1 [requires: jaraco.collections]
│       │       └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
│       ├── jaraco.email==3.1.0 [requires: jaraco.text>=1.3]
│       │   └── jaraco.net==9.3.1 [requires: jaraco.email]
│       │       └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
│       └── jaraco.net==9.3.1 [requires: jaraco.text]
│           └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
├── intellifire4py==2.2.2 [requires: pydantic]
├── lacrosse-view==1.0.1 [requires: pydantic>=1.9.0]
├── open-meteo==0.2.1 [requires: pydantic>=1.8.0,<2.0.0]
├── peco==0.0.29 [requires: pydantic>=1.9.0]
├── pvo==1.0.0 [requires: pydantic>=1.8.0,<2.0.0]
├── pyaussiebb==0.0.15 [requires: pydantic>=1.9.0,<2.0.0]
├── pyrainbird==4.0.0 [requires: pydantic>=1.10.4]
├── python-bsblan==0.5.11 [requires: pydantic>=1.9.0]
├── python-kasa==0.5.3 [requires: pydantic>=1,<2]
├── python-opensky==0.2.0 [requires: pydantic>=1.10.8]
├── pytraccar==1.0.0 [requires: pydantic>=1,<2]
├── pyunifiprotect==4.10.6 [requires: pydantic!=1.9.1]
├── radios==0.1.1 [requires: pydantic>=1.9,<2.0]
├── sfrbox-api==0.0.6 [requires: pydantic>=1.10.2]
├── systembridgeconnector==3.4.9 [requires: pydantic>=1.9.0]
├── tailscale==0.2.0 [requires: pydantic>=1.8.0,<2.0.0]
├── vehicle==1.0.1 [requires: pydantic>=1.8.0,<2.0.0]
├── withings-api==2.4.0 [requires: pydantic>=1.7.2,<2.0.0]
├── xbox-webapi==2.0.11 [requires: pydantic]
├── yolink-api==0.3.0 [requires: pydantic>=1.9.0]
├── youtubeaio==1.1.5 [requires: pydantic>=1.10.8]
└── zwave-js-server-python==0.50.1 [requires: pydantic>=1.10.0]

This issue will be the progress tracker of this migration.

❌ aiolivisi Library: [aiolivisi](https://github.com/StefanIacobLivisi/aiolivisi) Integrations using this library: [Livisi](https://github.com/home-assistant/core/tree/dev/homeassistant/components/livisi) Code owners: @StefanIacobLivisi @planbnet

Remarks:

✔️ aionotion

Library: aionotion
Integrations using this library: Notion
Code owners: @bachya

Remarks:
There seems to be no version that supports both v1 and v2, so this library should be directly bumped with the v2 update.

✔️ aioopenexchangerates

--
Library: aioopenexchangerates
Integrations using this library: OpenExchangeRates
Code owners: @MartinHjelmare

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: #125593 )

Remarks:

❌ aiopurpleair

--
Library: aiopurpleair
Integrations using this library: PurpleAir
Code owners: @bachya

Remarks:
There seems to be no version that supports both v1 and v2, so this library should be directly bumped with the v2 update.

✔️ aiowaqi

--
Library: aiowaqi
Integrations using this library: WAQI
Code owners: @joostlek

  • Supports Pydantic v2 (PR: Initial commit)
  • Is updated in HA (PR: #98000)

Remarks:

❌ demetriek

--
Library: demetriek
Integrations using this library: lametric
Code owners: @robbiet480 @frenck @bachya

Remarks:

✔️ elgato

--
Library: elgato
Integrations using this library: elgato
Code owners: @frenck

Remarks:

✔️ gcal-sync

--
Library: gcal-sync
Integrations using this library: Google Calendar
Code owners: @allenporter

Remarks:
Supports shims. Needs a release so we can bump the dependency beforehand. Latest version also has ical 5.0.1, and that has pydantic v2 support since 5.0.0.

✔️ google-nest-sdm

--
Library: google-nest-sdm
Integrations using this library: Nest
Code owners: @allenporter

Remarks:

✔️ ical

--
Library: ical
Integrations using this library: Local calendar
Code owners: @allenporter

Remarks:

❌ inflect

--
Library: inflect
Integrations using this library: Abode
Code owners: @shred86

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: # )

Remarks:
This seems to be a deeper problem. I need to dive in to this to know the status.

❌ intellifire4py

--
Library: intellifire4py
Integrations using this library: IntelliFire
Code owners: @jeeftor

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: # )

Remarks:

❌ lacrosse-view

--
Library: lacrosse-view
Integrations using this library: LaCrosse View
Code owners: @IceBotYT

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: # )

Remarks:

✔️ open-meteo

--
Library: open-meteo
Integrations using this library: Open Meteo
Code owners: @frenck

Remarks:
To verify: There has been no code change to support it, but in the lock file is 2.0.3 so I am expecting this works.
If this is as expected, it still needs a release and to be bumped.

❌ peco

--
Library: peco
Integrations using this library: PECO Outage Counter
Code owners: @IceBotYT

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: # )

Remarks:
There has been no explicit PR to fix support for v2. This might work but needs to be verified.

✔️ pvo

--
Library: pvo
Integrations using this library: PVOutput
Code owners: @frenck

Remarks:

❌ pyaussiebb

--
Library: pyaussiebb
Integrations using this library: Aussie Broadband
Code owners: @nickw444 @Bre77

  • Supports Pydantic v2 (commit)
  • Is updated in HA (PR: #99077)

Remarks:
Library does not support v1 shims. Should be bumped with the v2 bump.
cc @yaleman

✔️ pyrainbird

--
Library: pyrainbird
Integrations using this library: Rain Bird
Code owners: @konikvranik @allenporter

Remarks:
Still needs to update ical to =>5.0.0 as ical relies on pydantic as well.

✔️ python-bsblan

--
Library: python-bsblan
Integrations using this library: BSB-Lan
Code owners: @liudger

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: # )

Remarks:

✔️ python-kasa

--
Library: python-kasa
Integrations using this library: TP-Link Kasa Smart
Code owners: @rytilahti @TheGardenMonkey

Remarks:

✔️ python-opensky

--
Library: python-opensky
Integrations using this library: OpenSky Network
Code owners: @joostlek

Remarks:

✔️ pytraccar

--
Library: pytraccar
Integrations using this library: Traccar
Code owners: @ludeeus

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: #103318)

Remarks:

✔️ pyunifiprotect

--
Library: pyunifiprotect
Integrations using this library: asd
Code owners: @AngellusMortis @bdraco

Remarks:

✔️ radios

--
Library: radios
Integrations using this library: Radio Browser
Code owners: @frenck

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: #103614)

Remarks:

✔️ sfrbox-api

--
Library: sfrbox-api
Integrations using this library: SFR Box
Code owners: @epenet

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: #125732 )

Remarks:

❌ systembridgeconnector

--
Library: systembridgeconnector
Integrations using this library: System Bridge
Code owners: @timmo001

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: # )

Remarks:

✔️ tailscale

--
Library: tailscale
Integrations using this library: asd
Code owners: @frenck

Remarks:
To verify: There has been no code change to support it, but in the lock file is 2.0.3 so I am expecting this works.
If this is as expected, it still needs a release and to be bumped.

✔️ vehicle

--
Library: vehicle
Integrations using this library: RDW
Code owners: @frenck @joostlek

Remarks:

✔️ withings-api

--
Library: withings-api
Integrations using this library: Withings
Code owners: @vangorra

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: #101819)

Remarks:
Is now/will be replaced with a new lib without pydantic

❌ xbox-webapi

--
Library: xbox-webapi
Integrations using this library: Xbox
Code owners: @hunterjm

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: # )

Remarks:

✔️ yolink-api

--
Library: yolink-api
Integrations using this library: YoLink
Code owners: @matrixd2

Remarks:

✔️ youtubeaio

--
Library: youtubeaio
Integrations using this library: YouTube
Code owners: @joostlek

  • Supports Pydantic v2 (PR: )
  • Is updated in HA (PR: # )

Remarks:

✔️ zwave-js-server-python

--
Library: zwave-js-server-python
Integrations using this library: Z-Wave
Code owners: @home-assistant/z-wave

Remarks:

--

For the core members, feel free to update where needed. For others, let us know if something has updated in a comment and we will update it.

cc @cdce8p

frenck commented

As also reported on Discord. But replying here as well, as I was tagged. I have no intention of supporting v2 of Pydantic in my libraries.

Instead, I might be re-evaluating the use of Pydantic overall.

../Frenck

  • 👍1
raman325 commented

@frenck if you do find a good alternative please do share it here. Others may want to choose the same option. I have v2 semi working but since all the internals have changed and are IMO more complicated I haven't been interested much in investing time into it

bachya commented

A good reminder: one can retain the V1 standards while upgrading to the V2 library. This is an excellent way to unstick this PR without committing yourself to a complete upgrade (assuming they don't get rid of this path at some point).

https://docs.pydantic.dev/latest/migration/#continue-using-pydantic-v1-features

  • 👍2
frenck commented

As far as I am aware, it doesn't support everything backward. As said, I have no plans of supporting V2 at this point, and I am not willing to spend any second on it until I have explored other options.

../Frenck

  • 👎2
frenck commented

pydantic has been fully removed from tailscale and elgato (and thus no longer taking part in this issue).

  • ❤️1
frenck commented

pydantic has been fully removed from vehicle (RDW integration) and thus no longer taking part in this issue.

joostlek commented

I've been checking all the boxes :)

frenck commented

Right, but those don't take part in the migration process. It simply aren't using Pydantic at all anymore.

frenck commented

pydantic has been fully removed from pvo (PVOutput integration) and thus no longer taking part in this issue.

frenck commented

pydantic has been fully removed from open-meteo and thus no longer taking part in this issue.
PR @ Core: #103613

frenck commented

pydantic has been fully removed from radios (Radio Browser integration) and thus no longer taking part in this issue.
PR @ Core: #103614

frenck commented

Man, what a liberating feeling! I should have learned about other options before using Pydantic in the first place. It would have saved me from many headaches now and in the past.

  • 👎1
raman325 commented

@frenck Are you using mashumaro as a replacement? I ask because I don't see a way to use it to validate a dict based on a TypedDict which is what I am looking for, so may have to keep looking :/

AlexanderLanin commented

FYI: beartype

I found this issue, while searching why the used pydantic version is so old.
Maybe it's just me, but I then found beartype which I've never heard of before.
I've switched from pydantic to beartype today.
I'm still astonished how simple that was and how it immediately found the bug, that already cost me several hours today.

Steps I did:

  • remove every mention of pydantic
  • into __init__.py add:
from beartype.claw import beartype_this_package

beartype_this_package()
joostlek commented

Do you have a repo which we can look at?

AlexanderLanin commented

Sure, although I'm not sure it's worth looking: https://github.com/AlexanderLanin/homeassistant-sungrow/blob/main/custom_components/__init__.py

So far it found incorrect assignment to dataclasses and incorrect parameter types in functions. Both in addition to running mypy.

See https://beartype.readthedocs.io/en/latest/ for more info.

  • 👍1
  • 🎉1
kquinsland commented

See https://beartype.readthedocs.io/en/latest/ for more info.

That looks pretty promising. Thank you for sharing!
I - and I think others - get benefit from the ser/de + coercion aspects of Pydantic which beartype doesn't do :(.

jeeftor commented

I think this worked for my codebase:

try:
    from pydantic.v1 import Field, validator  # type: ignore # noqa F401 # pragma: no cover
    from pydantic.v1 import BaseModel  # type: ignore # pragma: no cover
except ImportError:
    from pydantic import Field  # type: ignore # pragma: no cover
    from pydantic import BaseModel  # type: ignore # pragma: no cover
  • 👍3
timmo001 commented

Pydantic will be removed from systembridgeconnector in #107957

The dependent package systembridgemodels now only uses standard dataclasses.

  • ❤️1
bachya commented

aionotion now uses mashumaro and can be removed from the list.

  • ❤️1
rytilahti commented

fyi, pydantic/pydantic#9042 will allow cleaning up the compat code a bit when it ships.

  • 👍1
jaraco commented

FYI, the latest release of inflect no longer depends on pydantic.

KyeRussell commented

fyi, pydantic/pydantic#9042 will allow cleaning up the compat code a bit when it ships.

This has now shipped, and is available in Pydantic 1.10.15.

AngellusMortis commented

fyi, pydantic/pydantic#9042 will allow cleaning up the compat code a bit when it ships.

This has now shipped, and is available in Pydantic 1.10.15.

It does not work. They also have no plans on fixing since v1 is out of support. They are working on v3 now and we still have not even gotten to v2. The Xbox integration (@hunterjm) is the only integration with any notable amount of users. Can we just consider moving to v2 or at least set a deadline for the integrations to migrate?

Owners refusing to support v2 via the shims is not fair to anyone else and it is preventing anyone from being able to use the performance improvements of v2. And once v3 comes out, it means a bunch of integrations are either going to go abandoned or those few owners are going to force everyone to rewrite their libraries just because they disagree with out a tool decided to implement something.

  • 👍1
joostlek commented

I want to spend more time on this when I am done with my thesis, I definetly want to speed the whole process up then

bdraco commented

xbox seems to support v2 now so it might be able to be bumped with v2 as well

OpenXbox/xbox-webapi-python@307a36d

KyeRussell commented

It does not work. They also have no plans on fixing since v1 is out of support. They are working on v3 now and we still have not even gotten to v2.

Unsure what you are referring to. Are you referring to this comment?

Indeed, looks like things weren't fully implemented there. PR welcome with a fix! This is relatively low priority though, given that it's on the V1 end of things.

There is by all accounts an active effort to fix / improve the v1 shim. The comment is just saying that it's low priority, and quite rightfully.

  • 👍1
IceBotYT commented

PECO should be compatible once #117165 gets merged

IceBotYT commented

PECO now supports Pydantic v2 and is now merged into HA 👍

  • ❤️2
Kludex commented

If helped is needed to migrate to V2, I can help.

  • 👍1
drc38 commented

If helped is needed to migrate to V2, I can help.

See MartinHjelmare/aioopenexchangerates#96 as @MartinHjelmare was wanting help to convert to mashumaro

MartinHjelmare commented

aioopenexchangerates has been updated to remove pydantic.

epenet commented

sfrbox-api has been updated to remove pydantic: #125732