[BUG] Setup failed for mobile_app
shmick opened this issue · 61 comments
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
Updated to 2023.6 and now get this error:
2023-06-07 17:17:59.843 ERROR (MainThread) [homeassistant.setup] Setup failed for mobile_app: Unable to import component: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/usr/lib/python3.10/site-packages/urllib3/util/ssl_.py)
HA 2023.6 moved to python 3.11, I see the log is referencing python 3.10, perhaps this is related?
Expected Behavior
No errors
Steps To Reproduce
Update to 2023.6
Environment
- OS: raspberry pi OS
- How docker service was installed: docker apt repoCPU architecture
arm64
Docker creation
version: "3.8"
services:
homeassistant:
container_name: homeassistant
image: lscr.io/linuxserver/homeassistant
network_mode: host
environment:
- PUID=1000
- PGID=1000
- TZ=America/Toronto
depends_on:
- mariadb
volumes:
- ./config-homeassistant:/config
restart: unless-stopped
mariadb:
image: lscr.io/linuxserver/mariadb:latest
container_name: mariadb
environment:
- PUID=1000
- PGID=1000
- TZ=America/Toronto
- MYSQL_DATABASE=hass
- MYSQL_USER=hass
- MYSQL_PASSWORD=<>
- MYSQL_ROOT_PASSWORD=<>
volumes:
- ./config-mariadb:/config
ports:
- 3306:3306
restart: unless-stopped
### Container logs
```bash
homeassistant | 2023-06-07 17:17:59.843 ERROR (MainThread) [homeassistant.setup] Setup failed for mobile_app: Unable to import component: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/usr/lib/python3.10/site-packages/urllib3/util/ssl_.py)
homeassistant | warnings.warn(
homeassistant |
homeassistant | 2023-06-07 17:19:03.253 WARNING (MainThread) [homeassistant.bootstrap] Support for the running Python version 3.10.11 is deprecated and will be removed in Home Assistant 2023.8; Please upgrade Python to 3.11
homeassistant | 2023-06-07 17:20:01.755 ERROR (MainThread) [homeassistant.components.automation.frontcam_snapshot] Frontcam Snapshot: Parallel action at step 3: parallel 3: Error executing script. Service not found for call_service at pos 1: Unable to find service notify.mobile_app_shmickphone13
homeassistant | 2023-06-07 17:20:02.755 ERROR (MainThread) [homeassistant.components.automation.frontcam_snapshot] Frontcam Snapshot: Error executing script. Service not found for parallel at pos 3: Unable to find service notify.mobile_app_shmickphone13
Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.
Same here after 2023.6 update
Same here :)
I think the legacy resolver needs to be removed:
docker-homeassistant/Dockerfile
Line 13 in dc7ddb3
Thanks for the heads up. We already pushed the python 3.11 wheels to the repo. We'll do a quick rebase to alpine 3.18 shortly
Can you guys test the builds from here and let us know if it works fine?
https://ci-tests.linuxserver.io/lspipepr/homeassistant/2023.6.0-pkg-8c9a9f56-dev-40519d9addcf47a7361f01821b35df617c673078-pr-71/index.html
Can you guys test the builds from here and let us know if it works fine?
mobile_app and cloud integrations are working fine now. I have issues with a few custom integrations: simpleicons, pirateweather, ui_lovelace_minimalist and weback_vacuum. Logs below
first this:
Logger: homeassistant.util.package
Source: util/package.py:107
First occurred: 08:53:45 (12 occurrences)
Last logged: 08:56:37
Unable to install package simpleicons==7.14.0: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/lsiopy/lib/python3.11/site-packages/simpleicons' Check the permissions. WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command.
Unable to install package python-forecastio==1.4.0: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/lsiopy/lib/python3.11/site-packages/yaml-stubs' Check the permissions. WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command.
Unable to install package aiofiles==0.8.0: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'WHEEL' Check the permissions. WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command.
Unable to install package websocket-client==1.4.2: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/lsiopy/bin/wsdump' Check the permissions. WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command.
then this:
Logger: homeassistant.setup
Source: setup.py:207
First occurred: 08:54:13 (4 occurrences)
Last logged: 08:56:37
Setup failed for custom integration simpleicons: Requirements for simpleicons not found: ['simpleicons==7.14.0'].
Setup failed for custom integration pirateweather: Requirements for pirateweather not found: ['python-forecastio==1.4.0'].
Setup failed for custom integration ui_lovelace_minimalist: Requirements for ui_lovelace_minimalist not found: ['aiofiles==0.8.0'].
Setup failed for custom integration weback_vacuum: Requirements for weback_vacuum not found: ['websocket-client==1.4.2'].
simpleicons,pirateweather,ui_lovelace_minimalistandweback_vacuum
These seems to be HACS components, which might need to be adapted(likely reinstalled) to use the same venv as hass itself.
Sorry for the probably dumb question, but how would one use that artifact ? I don't see anything on that page that looks like a docker repo link, but I'm on mobile so maybe I'm missing it
These seems to be HACS components, which might need to be adapted(likely reinstalled) to use the same venv as hass itself.
Reinstalling didn't work for me (both through redownload function or by removing and installing again). The error shows something about permissions, are you sure it's not the issue with container itself?
how would one use that artifact
Just replace lscr.io/linuxserver/homeassistant:latest with lspipepr/homeassistant:amd64-2023.6.0-pkg-8c9a9f56-dev-40519d9addcf47a7361f01821b35df617c673078-pr-71 in your docker-compose or in the command you use to run the container. (This one is for amd64 but you can change the url to point to the one for arm if you need)
Ah okay, I was trying to get that from lsci.io, I didn't realize these were on the official hub. That works, thanks.
Now I'm having what I imagine is the same issue as you, a bunch of my custom integrations aren't loading, for example :
Unable to install package pymyenergi==0.0.27: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/lsiopy/lib/python3.11/site-packages/pymyenergi' Check the permissions. WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command.
Looks like the permissions aren't the same anymore.
That said the switchbot and mobile official integrations did load fine this time, so at least that build does address the initial issue :)
Confirmed the test build resolves the mobile_app and cloud issue, however I now get the same missing dependency issue for HACs integrations mentioned by pacjo. Reinstalling the integrations does not resolve that issue. I don't get any permission errors like above, just the missing dependency error.
Logger: homeassistant.setup
Source: setup.py:207
First occurred: 10:21:18 AM (1 occurrences)
Last logged: 10:21:18 AM
Setup failed for custom integration tapo_control: Requirements for tapo_control not found: ['pytapo==3.1.12'].
EDIT: Correction, I do now have the permission error.
Logger: homeassistant.util.package
Source: util/package.py:107
First occurred: 10:21:00 AM (3 occurrences)
Last logged: 10:21:18 AM
Unable to install package pytapo==3.1.12: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/lsiopy/lib/python3.11/site-packages/rtp' Check the permissions. WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command.
Long story short, python 3.11 forces us to use a venv with pip. HA runs as an unprivilieged user. Previously, unprivilieged users running pip would get packages installed in their home folders and no perm issues there. But now we have a venv in the container so unprivileged users are also expected to use it and install things there. But the venv is owned by the root user.
A fix is incoming for the perm issue. However, since the venv is inside the container all those HACS installed packages would need to be reinstalled on container recreation (still need to test if HACS handles auto installs on start).
Could we just mount that directory as a volume, to make it persistent or will that mess with the venv ?
Sorry, not familiar with how venvs work.
@aptalca I'm not sure how true venv is needed here, or maybe it's a limitation with s6. I build a rootless container without s6-overlay over at my containers repo without having those issues. I updated last night and tested installing hacs components without a problem, existing ones were also fine. I just had to fix the legacy pip resolver as mentioned above.
If you want to be really pedantic, a venv isn't required, however with 3.11 Python is pushing hard against mixing install sources (distro repos, pip, etc.) and by default if you install any Python modules (or pip) from the distro repos it will then refuse to install modules from pip and throws a This environment is externally managed error (see PEP 668).
There are several options to work around it, we've gone with a venv because it's least likely to break when they make changes in the future.
Could we just mount that directory as a volume, to make it persistent or will that mess with the venv ?
That folder contains the 1,000+ python packages that we pre-install for HA.
OK, I do have a fix (I hope). I tested with simpleicons installed from HACS with our HACS mod. Upon enabling in HA, it installed the simpleicons python package via pip. All good. Then I recreated the container and HACS reinstalled the python package on HA start. All seems well. a new build should be available for testing at #71 within 20 min or so
Sounds great. One question, will it have to re-install those packages every time the container is started ? Just bouncing off your previous message where you suggested that might be the case.
Having HA essentially depend on the internet on first startup after an update is reasonable enough, just trying to confirm it won't depend on the internet on every startup if you're using hacs
Not restarts, those will be fine. But if you recreate the container, they will need to be reinstalled.
I'll continue investigating whether we can get the venv pip to install stuff to the config folder but I'm not very hopeful.
Great, so if the internet goes out just don't touch the container and it should be alright, noted ! Thanks
New test images pushed: https://ci-tests.linuxserver.io/lspipepr/homeassistant/2023.6.0-pkg-8c9a9f56-dev-0ccfb308b8c24325c4e640683e9058e5f09f8bc1-pr-71/index.html
I still have some issues. simpleicons work fine, but now I have problem with ui_lovelace_minimalist and weback_vacuum. Logs below
1st:
Logger: homeassistant.util.package
Source: util/package.py:107
First occurred: 17:51:16 (6 occurrences)
Last logged: 17:52:24
Unable to install package aiofiles==0.8.0: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'WHEEL' Check the permissions. WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command.
Unable to install package aiofiles==0.8.0: WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'WHEEL' Check the permissions. WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command.
Unable to install package aiofiles==0.8.0: WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'WHEEL' Check the permissions. WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command.
Unable to install package websocket-client==1.4.2: WARNING: Ignoring invalid distribution -ofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '_socket.py' Check the permissions. WARNING: Ignoring invalid distribution -ofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -iofiles (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command.
2nd
Logger: homeassistant.setup
Source: setup.py:207
First occurred: 17:51:43 (2 occurrences)
Last logged: 17:52:24
Setup failed for custom integration ui_lovelace_minimalist: Requirements for ui_lovelace_minimalist not found: ['aiofiles==0.8.0'].
Setup failed for custom integration weback_vacuum: Requirements for weback_vacuum not found: ['websocket-client==1.4.2'].
That new version fixes 5 of my 6 issues, the only one that still isn't loading is the new fork of the dyson local integration :
Setup failed for custom integration dyson_local: Requirements for dyson_local not found: ['libdyson-neon==1.0.2'].
There was just an update to "vendor libdyson" so it might be a different issue, I'll have to look into it properly.
Nice job though, thanks a lot !
The issue with ui_lovelace_minimalist is that it uses aiofiles==0.8.0 as a dep and it is a really old version. HA already installs aiofiles==23.1.0, which is much newer. When you try to install that theme through hacs, it attemps to uninstall the existing newer aiofiles and fails. Because I set the permissions to allow writing/installing new packages, but not delete existing ones.
The other addon has a similar issue where it tries to install websocket-client==1.4.2 but HA already has a newer version installed.
It's really best for the addons to update their dependencies so as not to clash with HA.
If you don't care, you can have a custom script run this on every start and the addons will be allowed to delete HA's python packages and downgrade them:
chown -R abc:abc /lsiopy/lib
See here for info on custom scripts: https://www.linuxserver.io/blog/2019-09-14-customizing-our-containers
So clearly a problem with custom integrations. Good to know.
If it's the same with the issue that @Ulrar is having, then I think it's safe to close this one as this is no longer lsio's problem.
Just tested the latest 2023.6 image and everything is working for me. Thanks for the quick turnaround @aptalca!
Sorry, I've been busy at work so wasn't able to keep up with the thread. I can confirm that the latest test build resolves my issue with the tapo_control custom integration. Thanks everyone!
EDIT: Sorry, I didn't realise it's been pushed to main. Let me install that and test.
EDIT 2: Confirmed release 2023.6.0-ls152 fixes my issue. Thanks again.
Yes, the initial issue reported in the OP has been fixed. There are some issues with custom integrations not being able to downgrade/delete existing HA packages.
As a fix for that, users can either create a custom script on startup to fix permissions on the entire venv folder (may take a while because there are over 47,000 files in there) via chown -R abc:abc /lsiopy
or, they can use our package installer mod to install whatever dependency version the custom integration needs. Since our package installer runs as root, it won't have issues with deleting/downgrading existing packages and it will run on every container start: https://github.com/linuxserver/docker-mods/tree/universal-package-install
Figured out the issue with dyson, it tries to install libdyson-neon which wants to write some files in site-packages/tests.
Chowning it solves the issue but that raises the question, is that common practice and should the tests directory be writable ? Or is it bad practice and should I open an issue on libdyson-neon to stop it from writing there in the first place?
As a fix for that, users can either create a custom script on startup to fix permissions on the entire venv folder (may take a while because there are over 47,000 files in there) via
chown -R abc:abc /lsiopy[...]
- What are the cons of chown-ing the entire folder apart for the slowness?
- Does issuing this command restore the same behavior that was present before 2023.6?
- Maybe a better question would be: in versions before 2023.6, were custom components able to overwrite HA's Python packages?
Thanks.
What are the cons of chown-ing the entire folder apart for the slowness?
Nothing else
in versions before 2023.6, were custom components able to overwrite HA's Python packages?
Yes
Not sure to understand if some manual steps are now required "for ever" potentially for each HACS project? Is this Python 3.11 it an upstream constraint or a linuxserver constraint?
Here are 2 integrations that do not work for me anymore:
Unable to install package google-cloud-speech==2.19.0: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/lsiopy/lib/python3.11/site-packages/google/cloud/speech' Check the permissions. WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command. Unable to install package pytz==2022.7: ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'LICENSE.txt' Check the permissions. WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command. Unable to install package pytz==2022.7: WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'LICENSE.txt' Check the permissions. WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command. Unable to install package pytz==2022.7: WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'LICENSE.txt' Check the permissions. WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution -ytz (/lsiopy/lib/python3.11/site-packages) WARNING: Ignoring invalid distribution - (/lsiopy/lib/python3.11/site-packages) WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available. You should consider upgrading via the '/lsiopy/bin/python3 -m pip install --upgrade pip' command.
Setup failed for custom integration frigate: Requirements for frigate not found: ['pytz==2022.7']
Already mentioned in blakeblackshear/frigate-hass-integration#495
To clarify, most of these issues are because a custom integration uses a pinned version of a dependency package, one that HA already has installed, but a different version. When HACS tries to install that (often) older pinned version, pip tries to uninstall the newer version that comes with HA. That removal of HA's dep is the action that is failing in the newer images.
In the past, HACS installed addons were able to replace HA installed packages with different versions and it didn't complain. I'd argue that behavior is not right. A custom addon should not be able to downgrade a built-in dependency package.
However, if you don't care and want HACS installed custom addons to able to downgrade HA's dependency packages, you can create a custom script to chown the /lsiopy folder so the HA user has the permissions to delete existing packages. That's a one time manual interaction as the custom script will run on every container start, and will recreate the experience users had with the python 3.10 based HA images.
If you want to fix it for a specific integration, you can use our universal package install mod to install the deps for that integration, which will run on every container start and install it.
Since it appears adding virtual env to this container has made updating hard or problematic for people here, I am curious what prompted the change to supporting virtual env? The official hass image doesn't use venv, maybe they had a reason to not pursue it?
change to supporting virtual env?
@Roxedus that doesn't really answer the question, is LSIO installing pip packages via apk which is causing a conflict with pip installed packages?
It just uncovered an issue that previously went unnoticed, which is custom integrations downgrading HA's dependencies. That should not have been happening, or allowed. But it was. No one noticed.
The use of venv consolidated the locations python packages are installed to into 1 location. So now we're aware of all these integrations force downgrading HA's deps because they fail when they attempt to delete one of HA's packages.
iirc installing pip from the repo is enough to trigger that issue. Python really wants everyone to use a venv with pip and they're getting more and more aggressive at pushing it.
which is custom integrations downgrading HA's dependencies.
I don't see how that can happen if the user running the container doesn't have access to modify those libraries.
Python really wants everyone to use a venv with pip and they're getting more and more aggressive at pushing it.
I get the benefits of using venv outside a container, but is this true for applications installed in a container as well? If a container is only running a Python app, IMO venvs seem pretty pointless running in a container.
@onedr0p No, pip packages gets put into the same place as OS python packages would, hence it getting marked as a externally managed environment.
This one covers it in a good way python/cpython#102134 (comment)
We agree using a venv is counterintuitive, but the pip devs doesn't.
I don't see how that can happen if the user running the container doesn't have access to modify those libraries.
Without venv:
- pip run as root will put them in the same place as the system packages, removing as necessary.
- pip run as unprivileged user will put the packages in the user's home folder somewhere. It doesn't delete the system packages, but the
PATHwill reference that user folder first, so python will use the user installed package instead of the system package, essentially overriding
With venv:
Everything gets written to the venv folder no matter which user calls pip, removing things as necessary. Permissions become a problem if user is unprivileged.
Is there any chance for this behavior to be pushed more "upstream", so that custom components fail also on the official HA docker image?
This would definitely help because, as of today, maintainers of custom components are not pushed into doing the "right thing" since the components only fail in this image.
@aptalca I do observe something like (number 2) in my hass container which is built to be rootless only.
home-assistant-0:~/deps$ python --version
Python 3.11.3
home-assistant-0:~/deps$ echo $PYTHONPATH
:/pip-packages
home-assistant-0:~/deps$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
home-assistant-0:~/deps$ pip install thing
Defaulting to user installation because normal site-packages is not writeable
I haven't tested a lot of different HACS integrations though so maybe I have not ran into the pip package collisions yet you and others are mentioning.
Permissions become a problem if user is unprivileged.
Yeah I bet this is a common issue when not building containers using best practices 😄
Is there any chance for this behavior to be pushed more "upstream", so that custom components fail also on the official HA docker image?
You are free to open an issue or discussion over there but in my experience frenck and baloob are pretty terrible people to interact with, so much to the point that I purposely do not interact with them anymore.
It's my opinion that this image should aim to maintain as much of the existing behavior of the official HA container even if it isn't "best practice". I think it's safe to assume the vast majority of users of this image also use custom components and will need a custom script to chown the directory. Otherwise it is unlikely to work out of the box for most users.
@blakeblackshear I bet having the support coming your way for people using the Frigate extension with this container isn't great. I can see that being annoying for other maintainers of hass extensions too.
The % of my users that are using this specific image is small, so I'm not concerned.
even if it isn't "best practice"
This has nothing to do with best practices. We didn't make a conscious decision to disallow specific behavior. I outlined the underlying issue above more than once. Chowning the whole folder for everyone is not an option, not with 47k files in there. On a normal system it takes up to 2 min, on a system that experiences the overlayfs docker bug, it can take 25 minutes or longer.
Otherwise it is unlikely to work out of the box for most users.
Why though? Does your integration really specifically need pytz==2022.7? Does it not work with any version newer or older? I doubt it. That package provides timezone and daylight savings changes info, which changes often. You're force downgrading all of your users to a version that has outdated info, which is a pretty big deal when you consider that this is a home automation software where a lot of the automation depends on schedules and time.
Does your integration really specifically need pytz==2022.7? Does it not work with any version newer or older?
Definitely not, but mine is just one of thousands of integrations that will run into this issue. You are going to have this conversation over and over and over.
Thanks everyone for clarifying things. It's a tough problem and I don't see how it can be efficiently fixed. On the one hand, sometimes we just need to specify a version of a python package, and just following HA's versions will create a lot more maintenance for the custom integration developers, which already generally have little time. On the other hand, I agree that it's a very, very bad thing to allow integrations to modify HA's dependencies, I'm actually surprised it didn't lead to chaos so far.
For now, it seems that the easiest and most reasonable way to fix this is indeed to use the universal package install mod. It requires maintenance on our side as users but hey, there's no perfect solution. Here is a simple example to fix the frigate and the ha-google-cloud-stt integrations. In docker-compose.yml we use the installer and install those both versioned python packages:
homeassistant:
...
environment:
...
- DOCKER_MODS=linuxserver/mods:universal-package-install
- INSTALL_PIP_PACKAGES=pytz==2022.7|google-cloud-speech==2.19.0
...
I do have a potential fix for the underlying issue. If you guys can test and report back, it would be much appreciated:
lspipepr/homeassistant:amd64-2023.6.2-pkg-c525e342-dev-d3b5c34164795b72dc4248b2a2d57503e9e8e1fa-pr-76
replace your image with the above and remove any custom scripts or mods you implemented to work around the issue.
What it does is, it attempts to emulate the previous behavior of pip installing packages in the user's home folder in runtime by creating and activating a second venv in /config/lsiopy and also including the original venv in PYTHONPATH for discovery. Since pip is using a separate folder to install, it shouldn't attempt to uninstall/remove from the original venv.
More details here: #76 (comment)
PS. It may not work right away if you're using portainer or Synology gui to update because they for some reason extract all the env vars from the running container and apply them externally to the recreated container, which breaks when we change internal env vars in the dockerfile, such as PATH and PYTHONPATH. In those cases, edit the docker config and manually remove the PATH and PYTHONPATH settings and recreate the container so they are not overridden with outdated values.
@aptalca why did you choose to put the new venv folder in /config? Couldn't it be put somewhere where it doesn't pollute the config folder?
That is where these packages lived before the rebase to alpine 3.18
That is where these packages lived before the rebase to alpine 3.18
What was the folder name?
Not sure, but it's some hidden folder, could be /config/.local/lib/python3.10/site-packages/
Thank you. And can that folder (/config/.local) be deleted now that we are migrating to this new method?
yes, you can delete that folder (as long as there is nothing else that's not python related in there). Take a backup first to be safe