migration broken since last release

Question

migration broken since last release

Closed this issue 8 months ago · 20 comments

ChrisOboe commented 9 months ago

Checklist

I have updated the integration to the latest version available
I have checked if the problem is already reported

The problem

Since updating to the lastest version homeassistant shows a migration error.

here is the relevant log:

Traceback (most recent call last):
  File "/nix/store/7z97x7nvmr0kpssp9xa3j058x4nmgx3f-homeassistant-2024.3.3/lib/python3.12/site-packages/homeassistant/config_entries.py", line 770, in async_migrate
    result = await component.async_migrate_entry(hass, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/hass/custom_components/valetudo_vacuum_camera/__init__.py", line 211, in async_migrate_entry
    await move_data_to_valetudo_camera()
  File "/var/lib/hass/custom_components/valetudo_vacuum_camera/__init__.py", line 305, in move_data_to_valetudo_camera
    os.makedirs(valetudo_camera_folder)
  File "<frozen os>", line 215, in makedirs
  File "<frozen os>", line 225, in makedirs
OSError: [Errno 30] Read-only file system: '/.storage'

it seems that it tries to write to /.storage (which doesn't exist)
my homeassistant storage dir lives in /var/lib/hass/.storage
all the other plugins and homeassistant itself uses the correct storage dir.

What version of an integration has described problem?

2024.04.1

What was the last working version of an integration?

No response

What vacuum model do you have problems with?

two dreames

Please firmware installed on your Vacuum.

valtudo

What type of platform you use?

Intel NUC (or generic x86_64)

What version of Home Assistant do you use?

2024.4.0

What type of installation are you running?

Home Assistant Core

Camera's settings.

No response

Logs or Errors shown in the HA snapshots (please enable the Debug Mode)

No response

Additional information

No response

Answer 1 · 2024-04-07T18:47:43.000Z

i just checked on my system hass is executed with / as PWD.
as a workarround i can modify my hass systemd service to change the pwd before starting hass.

Answer 2 · 2024-04-07T20:47:11.000Z

@ChrisOboe thanks a lot for this.. I will check why it didn't get the path automatically as it should.
On the contained version of HA I had no problem.
The previous version do not use the valetudo_camera folder in .storage.

Probably the issue is that it can't create the folder without psw.

Answer 3 · 2024-04-08T04:34:52.000Z

@ChrisOboe does the camera work after you created the folder with the proper password? If not would work on a patch for it.
https://github.com/sca075/valetudo_vacuum_camera/blob/20c46543f24840b98b3aa82098e8761755d733a9/custom_components/valetudo_vacuum_camera/__init__.py#L299-L306

as you can see I'm looking for the current working dir of home assistant and honestly I do not catch how this failed to create the folder on your system.

Answer 4 · 2024-04-08T20:36:12.000Z

@ChrisOboe I hope you could figure out a work around and once again thanks for highlight this issue.
when os.getcwd didn't actually got that your instance was working in /var/lib/hass/ is your config folder, I will investigate on what is the best method to avoid this in the future. Apologies for the inconvenience.

Answer 5 · 2024-04-09T04:26:04.000Z

same error. let me know if you need any logs/info from my system. I have HAAS running in a docker container. Can you also please provide a workaround ?

Answer 6 · 2024-04-09T04:39:39.000Z

same error. let me know if you need any logs/info from my system. I have HAAS running in a docker container. Can you also please provide a workaround ?

yes please can you provide the logs? I will try to fix this asap.

Answer 7 · 2024-04-09T04:45:33.000Z

not just migration, new install is broken too. I removed old integration and tried to reinstall from scratch and got error.

2024-04-09T04:43:09.802580123Z 2024-04-08 21:43:09.786 ERROR (MainThread) [aiohttp.server] Error handling request
2024-04-09T04:43:09.802772587Z Traceback (most recent call last):
2024-04-09T04:43:09.802814525Z   File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
2024-04-09T04:43:09.802853088Z     resp = await request_handler(request)
2024-04-09T04:43:09.802890076Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.803181241Z   File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
2024-04-09T04:43:09.803245104Z     resp = await handler(request)
2024-04-09T04:43:09.803281467Z            ^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.803316917Z   File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 114, in impl
2024-04-09T04:43:09.803354930Z     return await handler(request)
2024-04-09T04:43:09.803389930Z            ^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.803426856Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware
2024-04-09T04:43:09.803464919Z     return await handler(request)
2024-04-09T04:43:09.803500844Z            ^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.803536382Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/forwarded.py", line 83, in forwarded_middleware
2024-04-09T04:43:09.803573095Z     return await handler(request)
2024-04-09T04:43:09.803681058Z            ^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.803720608Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/request_context.py", line 26, in request_context_middleware
2024-04-09T04:43:09.803757846Z     return await handler(request)
2024-04-09T04:43:09.803793134Z            ^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.803890898Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/ban.py", line 88, in ban_middleware
2024-04-09T04:43:09.804034924Z     return await handler(request)
2024-04-09T04:43:09.804071112Z            ^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.804106687Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/auth.py", line 236, in auth_middleware
2024-04-09T04:43:09.804143237Z     return await handler(request)
2024-04-09T04:43:09.804178488Z            ^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.804231463Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/headers.py", line 32, in headers_middleware
2024-04-09T04:43:09.804268751Z     response = await handler(request)
2024-04-09T04:43:09.804304139Z                ^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.804339964Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/helpers/http.py", line 73, in handle
2024-04-09T04:43:09.804387002Z     result = await handler(request, **request.match_info)
2024-04-09T04:43:09.804424365Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.804468390Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/decorators.py", line 71, in with_admin
2024-04-09T04:43:09.804516303Z     return await func(self, request, *args, **kwargs)
2024-04-09T04:43:09.804616154Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.804676230Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/config/config_entries.py", line 222, in post
2024-04-09T04:43:09.804713468Z     return await super().post(request, flow_id)
2024-04-09T04:43:09.804759268Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.804796544Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/data_validator.py", line 73, in wrapper
2024-04-09T04:43:09.804833444Z     result = await method(view, request, data, *args, **kwargs)
2024-04-09T04:43:09.804882282Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.804919182Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/helpers/data_entry_flow.py", line 122, in post
2024-04-09T04:43:09.804955733Z     result = await self._flow_mgr.async_configure(flow_id, data)
2024-04-09T04:43:09.805005658Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.805043058Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/data_entry_flow.py", line 356, in async_configure
2024-04-09T04:43:09.805079709Z     result = await self._async_configure(flow_id, user_input)
2024-04-09T04:43:09.805128372Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.805164785Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/data_entry_flow.py", line 402, in _async_configure
2024-04-09T04:43:09.805259973Z     result = await self._async_handle_step(
2024-04-09T04:43:09.805298198Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.805333799Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/data_entry_flow.py", line 506, in _async_handle_step
2024-04-09T04:43:09.805380662Z     result: _FlowResultT = await getattr(flow, method)(user_input)
2024-04-09T04:43:09.805418037Z                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T04:43:09.805454787Z   File "/config/custom_components/valetudo_vacuum_camera/config_flow.py", line 224, in async_step_user
2024-04-09T04:43:09.805505838Z     os.makedirs(storage_path)
2024-04-09T04:43:09.805542426Z   File "<frozen os>", line 215, in makedirs
2024-04-09T04:43:09.805579188Z   File "<frozen os>", line 225, in makedirs
2024-04-09T04:43:09.805627251Z PermissionError: [Errno 13] Permission denied: '/run/s6-rc:s6-rc-init:OlgGKB/servicedirs/svc-homeassistant/.storage'

Answer 8 · 2024-04-09T05:51:57.000Z

I figure out that the issue might be caused from os.path.join.

 storage_folder = os.path.join(os.getcwd(), STORAGE_DIR) 
 valetudo_camera_folder = os.path.join(storage_folder, "valetudo_camera")

this lines 300 and 301 in init.py need to be replaced with:

 storage_folder = f"{os.getcwd()}/{STORAGE_DIR}"
 valetudo_camera_folder = f"{storage_folder}/valetudo_camera"

can you give it a try and let me know if it works on yours @carefulcomputer and @ChrisOboe configs?
alternatively you can download the main where I did modify those lines just few minutes ago #1a3e387

Answer 9 · 2024-04-09T06:00:49.000Z

i edited the file, and restarted haas. still getting the error when trying to set the vaccum name in integrations.
do i need to clear some cache or perform any other operations etc ?

2024-04-09T05:58:52.761256281Z     return await func(self, request, *args, **kwargs)
2024-04-09T05:58:52.761342494Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T05:58:52.761427320Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/config/config_entries.py", line 222, in post
2024-04-09T05:58:52.761515496Z     return await super().post(request, flow_id)
2024-04-09T05:58:52.761600171Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T05:58:52.761686322Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/data_validator.py", line 73, in wrapper
2024-04-09T05:58:52.761775573Z     result = await method(view, request, data, *args, **kwargs)
2024-04-09T05:58:52.761866861Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T05:58:52.762006288Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/helpers/data_entry_flow.py", line 122, in post
2024-04-09T05:58:52.762105814Z     result = await self._flow_mgr.async_configure(flow_id, data)
2024-04-09T05:58:52.762192089Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T05:58:52.762280153Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/data_entry_flow.py", line 356, in async_configure
2024-04-09T05:58:52.762368679Z     result = await self._async_configure(flow_id, user_input)
2024-04-09T05:58:52.762461942Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T05:58:52.762541418Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/data_entry_flow.py", line 402, in _async_configure
2024-04-09T05:58:52.762618306Z     result = await self._async_handle_step(
2024-04-09T05:58:52.762691119Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T05:58:52.762764145Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/data_entry_flow.py", line 506, in _async_handle_step
2024-04-09T05:58:52.762839145Z     result: _FlowResultT = await getattr(flow, method)(user_input)
2024-04-09T05:58:52.762913159Z                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-09T05:58:52.763030610Z   File "/config/custom_components/valetudo_vacuum_camera/config_flow.py", line 224, in async_step_user
2024-04-09T05:58:52.763119748Z     os.makedirs(storage_path)
2024-04-09T05:58:52.763195236Z   File "<frozen os>", line 215, in makedirs
2024-04-09T05:58:52.763270025Z   File "<frozen os>", line 225, in makedirs
2024-04-09T05:58:52.763344888Z PermissionError: [Errno 13] Permission denied: '/run/s6-rc:s6-rc-init:OlgGKB/servicedirs/svc-homeassistant/.storage'

Answer 10 · 2024-04-09T06:05:38.000Z

@carefulcomputer this is while editing options?? because it do not looks the startup sequence. Can you please download and instal the integration from the main repository, enable the logs for the camera and restart home assistant and provide me the startup sequence please?

edit: hold on, please revert to the 2024.04.0 I think I got to review the way how to get the working directory, some issue with python 12. with os.getcwd probably.

Answer 11 · 2024-04-09T06:14:18.000Z

thanks for looking into it! it is getting late for me. i will try tomorrow.

Answer 12 · 2024-04-10T04:54:55.000Z

any luck ?

Answer 13 · 2024-04-10T04:57:50.000Z

I'm working on it, in few minutes I will update the main repository.

Answer 14 · 2024-04-10T05:15:15.000Z

@carefulcomputer can you please download the main distribution, I added some try to handle the errors and as well some additional logging and a counter measure in case of failure. It should work now if all goes as planned.
Indeed I would need your kind feedback and I do really thanks you in advance for the cooperation here.

Answer 15 · 2024-04-10T13:56:34.000Z

thank you for new update. must be something really weird. still getting error.

2024-04-10T13:55:21.773107117Z            ^^^^^^^^^^^^^^^^^^^^^^
2024-04-10T13:55:21.773164530Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/headers.py", line 32, in headers_middleware
2024-04-10T13:55:21.773225318Z     response = await handler(request)
2024-04-10T13:55:21.773279556Z                ^^^^^^^^^^^^^^^^^^^^^^
2024-04-10T13:55:21.773317682Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/helpers/http.py", line 73, in handle
2024-04-10T13:55:21.773356532Z     result = await handler(request, **request.match_info)
2024-04-10T13:55:21.773395757Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10T13:55:21.773477083Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/decorators.py", line 71, in with_admin
2024-04-10T13:55:21.773544859Z     return await func(self, request, *args, **kwargs)
2024-04-10T13:55:21.773602072Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10T13:55:21.773659772Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/config/config_entries.py", line 222, in post
2024-04-10T13:55:21.773718098Z     return await super().post(request, flow_id)
2024-04-10T13:55:21.773775161Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10T13:55:21.773832124Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/data_validator.py", line 73, in wrapper
2024-04-10T13:55:21.773891937Z     result = await method(view, request, data, *args, **kwargs)
2024-04-10T13:55:21.773950125Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10T13:55:21.774009113Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/helpers/data_entry_flow.py", line 122, in post
2024-04-10T13:55:21.774068563Z     result = await self._flow_mgr.async_configure(flow_id, data)
2024-04-10T13:55:21.774125901Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10T13:55:21.774182627Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/data_entry_flow.py", line 356, in async_configure
2024-04-10T13:55:21.774243665Z     result = await self._async_configure(flow_id, user_input)
2024-04-10T13:55:21.774300478Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10T13:55:21.774358241Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/data_entry_flow.py", line 402, in _async_configure
2024-04-10T13:55:21.774418079Z     result = await self._async_handle_step(
2024-04-10T13:55:21.774475692Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10T13:55:21.774533205Z   File "/usr/local/lib/python3.12/site-packages/homeassistant/data_entry_flow.py", line 506, in _async_handle_step
2024-04-10T13:55:21.774593293Z     result: _FlowResultT = await getattr(flow, method)(user_input)
2024-04-10T13:55:21.774652131Z                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10T13:55:21.774710919Z   File "/config/custom_components/valetudo_vacuum_camera/config_flow.py", line 227, in async_step_user
2024-04-10T13:55:21.774771257Z     os.mkdir(f"{storage_path}/valetudo_camera")
2024-04-10T13:55:21.774830833Z FileNotFoundError: [Errno 2] No such file or directory: '/run/s6-rc:s6-rc-init:OlgGKB/servicedirs/svc-homeassistant/.storage/valetudo_camera'

Answer 16 · 2024-04-10T18:37:29.000Z

Okay at this point please revert to the previous version and wait for next one. The camera will be released next week and I will ask to the core guys as per it looks there's some issues with the basic os function. It is anyhow strange that on 3 instances "supervised" I use on my tests I can't reproduce this issue.

Answer 17 · 2024-04-11T00:07:54.000Z

can you please try on a container ? i have been using docker container and that has this problem.

Answer 18 · 2024-04-11T06:24:33.000Z

@carefulcomputer my production environment is in a docker container (supervised) on a server running Debian. I normally test also on this the camera, HA 2024.04.02. No problem on it.

You can use the SSH add on home assistant you can go on cd /config/.storage and there .. as you can see the camera already stored in the past the png with your maps, probably a raw file too if not a zip when you had the logs enabled, this is the background reason to create a folder over there.

In the PI4 running the OS I removed both cameras (integration therefore removed from home assistant) removed the folder and then reinstall the cameras. The folder was not created but the integration did setup both vacuums with no error (as it should be). Having although some strange snapshot issue with the rand256 vacuum currently after that but the other one running the original valetudo works as expected (no folder then store the data in .storage only).

I do not catch fully why those error happens and also why the mkdir or makedirs of OS isn't working..
https://github.com/sca075/valetudo_vacuum_camera/blob/c07205c2a300aef93073981e51afd80026208125/custom_components/valetudo_vacuum_camera/config_flow.py#L221-L235

The if should avoid if the directory .storage "do not exist" to create the new folder over there (you can comment those lines and the initial config process must end without error in your case). I need to check again the python 12 docs probably something wrong on this.. this requires time.. so if you want please go back to the know working version in your case or.. simply create the folder in .storage so that this works also for you.

Answer 19 · 2024-04-12T05:37:22.000Z

is .storage folder supposed to map to a physical folder in home assistant ?

Here is what I did..

i shelled into haas docker image.
went to /run/s6-rc:s6-rc-init:gEioCK/servicedirs/svc-homeassistant folder
there was no .storage folder. so created .storage/valetudo_camera
also create a test file to see if it would show up somewhere in mapped docker volume on my machine
exited out of docker shell.
searched for test file in haas docker volume, but couldn't find it.

However the integration started working.

so it may not be the issue with folder creation, but rather mapping ?
if this integration is supposed to use haas's mounted /config volume, why is it trying to use /run/s6-rc:s6-rc-init:gEioCK/servicedirs/svc-homeassistant/.storage ? should it not use /config/.storage ?

Answer 20 · 2024-04-12T06:13:41.000Z

2. went to /run/s6-rc:s6-rc-init:gEioCK/servicedirs/svc-homeassistant/ folder

@carefulcomputer thanks for the update and your time on this.

Interesting.. typically for HA contained is the root /config/.storage and core is /var/lib/hass/.storage in the file explorer you will not see it as it is a reserved folder for HA (os .folder is hidden). Both of you were running the camera so far I understand and there was no issue os.getcwd() function that is used to determinate the working dir of the HA instance (python wise) .. anyhow this is quite an interesting point too #115.. please have a look at it.

Happy to know that both of you are up and running.
@ChrisOboe is too so far I understand you had permission issues and you gave it.. as work around.. and it worked.
@carefulcomputer strange path.. I do not know if this.. correct but if you are in SSH (using the add on) just type cd /config/.storage and ls ..

I can add just some additional logging anyhow from coding point of view I do not see any "bug".. on the several tests I made as in #115 I noticed too some environmental changes during my test..