Load average model suddenly stopped working
Opened this issue · 13 comments
Version of the custom_component
v1.8.0
Home Assistant: 2023.8.4
Describe the bug
Since rebooting Home Assistant yesterday, none of the house load average entities are reporting data. I've tried rebooting Home Assistant again, reloading the integration and going through the setup wizard again. The Solcast forecast is still working.
Debug log
I see the following error in the logs every 30 minutes.
First occurred: 2 September 2023 at 17:46:03 (19 occurrences)
Last logged: 11:30:01
Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/config/custom_components/foxess_em/average/average_controller.py", line 70, in async_refresh
await self._model.refresh()
File "/config/custom_components/foxess_em/average/average_model.py", line 44, in refresh
await self._update_history(self._tracked_sensors[sensor])
File "/config/custom_components/foxess_em/average/average_model.py", line 55, in _update_history
await self._update_item(item)
File "/config/custom_components/foxess_em/average/average_model.py", line 93, in _update_item
values_dict[0] = {
~~~~~~~~~~~^^^
IndexError: list assignment index out of range
@andrewmooreio interesting! Can you add a screenshot of the last 48h of load data please?
Here you go. Sorry, on mobile so not particularly granular. If you need a more detailed graph let me know.
Hi @nathanmarlor. Do you need any further info from me? Issue is still present with that same recurring error.
@andrewmooreio apologies, been busy with work. The fact that it's choking on values_dict[0] (i.e. accessing the first element a list of history values) tells me that the recorder isn't functioning properly. Can you double check that EM is setup to use the correct load power entity? Other than that I can only suggest querying the SQLite database directly and looking for any anomalies (EM already filters empty, unknown and unavailable entries). FYI this could have happened a while ago, and you've only just noticed following a reboot - be interested to know when your peak/total total load values last changed?
Thanks @nathanmarlor. I did briefly detach my BMS from the inverter on Saturday for some maintenance. The inverter was up and running the whole time, but obviously it restarts and goes through all the internal checks every time you disconnect or reconnect the BMS. Looking at the time period of the maintenance, there aren't any gaps in the recorder data.
The last update on the load predictions was 18:00-ish on Saturday, which is about the same time I finished my maintenance… which can't be a coincidence.
Perhaps at some point in this process, some data got corrupt. As you've said though, EM should filter out any anomalies. Also, doesn't EM only look back 48 hours at load data? So if there was something funky happening on Saturday, it should be outside of the current data set?
You're right, it should only be looking back at 48h. Effectively something in your data is making this code block return an empty list:
history_list = await recorder.async_add_executor_job(
history.state_changes_during_period,
self._hass,
from_date,
to_date,
str(item.sensor_name),
)
values = history_list.get(item.sensor_name)
values_dict = [
{
"datetime": value.last_changed.astimezone(tz.UTC),
"value": float(value.state),
}
for value in values
if value.state not in ("", "unknown", "unavailable")
]
# Add start/final value to ensure even resampling later
values_dict[0] = {
"datetime": from_date.astimezone(tz.UTC),
"value": 0,
}
``
Is there an easy way to “clear out” EM, i.e. any cached data it has stored?
EM doesn't store any data relating to load, it retrieves it directly from HA every 30 minutes to produce peak/total averages.
Have you been able to query the SQLite DB?
I more meant the model that EM builds, is that stored persistently?
Not had a chance to browse the database yet, I'll try first thing tomorrow.
I'm not familiar with interacting with databases, but I've given it a go and what I'm seeing returned seems odd.
I'd expect the following query to return the top 10 entity_ids by how many times they occur, yet it just returns any empty entity_id with every row counted:
sqlite> SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 10;
entity_id|count
|6784394
Similarly, if I just select 10 rows, the entity_id column is empty:
sqlite> SELECT * FROM states LIMIT 10;
state_id|entity_id|state|attributes|event_id|last_changed|last_changed_ts|last_updated|last_updated_ts|old_state_id|attributes_id|context_id|context_user_id|context_parent_id|origin_idx|context_id_bin|context_user_id_bin|context_parent_id_bin|metadata_id
3327857||0.346||||||1693105920.61329||492||||0|�4�eb���" �q�|||492
3327858||242.1||||||1693105921.72473||446||||0|�4�����/ʀ�|||446
3327859||240.4||||||1693105922.0797||519||||0|�4� Y����Z�=�|||519
3327860||0.385||||||1693105922.09238||190||||0|�4� ,#����4�*T|||190
3327861||-0.385||||||1693105922.09256||188||||0|�4� ,N�.�vg���^|||188
3327862||241.8||||||1693105922.0927||184||||0|�4� ,O�����FS|||184
3327863||50.06||||||1693105922.09282||195||||0|�4� ,t�%۪�*��|||195
3327864||23.1||||||1693105922.09301||206||||0|�4� -Y-L �}|||206
3327865||0.365||||||1693105922.09317||183||||0|�4� -�� �2�2�|||183
3327866||3332||||||1693105922.09329||208||||0|�4� -��:�B�Z���|||208
Perhaps this is normal, some pointers would be appreciated. Happy to run any queries you send over.
I rebooted HA core yesterday at around 16:00 to update foxess_modbus and now EM has sprung into life ¯_(ツ)_/¯
Strange.
Just rebooted HA again for another integration update, EM has stopped working with the same error posted above.
Will reinstall EM and see if that fixes it.
Just an update on this @nathanmarlor. I think the issue was due to a device I had set up in aux power (devices to exclude). I had my air conditioner plug added in here, since it's been cooler I have had the air conditioner unplugged, so the device was unavailable.
I've removed the air conditioner plug from aux power devices and EM is functioning as expected.