Alexa and the recent Large Attribute limiting

Question

Alexa and the recent Large Attribute limiting

Jeppedy opened this issue 2 years ago · 18 comments

The problem

"Prevent overly large attributes from being stored in the database #87105"

Amazon Alexa seems to have attributes that exceed the limit.

Logger: homeassistant.components.recorder.db_schema
Source: components/recorder/db_schema.py:491
Integration: Recorder (documentation, issues)
First occurred: 4:40:11 PM (2 occurrences)
Last logged: 4:40:13 PM

State attributes for sensor.bedroom_dot_next_alarm exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored

Looks to be due to a large number of alarms set. While not a bug, the original developer said he'd never seen a valid situation with an attribute payload over 2k.

What version of Home Assistant Core has the issue?

2023.3.0

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Recorder

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

Answer 1 · 2023-03-06T19:05:50.000Z

Want to add to this, I have the same issue but with phone notifications. Some come up with the same error of

State attributes for sensor.oneplus_a6003_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored

While I don't """need""" to save those, at least truncating them or something would make more sense than just dropping them entirely.

Answer 2 · 2023-03-07T08:32:53.000Z

Same here

Logger: homeassistant.components.recorder.db_schema Source: components/recorder/db_schema.py:491 Integration: Recorder (documentation, issues) First occurred: March 6, 2023 at 08:33:16 (4 occurrences) Last logged: 05:00:59
State attributes for sensor.backup_state exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored

Answer 3 · 2023-03-07T15:32:09.000Z

Guess I'll add one from my recent log sweep as well... local weather station within the National Weather Service integration. Yes there's lots of data included, data I want to be able to use and see. Unless I am misunderstanding what is being cut off.

**Edit- the more I look at this, my data isn't truncated until well past the point I care to display in HA. Maybe a moot issue, still the errors are annoying.

Logger: homeassistant.components.recorder.db_schema
Source: components/recorder/db_schema.py:491
Integration: Recorder (documentation, issues)
First occurred: 9:26:53 AM (1 occurrences)
Last logged: 9:26:53 AM

State attributes for weather.kevv_hourly exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored

Answer 4 · 2023-03-07T19:11:42.000Z

Same issue here but with serveral entities:

`Logger: homeassistant.components.recorder.db_schema
Source: components/recorder/db_schema.py:491
Integration: Recorder (documentation, issues)
First occurred: 6. März 2023 um 20:47:55 (5274 occurrences)
Last logged: 19:57:17

State attributes for weather.hourly exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
State attributes for sensor.461748 exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
State attributes for sensor.fr24_aircraft_2 exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored`

Answer 5 · 2023-03-08T15:57:05.000Z

same here:

2023-03-08 16:53:49.721 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.tv_heute_20_uhr exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
2023-03-08 16:53:49.878 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.tv_heute_22_uhr exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored

Answer 6 · 2023-03-08T23:53:54.000Z

I noticed my Raspberry had become exceedingly slow (load around 3, TCP IO'd out), and rabbit holed into this issue. Here's what happened:

I have a Grocy instance running where I keep track of products, stock and a shopping list. There's a new integration Shopping List with Grocy on HACS (thanks Anrolosia!) which can among other things display the entire product list from Grocy. I've imported about 550 products from Picnic (an online grocery store) in Grocy, along with their images. The integration fetches all data from the Grocy API, which requires authentication even for the images. So, instead of exposing the API token to the frontend, it fetches the image data from the API in the service, and sends it along with the other product attributes as a MQTT message - one message per product, with the image bytes included. A bit of an awkward solution perhaps, but at least it's secure.

The new limitation caused the integration to fail its setup. Either itself or HomeAssistant keeps retrying, creating a massive load on the MQTT server, sending and receiving these image bytes over and over for all 550+ products because they won't get processed and accepted.

So, that's the story of how a seemingly innocent assumption of what a maximum payload should be can lead to really weird issues down the line. The entire Raspberry was slowed down to a crawl! I'd also say that in this case the decision was made too lightly, as the new limit is a huge change from the preceding database limit, which in that sense predictably breaks quite a few real-world cases.

I personally would like to see the new limitation:

Rolled back ASAP due to unforeseen consequences
Rolled out again in phases:
1. Not enforced yet, but with warnings "attribute will be stored but may impact performance, see doc X", and official and documented configuration options to a) enable enforcement b) set the limit size c) disable warnings
2. Adapted to community feedback on e.g. what the default limit size should be
3. Rolled out with warnings changed to "attribute will dropped from version X onwards, unless config option Y is set to Z" - where version X lies months in the future
4. Rolled out with enforcement by default, still honoring custom configuration

This way anyone that would be affected has some time to adapt - and it also saves some Raspberry CPU time.

Answer 7 · 2023-03-11T17:01:28.000Z

I have a similar problem, and it doesn't just apply to Alexa integration.

homeassistant 2023-03-11 16:53:06.728 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.samsung_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored homeassistant 2023-03-11 16:53:11.541 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for camera.xiaomi_cloud_map_extractor exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored homeassistant 2023-03-11 16:53:27.418 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.samsung_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored homeassistant 2023-03-11 16:54:34.975 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.samsung_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored homeassistant 2023-03-11 16:56:38.397 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for camera.xiaomi_cloud_map_extractor exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored homeassistant 2023-03-11 16:58:11.546 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for camera.xiaomi_cloud_map_extractor exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored

Answer 8 · 2023-03-12T20:41:05.000Z

Same here, for Sensor REST request.

Answer 9 · 2023-03-23T10:47:27.000Z

Same here with different integration, I get more than 1000 warnings per day

v 2023.3.5

Logger: homeassistant.components.recorder.db_schema
Source: components/recorder/db_schema.py:491
Integration: Recorder ([documentation](https://www.home-assistant.io/integrations/recorder), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+recorder%22))
First occurred: March 22, 2023 at 23:52:07 (1489 occurrences)
Last logged: 12:36:49
State attributes for sensor.watchman_missing_entities exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored

Answer 10 · 2023-03-27T02:04:37.000Z

I'm getting all the errors below, When this happens, I'm no longer able to tell Alexa to control my Home Assistant devices; requiring me to reboot my entire Home Assistant VM.
What's the solution to fix this without having to reboot?

EDIT: I ended up turning off this sensor, since it wasn't very useful to me anyway. However, I want to make sure there isn't another issue that's causing this.

2023-03-26 03:34:10.707 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.sm_n960u_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
2023-03-26 03:42:38.276 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.sm_n960u_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
2023-03-26 03:49:20.714 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.sm_n960u_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
2023-03-26 03:52:36.777 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.sm_n960u_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
2023-03-26 04:02:38.641 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.sm_n960u_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
2023-03-26 04:04:22.838 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.sm_n960u_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
2023-03-26 04:11:36.893 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.sm_n960u_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
2023-03-26 04:12:39.413 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.sm_n960u_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
2023-03-26 04:19:25.334 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.sm_n960u_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
2023-03-26 04:22:37.529 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.sm_n960u_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored
2023-03-26 04:24:07.056 WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.sm_n960u_active_notification_count exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored

Answer 11 · 2023-04-15T18:52:48.000Z

With regard to the phone active notification counts issue:

From past experience my understanding is that the the homeassistant development team and the phone companion app development teams are separate. Nevertheless, I would have thought that the teams could work together to prevent the conflicts between the applications.

I have seen the companion app development team push back on occasions with the statement that their application simply pass on (dumps) the data provided by the phone verbatim.

Personally I would like to see just the notification title and text in the attributes for this sensor...

Writing a parser that (per platform) extracts just the pertinent data is not simple, I have attempted to do so using a template sensor with limited success as the attribute names involved for all of the various notifications are (to my perception) inconsistent.

The fact that the attributes for notifications can exceed the storable volume permitted by recorder suggests to me that the attributes for notifications could, or should be omitted by the companion app development team.

The only reliable method I have found to retain recording of the notification count but get rid of the error messages related to the length of the attribute data is to implement a template sensor for the state only


template:
- trigger:
  - id: to_valid
    for:
      seconds: 10
    platform: state
    not_to:
    - unknown
    - unavailable
    entity_id:
    - sensor.phone_active_notification_count
  sensor:
  - unique_id: phone_active_notification_count_only
    name: phone_active_notification_count_only
    state: >
      {%- set s = states('sensor.phone_active_notification_count') -%}
      {%- if( s is none ) or not is_number(s) -%}
        {%- set s = states("sensor.phone_active_notification_count_only") -%}
      {%- endif %}
      {{s | int(0)}}
    unit_of_measurement: 'total'
    state_class: measurement

Answer 12 · 2023-06-08T14:33:48.000Z

Have the same log message for ZONT controller sensors requested by REST request. Payload is more then 3K bytes.

Answer 13 · 2023-06-26T20:15:42.000Z

This is an "I'm also having this issue" comment.

Answer 14 · 2023-06-26T23:34:30.000Z

Just a note, my issue was just a 1-time thing. It only happened once. I'm using a Nortek Zigbee USB HUSBZB-1 combo controller on Win11.

Answer 15 · 2023-07-15T07:16:56.000Z

Another with the same problem :(

Answer 16 · 2023-07-17T00:20:45.000Z

Hey there @home-assistant/cloud, @ochlocracy, @jbouwh, mind taking a look at this issue as it has been labeled with an integration (alexa) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of alexa can trigger bot actions by commenting:

@home-assistant close Closes the issue.
@home-assistant rename Awesome new title Renames the issue.
@home-assistant reopen Reopen the issue.
@home-assistant unassign alexa Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

alexa documentation
alexa source
_{^{(message by IssueLinks)}}

Answer 17 · 2023-07-17T07:44:43.000Z

This happens especially when extra state attributes be come to large. It is not a best practice to use extra state attributes that change a lot or have a large size. I see alexa was linked, but I do not believe this is something specific to alexa,
Alexa it self does not supply entities, but it might update them if there is a state change. So the the cause for these warnings should be search for in the integrations that supply the entities.
So please open issues for these integrations that produce entities that have these issues instead. (except for custom integrations, as they are officially not supported by Home Assistant).

Answer 18 · 2023-08-05T20:10:49.000Z

I am closing this issue since there this does not seems an issue with Alexa. It seems that the entities in trouble have a state size that is too big to process.