Increased CPU load on HA server with 1.4.0
Closed this issue · 8 comments
After installing 1.4.0 observed a significant CPU load (up to 95%) on HA server.
It happens when I open a test view containing ~10 Entities cards with ~20 template-entity-row rows.
Templates are rather simple like
active: >-
{{ is_state('input_boolean.test_boolean','on') }}
or
secondary: >-
{{relative_time(states[config.entity].last_changed)}} ago
or
state: >
_(component.device_tracker.entity_component._.state.not_home)
or
state: |
_(ui.errors.config.key_missing, key, {{states("sun.sun")}})
Have to stop HA in docker to "calm down" CPU.
After opening the same view - same high CPU load.
Oh! I see what this is. I'll try to fix it.
Pretty sure it's actually the same problem as #120
might be odd, but having the #120 issue happening on many, but not all, I can not confirm cpu load to have increased here at all.
Maybe it is the same thing: on many templates but not all?
any logs in your system indicating trouble that could lead to cpu increase?
@thomasloven
Thanks for fixes, have not tested yet.
The test view which caused the problem contained this code:
- type: custom:template-entity-row
entity: person.ildar
name: localized (failed)
state: '{{_(states("person.ildar"))}}'
- type: custom:template-entity-row
entity: person.ildar
name: localized (failed)
state: '{{_(states.person.ildar.state)}}'
These line were for testing "_(...)" functions.
I know how to use these functions - these lines were only to remind about a wrong use.
Starting from 1.4.0 this caused errors like this:
2024-02-12 07:00:46.277 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: '_' is undefined when rendering '{{_(states("person.ildar"))}}'
2024-02-12 07:00:46.278 ERROR (MainThread) [homeassistant.helpers.event] Error while processing template: Template<template=({{_(states("person.ildar"))}}) renders=2>
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 572, in async_render
render_result = _render_with_context(self.template, compiled, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 2332, in _render_with_context
return template.render(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/usr/local/lib/python3.12/site-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "<template>", line 1, in top-level template code
File "/usr/local/lib/python3.12/site-packages/jinja2/sandbox.py", line 391, in call
if not __self.is_safe_callable(__obj):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/custom_templates/__init__.py", line 292, in is_safe_callable
or self.ct_original_is_safe_callable(obj))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 2690, in is_safe_callable
return isinstance(obj, AllStates) or super().is_safe_callable(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/jinja2/sandbox.py", line 275, in is_safe_callable
getattr(obj, "unsafe_callable", False) or getattr(obj, "alters_data", False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 2367, in _fail_with_undefined_error
raise ex
File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 2364, in _fail_with_undefined_error
return super()._fail_with_undefined_error(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: '_' is undefined
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 698, in async_render_to_info
render_info._result = self.async_render(
^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 574, in async_render
raise TemplateError(err) from err
homeassistant.exceptions.TemplateError: UndefinedError: '_' is undefined
And these errors flooded the log very fast (lots of errors within a short time) causing a constant high CPU load (and increased writings to DB, increased writing to HDD).
I commented these lines now))).
So, in some extent I myself caused this. From other side, HA seems to be "too dependent" on a wrong template.
Interesting about the log issue... I'll give that a think-over.
(This is different than the other one though. That one was in the editor, and I may be able to do something about this one.)
This glitch caused (as I said) huge CPU load -> DB size jumped from ~1GB to ~8GB (some unknown integration seems to flood DB when CPU is huge & python started working in a wrong way), now sorting this out))))
is there any chance the Ip Bans we sometimes experience could have been related to this issue here?
Ive just found out about https://github.com/custom-cards/secondaryinfo-entity-row?tab=readme-ov-file#old-way---custom-template-language on secondary-info, stating those templates can only be seen by Admin users.
open issue there: custom-cards/secondaryinfo-entity-row#56
I can't imagine the frontend to be completely banned because of that.... but, since I noticed those bans be registered at the same time the template-entity-row was updated, we'd better be 100% sure it cant be related.
what's more, the bans were not only for non-admins, but my own app, and of my fellow admin were also logged out and prohibited
I asked in #ios_and_mac on discord here before I read the post on secondaryinfo-entity-row
If so, then the stock markdown card would have the same issue.
I'll look into it, though.