Home Assistant and SSD TBW.
pvvx opened this issue · 29 comments
recorder in configuration.yaml:
recorder:
purge_keep_days: 31
commit_interval: 60
Operating system on MMC. The main record on the MMC disk comes from Bluez to '/var/lib/bluetooth/'.
Home assistant database on SSD-NVME. Only write operations to 'home-assistant_v2.db-wal' and 'home-assistant_v2.db'.
All other write operations on ramdisk.
cat /sys/block/mmcblk1/stat | awk '{printf "%.3f\n", $7*512/1048576}'
cat /sys/block/nvme0n1/stat | awk '{printf "%.3f\n", $7*512/1048576}'
smartctl -A -j /dev/nvme0 | awk '/"data_units_written"/ {printf "%0.3f\n", $2*512/1000000}'
View all active disk writes in the system:
fatrace -t -f W
On the last 256 GB SATA solid-state drive, Home Assistant worked for 2 years before the TBW of the disk was exhausted. In the last few months, 'Available_Reservd_Space' has been exhausted on the disk ('Reallocated_Sector_Count' has reached its maximum). Uninterruptible power supply was used.
Sorry if the question seems dumb: this only affects the SSD if the BT receiver is connected directly to the machine running HA, am I correct?
Also in the Warnings I see reported that:
_Warnings
Requires BT adapter with USB2.0 HS. Otherwise, there will be many gaps in the reception. Do not use ESP32xxx - it cannot work with BLE at normal speed and drains the batteries of all BLE devices when connected! ESPHome does not work with Bluetooth 5.0 and misses a lot of advertising packets from BLE devices._
But I see many in the forums using ESPHome to connect (that's my config: an ESPHome mainly dedicated to BT devices (3 of them actually) + BTHome protocolo + Home Assistant). Is this safe or this will drain the BLE battery (still don't understand how since the ESP should be in "passive" mode and only get data when the device sends it out)?
ESP should be in "passive" mode
misses a lot of advertising packets from BLE devices
Sorry if the question seems dumb: this only affects the SSD if the BT receiver is connected directly to the machine running HA, am I correct?
When using Bluez. This is not the main problem.
The main problem is unoptimized database recording in HA and affects when you have many devices.
More than 60 different devices:
Start on a new SSD:
Thanks, so I can just exclude the sensor from the recorder: in config.yaml and I'm going to be fine?
Then there will be no measurement history.
For recording, all unnecessary and uninteresting entities, such as reception levels and others, are disabled. Store depth limitation for one month and data saved at one minute intervals.
There is a desire to use the database for 1 year, but in HA this is impossible - it will either work slowly on weak machines, or the size of the database will be TB. And every day the database in TB will be completely rewritten :(
The ratio of the desired device memory (RAM) size for normal operation of Home Assistant is approximately 2 GB + database file size.
Pardon my confusion: is this to say that there is no good way to connect BLE sensors to Home Assistant? Based on what I can see here, using a BT dongle on the HA machine ruins the SSD, and using an ESPHome device in the middle misses advertisements and wastes battery on the sensors. I have ten LYWSD03MMC, is there any solution that truly works?
If I understood this correctly you can exclude the BLE sensor form recording to the DB, and if you need an history record for any of them create a dummy sensor that updates every once in a while, like when the values changes for more than 0.5 °C, or more than 1% HR.
That's my config.yaml setup, the "entity_globs:" section accepts wildcards so it should be pretty easy to filter the BLE sensors:
recorder: purge_keep_days: 14 commit_interval: 60 exclude: domains: # - sensor - automation - script - light - camera - delay - group - media_player - timer entities: - light.ingresso entity_globs: [...] - sensor.termometro*
This still does not allow Home Assistant to work for a long time on eMMC and especially on SD cards. Only on SSD or HDD.
With SSDs, it is additionally necessary to provide power redundancy, due to the lack of a “fast parking” function for SSDs in Linux. SSDs themselves have this function, but there is no implementation in Linux.
This still does not allow Home Assistant to work for a long time on eMMC and especially on SD cards. Only on SSD or HDD. With SSDs, it is additionally necessary to provide power redundancy, due to the lack of a “fast parking” function for SSDs in Linux. SSDs themselves have this function, but there is no implementation in Linux.
That' s a generic HA anyways, not related to BLE devices in particular (I have switched to SSD a long time ago also gaining some speed at boot). My plan is also to have a full HW backup ready to switch in in case of failure (another raspberry, with another SSD); at the cost SSD are now I can consider it like some sort of "consumable", to switch completely every 2/3 years.
I'm interested in the power redundancy thing, which I was not aware of: what am I risking right now using a SSD without it? I don't think it's physical "parking" since it doesn't have moving parts, so what's the problem?
I don't think it's physical "parking" since it doesn't have moving parts, so what's the problem?
Failure due to loss of markings or programs due to a power failure. Server SSDs have additional capacities and other elements and a program for emergency reset of vital data. This is not the case in household ones.
https://www.phoronix.com/news/NVMe-Abrupt-Shutdown-Linux
They never finished it.
smartctl -a /dev/nvme0
Unsafe Shutdowns: 3
Hello,
You don't have to worry about that. First, under GNU/Linux, the data is not committed to disk on every write. For example for ext4 it is usually 4 seconds.
https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
commit=nrsec (*) Ext4 can be told to sync all its data and metadata
every 'nrsec' seconds. The default value is 5 seconds.
This means that if you lose your power, you will lose
as much as the latest 5 seconds of work (your
filesystem will not be damaged though, thanks to the
journaling). This default value (or any low value)
will hurt performance, but it's good for data-safety.
Setting it to 0 will have the same effect as leaving
it at the default (5 seconds).
Setting it to very large values will improve
performance.
Secondly, the SSD firmware decides where to write data and data is not always contiguous. The firmware choose to write where ZEROs are written. Thirdly, you can trim your SSD using Linux trim or on file removal, which gives SSD firmware more space to choose.
You may use the discard option to automatically trim your partitions. This is an example in /etc/fstab
/dev/sda1 /mnt/data ext4 defaults,nofail,discard 0 0
Last if you are really worried you may use a RAM disc for a folder or some files.
Again, just don't worry about that!!!
@ffries
Only HA is installed on the SSD.
/var/lib/bluetooth/ :
eMMC (system) and NVMe (Home Assistant), All logs are written to RAMdisk
UUID=xxxxxx-xxxx-xxxx-xxxx-xxxxxxxx / ext4 defaults,noatime,commit=600,errors=remount-ro,x-gvfs-hide 0 1
UUID=xxxxxx-xxxx-xxxx-xxxx-xxxxxxxx /mnt/nvme ext4 defaults,noatime,commit=600,discard 0 2
Last if you are really worried you may use a RAM disc for a folder or some files.
https://stackoverflow.com/questions/65957677/bluez-change-local-storage-directory
Symlinking does not work for me. The service fails with the error "Not a directory".
Again, just don't worry about that!!!
Nobody cares - it's just fun.
Unsafe Shutdowns: ++++ ?
Linux (Bluez, Bleak and over API) only partially supports Bluetooth version 4.2 (released December 2, 2014) ---- ?
PS: SSD is a consumable item. Change every 2 years. Single board computer with eMMC - also replace every 2 years.
Replace the SD card on RPI 5 with a new one every month or after a couple of weeks.
Replace the SD card on RPI 5 with a new one every month or after a couple of weeks.
With Raspberry Pi, enabling the Overlay Filesystem (which uses the RAM to store changes) allows setting the SD card in read-only mode (or defining an overlayed read-only partition for the root filesystem and a non-overlayed RW partition for the data). It can be configured via raspi-config. With the Overlay Filesystem, the system can be turned off without shutdown (but if a RW data partition is defined, it will not be protected).
Home Assistant is very slow on an SD card. It's the same on an SSD drive via ATA.
It takes a very long time for the graphs to be drawn and various errors constantly appear.
Only on NVMe (PCIe) and eMMC Home Assistant starts to work decently. Requires disk read-write speed of 1 GB/sec.
With the Overlay Filesystem, the system can be turned off without shutdown (but if a RW data partition is defined, it will not be protected).
If the card is recording enabled, any power cycle may cause the entire card to fail.
@pvvx I am running HA on an Nvme and SSD on different systems without any issue and I have around 100 objects. I have been a GNU/Linux user for more that 20 years. Please don't mess-up an NVMe/SSD with an SD card.
The NVMe/SSD has an internal controller that prevents wearing, moving data at different places. This is done in hardware without user intervention.
On the converse, data on an SD card does not have an internal controller. An SD card can indeed wear very quickly when data is constantly written at the same place (for example logs).
The graphs shown are not real disk access but only disk commits. Data is not written instantly but only after a few seconds. Therefore an NVMe/SSD will not wear even when written 100 times a second. Under GNU/Linux, data is written to RAM and then committed to disk. This is done without intervention.
i.e. your graphs show commits to RAM and every 5 seconds or so, data is written to disk. Only the OS will decide when to write to disk. To my knowledge this is not possible to access the real statistics of disk writing as this is done transparently in the OS and this is kind of a "black-box".
All this discussion is pointless and you should close this issue. User without GNU/Linux knowledge might become confused if they don't know the difference between CF and NVMe/SSD and no previous system knowledge.
If you need more information, please ask otherwise please mark this issue as resolved.
@ffries - what are you talking about?
-
commit=600 -
The record count data shown in the graph comes from the NNME controller, not the OS.
Study the subject before you write nonsense.
If you need more information, please ask otherwise please mark this issue as resolved.
It looks like you need some guiding questions. :)
- What happens to NVMe if you turn off the power during recording?
- What will be the TBW if you constantly add one byte to each file? That is, a refined concept of “write amplification.” The resource of inexpensive NVMe is up to 1000..1500 cell rewrites or erases. When appending, 2 actions occur - erasing a large block and writing a sector. The maximum calculated TBW is disk volume * 1000.
- How does caching work on an NVMe SSD? How much does the number of cell rewrites actually increase? :)
I am running HA on an Nvme and SSD on different systems without any issue and I have around 100 objects.
Your objects change value once a day. That's 100 kilobyte records per day :)
I have 60 BLE only devices, and each has 5..10 objects. They change values every 10 seconds or more often. This is writing about 400 database results per second, plus overwriting BLE scan files in Bluez.
I have been a GNU/Linux user for more that 20 years.
I have been using GNU/Linux and other operating systems since their inception.
I started with development on 8-bit processors, then development and creation of BIOS for initial processors i8088...386.. In our country, every second factory or production facility uses my developments.
Participated in all popular electronic folk topics since 1985. My developments were used throughout them.
If you have technical questions, not nonsense, ask. :P
The BLE devices do update quite often.
I assume it will ameliorate (eliminate?) the excessive write situation if I convert it to be using Zigbee instead of BLE?
(ref: https://smarthomescene.com/guides/convert-xiaomi-lywsd03mmc-from-bluetooth-to-zigbee/)
It states that in Zigbee firmware the sensor only reports every 5 minutes, which I hope can be made configurable.
Or tweak BLE Measurement Interval/Advertisement Interval to max so HA updates every 3min?
I wish to understand what is this case about...
I don't think I fully understand this issue. I have raspberry pi 4 with 32 GB sd card and 4 thermometers in BTHome v2 mode. Does this affect me? Should I worry about lifespan of the sd card? I was expecting to changed it every year to be safe, but reading this meaning I should do it more frequently or switch to HDD? I do not need so detailed history (only maybe for week, then its okay to have average value for lets say 1 hour period, is there an easy way to setup it like that?)
The BLE devices do update quite often.
Zigbee Tyua:
WiFi:
Result:
Step - this HA rewrites the database every day
NanoPi-R5S, RAM: 4GB, 16GB eMMC, NVME PCIe 512GB
I do not need so detailed history (only maybe for week, then its okay to have average value for lets say 1 hour period, is there an easy way to setup it like that?)
The entire database is rewritten every day to remove old data.
When using an SD card, there will be problems with a couple of dozen of any sensors. Errors will occur due to very low performance.
For comfortable use of HA, a media speed of 1 GB per second is required.
On Debian(Armbian tested) you can bypass the option to write the BlueZ cache to disk in two ways:
First one: recompile BlueZ with the modified STORAGEDIR option in a folder say /tmp/lib/bluetooth
The Second way is more simple:
Already on a working system (Armbian tested):
- Install bindfs:
sudo apt install bindfs - Add an entry in /etc/fstab with content like:
...
tmpfs /bluetooth_cache tmpfs defaults,nosuid 0 0
bindfs#/bluetooth_cache /var/lib/bluetooth fuse user,comment=x-gvfs-hide 0 0
(pay attention to the option comment=x-gvfs-hide - it hides from BlueZ that this is a binded folder).
3. Stop the bluetooth service:
sudo systemctl stop bluetooth.service
4. Delete the old folder before rebooting (this is important, otherwise we will get boot errors after rebooting):
sudo rm -Rf /var/lib/bluetooth
5. Reboot
My Orange Pi (Armbian only SD-card) + Home assistant were disabled logging to the memory card, the memory card by the way is used Smasung High Endurance.
In Home assistant instead of standard database MariaDB is used (it is strongly advised to install SD-card less host writes to it, it can be installed via Add-ons store).
In the configuration for this database is used database storage no more than a two month and disabled some records of sensors such as System Monitor - the database itself is compressed in memory and takes up little space - about 90MB for a month (it is three sensors LYWSD03MMC, BTH01, TH05 and so on).
recorder:
db_url: !secret mariadb_url
purge_keep_days: 60
commit_interval: 1200
exclude:
entities:
- sensor.last_boot
- sensor.processor_temperature
- sensor.processor_use
- sensor.memory_use_percent
- sensor.disk_use_percent
- sensor.date_time
- sensor.home_assistant_supervisor_cpu_percent
- sensor.home_assistant_supervisor_memory_percent
When my stats for the day "Uptime MMC writen" with binded /var/lib/bluetooth folder in tmpfs are accumulated - I will post a graph comparison. If anyone is interested.
about 90MB for a month (it is three sensors
90MB per month / three sensors = 30MB per sensor?
I have 89 or more devices in one of the HAs.
This will come out to 89*30 = 2670 MB.
The actual size is 6059 MB. The database is over a year old.
The size of the database has increased by one and a half times after the recent (week) connection of 9 devices with reporting 10 seconds. But there is a desire to set a period of 3 seconds for a dozen devices. :)
recorder:
purge_keep_days: 31
commit_interval: 60
On Debian(Armbian tested) you can bypass the option to write the BlueZ cache to disk in two ways: First one: recompile BlueZ with the modified STORAGEDIR option in a folder say /tmp/lib/bluetooth
@Deoptim According to the "Uptime MMC writen" graph, the recording decreased by 320 megabytes per day.
@pvvx
If you want to change the database to MariaDB - here I tried this way of conversion SQLite to MySQL (original source).
There is even more economical option (for SD-Card or eMMC host writes) to use mariadb-inmemory Add-on (discussion).
In a month I will post the results of the comparison with the original database "home-assistant_v2.db" / local MariaDB / In-memory MariaDB.
i'm using this trick, i'm running homeassistant os + sd + hdd on a rpi... go to /var/lib/bluetooth/MACADDRESS and just run
rm -r cache && > cache
no more unnecessary writes to the microsd and all (seems) working at the moment, no needs to disable active scan (i have a switchbot bot to activate and requires the active scan to work)
(i just started studing linux so i did all by my self, very happy :D)
These are all problems with Bluez and it is not clear why it is used in Home Assistant. Bluez does not yet fully support Bluetooth 4.2 (Bluez: Core specification 4.2, 2014). Modern Bluetooth has version 5.4.
@pvvx
Here's a late one, but here's a little statistic nonetheless:
Russian Language
(Google Translated)
@Deoptim
Для тестов HA 20 октября 2023 был установлен новый NVMe 256G (PCIe MAXIO MAP1202A-F1C 2 x YMC3G001Tb62BA1C0) в новую NanoPi-R5S (RAM: 4GB LPDDR4X, 16GB eMMC). Сама база HA переехала с прошлой платформы, с SSD.
На сегодня:
Со старта HA работает постоянно (питание автономное). Кроме HA ничего другого не установлено, docker не используется. Система на eMMC, HA c базой на NVMe. Производилось несколько обновлений всего ПО, но это копейки по сравнению с суточной записью HA.
Датчиков и прочих устройств около сотни - 3 USB адаптера: 2 BLE, 1 Zigbee, по проводной сети всё остальное. В настройках HA приняты меры фильтрации записи лишнего в базу...
С примерно с 30-ти датчиков данные обновляются каждые 10 секунд. С других в среднем к 30 сек.
К примеру группа BLE на чипах PHY622x2 исключительно для теста прошивок:
Аналогично на все поддерживаемые устройства (это примерно к 20 типам устройств, часть из них ещё с 3-мя вариантами интерфейсов связи - Zigbee, BLE, Zigbee+BLE).
Остальные участвуют в местной автоматизации. Пример бака с водой:
Если система отработает год - полтора, то всё равно менять на новую платформу, т.к. для комфортной работы требования к производительности и ресурсам всегда растут.