Can you please link to a tutorial/guide on how I'm actually supposed to implement these?
Jsalas424 opened this issue · 6 comments
Jsalas424 commented
I'm having issues transitioning from smartmon.sh to smartmon.py and would appreciate some guidance.
I have copied smartmon.py into /usr/local/bin and made it an executable file. I have installed the prometheus python client from here: https://github.com/prometheus/client_python. I have added the --collector.textfile.directory=/var/state/prometheus flag to my node exporter and I was previously successfully scraping metrics using the smartmon.sh program but once I switched the smartmon.py I can't get my service file to work correct.
When I was using smartmon.sh, I was running this service file successfully:
[Unit]
Description=Export smartctl metrics to Prometheus Node Exporter
[Service]
Nice=-10
ExecStart=/bin/sh -c 'exec /usr/local/bin/smartmon.py > /var/state/prometheus/smartmon.prom'
# Write nothing except the output file
ProtectSystem=strict
ReadWritePaths=/var/state/prometheus
# Shell needs a temp directory
PrivateTmp=true
ProtectHome=tmpfs
[Install]
WantedBy=multi-user.target
I've tried two other iterations shown below, but ultimately the python script runs one time and then shuts off. What am I doing wrong?
[Unit]
Description=Export smartctl metrics to Prometheus Node Exporter
[Service]
Nice=-10
#ExecStart=/bin/sh -c 'exec /usr/local/bin/smartmon.py > /var/state/prometheus/smartmon.prom'
#ExecStart=/usr/local/bin/smartmon.py
ExecStart=/bin/sh -c '/usr/local/bin/smartmon.py | sponge /var/state/prometheus/smartmon.prom'
[the rest is the same as the above service file]
Here is also my node_exporter service file:
GNU nano 5.4 /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/var/state/prometheus
[Install]
WantedBy=multi-user.target
Thank you for the help!
dswarbrick commented
Neither the smartmon.sh nor the smartmon.py scripts are daemons. They are intended to run once, then exit.
You will need a systemd timer to make the service run at regular intervals, and should change Type=simple to Type=oneshot in your service.
You can find some inspiration in the systemd units installed by the Debian prometheus-node-exporter-collectors package: https://salsa.debian.org/go-team/packages/prometheus-node-exporter-collectors/-/tree/debian/sid/debian
Jsalas424 commented
@dswarbrick thank you for the help, I did get this running but have run into another issue. After implementing the systemd timer method, my syslog is flooded with error messages of the form "... was collected before with the same name and label values"
TracheServ node_exporter[654339]: ts=2022-06-19T23:51:12.769Z caller=stdlib.go:105 level=error msg="error gathering metrics: 413 error(s) occurred:\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"device_model\" value:\"Samsung SSD 860 EVO 250GB\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"RVT04B6Q\" > label:<name:\"model_family\" value:\"Samsung based SSDs\" > label:<name:\"serial_number\" value:\"S59WNJ0N304506K\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"device_model\" value:\"Samsung SSD 850 EVO 250GB\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"EMT01B6Q\" > label:<name:\"model_family\" value:\"Samsung based SSDs\" > label:<name:\"serial_number\" value:\"S21NNSAFC02256H\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"device_model\" value:\"Samsung SSD 860 EVO 1TB\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"RVT04B6Q\" > label:<name:\"model_family\" value:\"Samsung based SSDs\" > label:<name:\"serial_number\" value:\"S5B3NDFNA02148D\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"device_model\" value:\"Samsung SSD 860 EVO 1TB\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"RVT04B6Q\" > label:<name:\"model_family\" value:\"Samsung based SSDs\" > label:<name:\"serial_number\" value:\"S5B3NDFN915923H\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"device_model\" value:\"WDC WD4003FRYZ-01F0DB0\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"01.01H01\" > label:<name:\"model_family\" value:\"Western Digital Gold\" > label:<name:\"serial_number\" value:\"VBGYEPVF\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"device_model\" value:\"ST3000NM0033 81Y9799 81Y3865IBM\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"BB59\" > label:<name:\"model_family\" value:\"\" > label:<name:\"serial_number\" value:\"Z1Y0K9N8\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"device_model\" value:\"TOSHIBA MG03ACA100\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"FL1D\" > label:<name:\"model_family\" value:\"Toshiba 3.5\\\" MG03ACAxxx(Y) Enterprise HDD\" > label:<name:\"serial_number\" value:\"44N2KLTFF\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"device_model\" value:\"TOSHIBA MG03ACA100\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"FL1D\" > label:<name:\"model_family\" value:\"Toshiba 3.5\\\" MG03ACAxxx(Y) Enterprise HDD\" > label:<name:\"serial_number\" value:\"3423KG6HF\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:17517 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:27 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:84 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:33 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:1.3290518032e+10 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:34218 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:7206 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:1956 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:33 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:1.33061639514e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:13054 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:18 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:360 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:36 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:1.05486405539e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:13054 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:17 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:357 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:36 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:1.05485863923e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:4900 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:198 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:47 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_event_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"current_pending_sector\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:1.20790859e+08 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:67 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:2 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:2.860474765e+09 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:59364 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:52 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reported_uncorrect\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"command_timeout\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:48 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:2568 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:48 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"hardware_ecc_recovered\" > gauge:<value:1.20790859e+08 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"current_pending_sector\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:4458 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:78 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:50387 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:78 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:11826 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:39 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_event_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:8.9431202034e+10 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_read\" > gauge:<value:5.0053092712e+10 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:6339 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:149 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:59902 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:149 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:11965 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:38 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_event_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:9.751680071e+10 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_read\" > gauge:<value:5.2769926446e+10 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:96 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:95 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:67 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:93 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:92 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:8 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:67 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:97 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:76 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:64 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:97 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:76 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:64 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:127 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_event_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"current_pending_sector\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:200 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:81 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:93 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:94 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:33 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reported_uncorrect\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"command_timeout\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:52 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:48 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"hardware_ecc_recovered\" > gauge:<value:49 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"current_pending_sector\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:200 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:101 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > la
My temporary solution was editing the service file to set StandardError=null as such:
GNU nano 5.4 /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/var/state/prometheus
StandardError=null
[Install]
WantedBy=multi-user.target
Any suggestions on how to deal with this?
dswarbrick commented
@Jsalas424 This sounds like the collector is running (and appending to the spool file) multiple times within the same scrape interval. Do you perhaps have it still enabled as a cronjob, or perhaps have multiple instances of the systemd timer, executing the service multiple times?
If you see duplicate metrics in the /var/state/prometheus/smartmon.prom, that would be a strong indicator of such a thing.
Jsalas424 commented
Hi @dswarbrick, per your advice here's what I found in that directory:
root@TracheServ:~# ls /var/state/prometheus/
smartctl.prom smartmon.prom
root@TracheServ:~# tail /var/state/prometheus/smartmon.prom
# HELP smartmon_smartctl_run SMART metric smartctl_run
# TYPE smartmon_smartctl_run gauge
smartmon_smartctl_run{device="/dev/sda",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdb",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdc",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdd",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sde",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdf",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdg",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdh",disk="0"} 1655020313
root@TracheServ:~# tail /var/state/prometheus/smartctl.prom
# HELP smartmon_smartctl_run SMART metric smartctl_run
# TYPE smartmon_smartctl_run gauge
smartmon_smartctl_run{device="/dev/sda",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdb",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdc",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdd",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sde",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdf",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdg",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdh",disk="0"} 1652064537
root@TracheServ:~# ls /usr/local/bin/
cmdprox.py node_exporter pip pip3 pip3.7 prox pve_exporter smartmon.py smartmon.sh
root@TracheServ:~# ls /etc/systemd/system
ceph.target.wants node_exporter.service sshd.service
dbus-org.freedesktop.timesync1.service nut-server.service sysinit.target.wants
default.target.wants prometheus-pve-exporter.service syslog.service
getty.target.wants pve-manager.service timers.target.wants
ifupdown-wait-online.service remote-fs.target.wants zed.service
influxd.service smartd.service zfs-import.target.wants
iscsi.service smart_exporter.service zfs.target.wants
mnt-pve-spare.mount smart_exporter.service.save zfs-volumes.target.wants
multi-user.target.wants smart_exporter.timer
network-online.target.wants sockets.target.wants
So I just deleted the smartmon.sh, but that didn't fix the issue. So I kept hunting and here's what else I found.
root@TracheServ:~# cat /etc/systemd/system/smart_exporter.timer
[Unit]
Description=Run smart metrics collection every 5 minutes
ConditionPathExists=/usr/sbin/smartctl
ConditionPathExistsGlob=|/dev/sd*
ConditionPathExistsGlob=|/dev/hd*
# TODO: Figure out if NVMe returns usable metrics through smartmon.sh.
#ConditionPathExistsGlob=|/dev/nvme*
[Timer]
OnBootSec=0
OnUnitActiveSec=5min
[Install]
WantedBy=timers.target
root@TracheServ:~# cat /etc/systemd/system/smart_exporter.service
[Unit]
Description=Export smartctl metrics to Prometheus Node Exporter
[Service]
Type=oneshot
Nice=-10
ExecStart=/bin/sh -c 'exec /usr/local/bin/smartmon.py | sponge /var/state/prometheus/smartmon.prom'
# Write nothing except the output file
ProtectSystem=strict
ReadWritePaths=/var/state/prometheus
# Shell needs a temp directory
PrivateTmp=true
ProtectHome=tmpfs
[Install]
WantedBy=multi-user.target
root@TracheServ:~# cat /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/var/state/prometheus
StandardError=null
# Add that --collector.textfile flag after execstart once you figure out how to use smart monitoring
#--collector.textfile.directory=/var/state/prometheus
[Install]
WantedBy=multi-user.target
root@TracheServ:~# ls /etc/systemd/system/multi-user.target.wants
smart_exporter.service smartmontools.service
So there i a smartmontools/service AND a smart_exporter.service but I can't figure out how to find what's triggering both of these.
Any other hints?
dswarbrick commented
@Jsalas424 You have two metrics files in your /var/state/prometheus directory with identical metrics (albeit different timestamps). This is why you are seeing the error message.
It looks like the smartctl.prom file has the older timestamps (by roughly 34 days), so presumably whatever collector was producing that has stopped. If you remove the smartctl.prom file, node_exporter should stop complaining.
Jsalas424 commented
@dswarbrick you are correct, the last step was to delete smartctl.prom! Thanks for everything!