mdadm script fails with Intel Matrix Storage RAID
Opened this issue · 5 comments
Ups, looks like I opened another issue in the wrong place
Now creating another one here, please feel free to close a duplicate.
The problem
I am trying to setup mdadm application monitoring. mdadm script fails with
/etc/snmp/mdadm /etc/snmp/mdadm: line 53: (2 - ): syntax error: operand expected (error token is ")")
debug run of the same script is attached.
mdadm-debug-run.log
[librenms@monitoring ~]$ ./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS | 24.5.0-14-gc777d5429 (2024-05-29T22:42:05+03:00)
DB Schema | 2024_04_29_183605_custom_maps_drop_background_suffix_and_background_version (294)
PHP | 8.3.7
Python | 3.9.18
Database | MariaDB 11.3.2-MariaDB
RRDTool | 1.7.2
SNMP | 5.9.1
===========================================
[OK] Composer Version: 2.7.6
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK] Database and column collations are correct
[OK] Database schema correct
[OK] MySQL and PHP time match
[OK] Active pollers found
[OK] Dispatcher Service not detected
[OK] Locks are functional
[OK] Python poller wrapper is polling
[OK] Redis is unavailable
[OK] rrdtool version ok
[OK] Connected to rrdcached
[librenms@monitoring ~]$
Monitored host has four disks in two software raid arrays.
Two SATA 512GB SSDs are configured in RAID1 by Intel Matrix Storage RAID in EFI.
Two NVMe 1TB disks are configured in RAID1 by operating system.
mdadm configuration, lshw output, lsblk output is added below.
Thanks for looking into this.
cat /etc/mdadm.conf
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md/imsm UUID=f14c58a0:f20c0ab1:ddadb5be:dc8ddb75
ARRAY /dev/md/0 metadata=1.2 UUID=60a657e6:bb76e556:b27a83db:734a3edc name=<REMOVED>
ARRAY /dev/md/Volume0 container=f14c58a0:f20c0ab1:ddadb5be:dc8ddb75 member=0 UUID=bf4cc729:5d9bf0ae:ce01b1bf:46fcb1f4
-------------------------------------------------------
cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 nvme0n1[2] nvme1n1[1]
976630464 blocks super 1.2 [2/2] [UU]
bitmap: 3/8 pages [12KB], 65536KB chunk
md126 : active raid1 sda[1] sdb[0]
475099136 blocks super external:/md127/0 [2/2] [UU]
md127 : inactive sda[1](S) sdb[0](S)
10402 blocks super external:imsm
unused devices: <none>
mdadm --examine /dev/md/imsm
/dev/md/imsm:
Magic : Intel Raid ISM Cfg Sig.
Version : 1.1.00
Orig Family : 3690cb86
Family : 3690cb86
Generation : 000ed44c
Creation Time : Thu Jan 18 14:13:41 2024
Attributes : All supported
UUID : f14c58a0:f20c0ab1:ddadb5be:dc8ddb75
Checksum : 99b796dd correct
MPB Sectors : 1
Disks : 2
RAID Devices : 1
Disk01 Serial : S42YNE0M500509N
State : active
Id : 00000001
Usable Size : 1000204814 (476.93 GiB 512.10 GB)
[Volume0]:
Subarray : 0
UUID : bf4cc729:5d9bf0ae:ce01b1bf:46fcb1f4
RAID Level : 1
Members : 2
Slots : [UU]
Failed disk : none
This Slot : 1
Sector Size : 512
Array Size : 950198272 (453.09 GiB 486.50 GB)
Per Dev Size : 950200320 (453.09 GiB 486.50 GB)
Sector Offset : 0
Num Stripes : 3711712
Chunk Size : 64 KiB
Reserved : 0
Migrate State : idle
Map State : normal
Dirty State : clean
RWH Policy : off
Volume ID : 1
Disk00 Serial : S42YNE0M500504E
State : active
Id : 00000000
Usable Size : 1000204814 (476.93 GiB 512.10 GB)
mdadm --examine /dev/md/0
/dev/md/0:
MBR Magic : aa55
Partition[0] : 1953260927 sectors at 1 (type ee)
mdadm --examine /dev/md/Volume0
/dev/md/Volume0:
MBR Magic : aa55
Partition[0] : 950198271 sectors at 1 (type ee)
lshw -class disk -class storage
*-raid
description: RAID bus controller
product: SATA Controller [RAID Mode]
vendor: Intel Corporation
physical id: 17
bus info: pci@0000:00:17.0
logical name: scsi0
logical name: scsi1
logical name: scsi5
version: 00
width: 32 bits
clock: 66MHz
capabilities: raid msi pm bus_master cap_list emulated
configuration: driver=ahci latency=0
resources: irq:37 memory:aaf24000-aaf25fff memory:aaf27000-aaf270ff ioport:3050(size=8) ioport:3040(size=4) ioport:3020(size=32) memory:aaf26000-aaf267ff
*-disk:0
description: ATA Disk
product: Samsung SSD 860
physical id: 0
bus info: scsi@0:0.0.0
logical name: /dev/sda
version: 1B6Q
serial: S42YNE0M500504E
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=43866b72-683b-4ac0-887f-94fdb01937e4 logicalsectorsize=512 sectorsize=512
*-disk:1
description: ATA Disk
product: Samsung SSD 860
physical id: 1
bus info: scsi@1:0.0.0
logical name: /dev/sdb
version: 1B6Q
serial: S42YNE0M500509N
size: 476GiB (512GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=43866b72-683b-4ac0-887f-94fdb01937e4 logicalsectorsize=512 sectorsize=512
*-cdrom
description: DVD-RAM writer
product: DVD-RAM GHC0N
vendor: HL-DT-ST
physical id: 0.0.0
bus info: scsi@5:0.0.0
logical name: /dev/cdrom
logical name: /dev/sr0
version: MA02
capabilities: removable audio cd-r cd-rw dvd dvd-r dvd-ram
configuration: ansiversion=5 status=nodisc
*-nvme
description: NVMe device
product: Samsung SSD 980 PRO with Heatsink 1TB
vendor: Samsung Electronics Co Ltd
physical id: 0
bus info: pci@0000:17:00.0
logical name: /dev/nvme0
version: 5B2QGXA7
serial: S6WSNJ0W122428R
width: 64 bits
clock: 33MHz
capabilities: nvme pm msi pciexpress msix nvm_express bus_master cap_list
configuration: driver=nvme latency=0 nqn=nqn.1994-11.com.samsung:nvme:980PRO:M.2:S6WSNJ0W122428R state=live
resources: irq:38 memory:c5e00000-c5e03fff
*-namespace:0
description: NVMe disk
physical id: 0
logical name: /dev/ng0n1
*-namespace:1
description: NVMe disk
physical id: 1
bus info: nvme@0:1
logical name: /dev/nvme0n1
size: 931GiB (1TB)
configuration: logicalsectorsize=512 sectorsize=512 wwid=eui.002538b131408bde
*-nvme
description: NVMe device
product: Samsung SSD 980 PRO with Heatsink 1TB
vendor: Samsung Electronics Co Ltd
physical id: 0
bus info: pci@0000:18:00.0
logical name: /dev/nvme1
version: 5B2QGXA7
serial: S6WSNS0W404164V
width: 64 bits
clock: 33MHz
capabilities: nvme pm msi pciexpress msix nvm_express bus_master cap_list
configuration: driver=nvme latency=0 nqn=nqn.1994-11.com.samsung:nvme:980PRO:M.2:S6WSNS0W404164V state=live
resources: irq:40 memory:c5d00000-c5d03fff
*-namespace:0
description: NVMe disk
physical id: 0
logical name: /dev/ng1n1
*-namespace:1
description: NVMe disk
physical id: 1
bus info: nvme@1:1
logical name: /dev/nvme1n1
size: 931GiB (1TB)
capabilities: partitioned partitioned:dos
configuration: logicalsectorsize=512 sectorsize=512 signature=4751c961 wwid=eui.002538b431402d4c
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 476.9G 0 disk
├─md126 9:126 0 453.1G 0 raid1
│ ├─md126p1 259:3 0 600M 0 part /boot/efi
│ ├─md126p2 259:4 0 1G 0 part /boot
│ └─md126p3 259:5 0 375.5G 0 part
│ ├─almalinux-root 253:0 0 70G 0 lvm /var/named/chroot/usr/share/GeoIP
│ │ /var/named/chroot/usr/lib64/named
│ │ /var/named/chroot/usr/lib64/bind
│ │ /var/named/chroot/etc/named
│ │ /var/named/chroot/etc/services
│ │ /var/named/chroot/etc/protocols
│ │ /var/named/chroot/etc/crypto-policies/back-ends/bind.config
│ │ /var/named/chroot/etc/rndc.key
│ │ /var/named/chroot/etc/named.rfc1912.zones
│ │ /var/named/chroot/etc/named.conf
│ │ /var/named/chroot/etc/named.root.key
│ │ /var/named/chroot/etc/localtime
│ │ /
│ ├─almalinux-swap 253:1 0 15.5G 0 lvm [SWAP]
│ ├─almalinux-var_tmp 253:2 0 10G 0 lvm /var/tmp
│ ├─almalinux-var 253:3 0 40G 0 lvm /var/named/chroot/var/named
│ │ /var
│ ├─almalinux-home 253:4 0 30G 0 lvm /home
│ └─almalinux-tmp 253:5 0 10G 0 lvm /tmp
└─md127 9:127 0 0B 0 md
sdb 8:16 0 476.9G 0 disk
├─md126 9:126 0 453.1G 0 raid1
│ ├─md126p1 259:3 0 600M 0 part /boot/efi
│ ├─md126p2 259:4 0 1G 0 part /boot
│ └─md126p3 259:5 0 375.5G 0 part
│ ├─almalinux-root 253:0 0 70G 0 lvm /var/named/chroot/usr/share/GeoIP
│ │ /var/named/chroot/usr/lib64/named
│ │ /var/named/chroot/usr/lib64/bind
│ │ /var/named/chroot/etc/named
│ │ /var/named/chroot/etc/services
│ │ /var/named/chroot/etc/protocols
│ │ /var/named/chroot/etc/crypto-policies/back-ends/bind.config
│ │ /var/named/chroot/etc/rndc.key
│ │ /var/named/chroot/etc/named.rfc1912.zones
│ │ /var/named/chroot/etc/named.conf
│ │ /var/named/chroot/etc/named.root.key
│ │ /var/named/chroot/etc/localtime
│ │ /
│ ├─almalinux-swap 253:1 0 15.5G 0 lvm [SWAP]
│ ├─almalinux-var_tmp 253:2 0 10G 0 lvm /var/tmp
│ ├─almalinux-var 253:3 0 40G 0 lvm /var/named/chroot/var/named
│ │ /var
│ ├─almalinux-home 253:4 0 30G 0 lvm /home
│ └─almalinux-tmp 253:5 0 10G 0 lvm /tmp
└─md127 9:127 0 0B 0 md
sr0 11:0 1 1024M 0 rom
nvme0n1 259:0 0 931.5G 0 disk
└─md0 9:0 0 931.4G 0 raid1
└─md0p1 259:2 0 931.4G 0 part /vms
nvme1n1 259:1 0 931.5G 0 disk
└─md0 9:0 0 931.4G 0 raid1
└─md0p1 259:2 0 931.4G 0 part /vms
This patch suits my needs:
--- mdadm.orig 2024-05-31 08:52:52.153777132 +0300
+++ mdadm 2024-05-31 11:25:44.772243743 +0300
@@ -40,6 +40,10 @@
[[ "${mdadmArray}" =~ '/dev/md'[[:digit:]]+'p' ]] && continue
mdadmName="$(basename "$(realpath "${mdadmArray}")")"
+
+ # Ignore inactive arrays
+ [[ $(grep "^${mdadmName}" /proc/mdstat) =~ 'inactive' ]] && continue
+
mdadmSysDev="/sys/block/${mdadmName}"
degraded=$(maybe_get "${mdadmSysDev}/md/degraded")
@shpokas Thanks. Tested that and does not appear to break anything. What does /proc/mdstat
look like there?
There are likely some other things that need cleaned up as well.
Derp! Sorry, missed that. You did. Thanks!
Currently pondering best way to handle inactive arrays.
ls /sys/block/md0/slaves/
ls -l /sys/block/md0/
cat /sys/block/md0/md/level
cat /sys/block/md0/md/raid_disks
Sorry, those commands should be for md127
.
Currently thinking of just adding a counter for inactive.