Tracknodes keeps a history of node state and comment changes. It allows system administrators of HPC systems to determine when nodes were down and discover trends such as recurring issues. Supports Torque, PBSpro and has limited support for SLURM.
$ pip install tracknodesor
$ easy_install tracknodesSetup a cronjob on an admin node. This step is required for node state changes to be tracked.
$ crontab -u root -e
# Track Node State Every Minute
* * * * * (/usr/bin/tracknodes --update >/dev/null 2>&1)Use the below command to see the history of node changes.
$ tracknodes
History of Nodes
=========
n101 | 2016-11-28 21:30:01 | online | ''
n101 | 2016-11-28 20:30:01 | offline,down | 'Hardware issue bad DIMM'
n092 | 2016-11-27 19:30:01 | online | ''
n092 | 2016-11-27 12:00:01 | offline | 'Hardware issue failed disk'
n021 | 2016-11-27 09:00:01 | online | ''
n021 | 2016-11-26 19:00:01 | offline,down | 'DIMM Configuration Error'
-- --You can setup the configuration file for tracknodes to change the database location or the command to get node status. Use the below as an example.
$ cat /etc/tracknodes.conf
---
dbfile: "/opt/tracknodes.db"
cmd: "/opt/pbsnodes"Tracknodes uses a sqlite database to store the node history, you can determine what database its using with the -v argument.
$ tracknodes -v
Resource Manager Detected as torque
cmd: /opt/pbsnodes
dbfile: ~/.tracknodes.db
...For usage information you can use --help.
$ tracknodes --help
Usage: tracknodes [options]
Options:
-h, --help show this help message and exit
-U, --update Update Database From Current Node States
-f DBFILE, --dbfile=DBFILE
Database File
-c CMD, --cmd=CMD
Location of command to show node state, example: /opt/pbsnodes, /opt/sinfo
-v, --verbose Verbose Outputtracknodes is released under the GPLv3 License.