/gumoniter

GPU user moniter, a collection of python scripts to collect gpu usages

Primary LanguagePythonMIT LicenseMIT

gumoniter

GPU user moniter, a collection of python scripts to collect gpu usages

Environment Setup

  • This script runs on admin node, slave nodes should have gpustat installed.
pip install gpustat
  • slave nodes must be able to ssh without password, it can be achieved by ssh-key-copy

Configure

This script can be configured in file 'config'.

  • interval: collects data for every interval minutes
  • hosts: a list of hostname or ip_address (sshable)
  • log_path: log files path. One file for one day.

Architecture

  • moniter.py: a data collector that queries GPUs and processes information from nodes.
  • report.py: a post-process script that compute and report the desired metrics from collected data.