LLNL/Caliper

Is it possible to get a GPU memory high watermark?

balos1 opened this issue · 3 comments

balos1 commented

... for CUDA and/or ROCm.

daboehme commented

Currently, no. I'll look into it and see if we can record this. We have support for recording HWMs from Umpire if you're using that.

jrmadsen commented

@daboehme I think you'd have to use the C APIs of nvidia-smi/rocm-smi and poll the current memory usage (potentially in a background thread), keeping track of the max. I am not aware of of any API analogous to getrusage that would provide a straightforward way to get this.

daboehme commented

@daboehme I think you'd have to use the C APIs of nvidia-smi/rocm-smi and poll the current memory usage (potentially in a background thread), keeping track of the max. I am not aware of of any API analogous to getrusage that would provide a straightforward way to get this.

Yeah, that sounds like the way to go. Thanks!