Python API - use_cuda option

[Chris-Charisis]

Hello! In Python, when I am initializing a Summarizer with one of your pretrained models:

summarizer = Summarizer(model_name_or_path="pszemraj/long-t5-tglobal-base-16384-booksum-V12",use_cuda=True)

I am seeing the following output message:

INFO Loaded model pszemraj/long-t5-tglobal-base-16384-booksum-V12 to cpu

However, I have an Nvidia GPU available, that doesn't seem to be utilized. As far as I know, it is available for use, because I am running the following code to check it:
import GPUtil
print(GPUtil.getAvailable())
print(GPUtil.showUtilization())

and the result I am getting is:

[0]
| ID | GPU | MEM |
------------------
|  0 |  5% | 11% |
None

which means based on the GPUtil documentation that the GPU with ID 0 is available for use.

Is there a way to declare that the Summarizer must run on GPU instead of CPU, as it currently happens by default? The use_cuda option doesn't seem to do the trick.

Thank you in advance!

[pszemraj]

Hi! very strange issue, thanks for bringing this to my attention - will look into this

In the interim and to help debug, the the transformers model object is accessible as an attr of the Summarizer class object (see here) so it should be fairly straightforward to force it to cuda:

from textsum.summarize import Summarizer

summarizer = Summarizer()
print(f"the model was initialized on:\t{summarizer.model.device}")
summarizer.model = summarizer.model.to('cuda')
print(f"after to(), model is now on:\t{summarizer.model.device}")

For what it's worth, this gives me:

the model was initialized on:	cuda:0
after to(), model is now on:	cuda:0

let me know if that helps 👍

[Chris-Charisis]

Thanks for your quick answer! The initial problem was in the pytorch installation, affecting the whole CUDA running of the model. I would suggest to add in your installation guide an explicit part instructing to install pytorch (a specific or the latest version) to avoid any such issues. Keep up with your good work!