When using multiple GPUs, the process on the other GPU will occupy a certain amount of memory on the main GPU

[gaozhiguang]

Hi , i use the code here for another program, i replace the bert here for another pretrained model, and then when i use multiple gpus for training, something like this happens:

i use the 5,7th gpu, and the process running on GPU07 occupy a 777 MB memory of GPU05, and my program will stop because the error: cuda out of memory.
What is the 777MB on GPU05, how can i fix this?