Multi-GPU training

[ivancarapinha]

Hello,
Could you please specify the steps to enable multi-GPU training, please?
I set distributed_run=True in hparams.py and then set --n_gpus=2 and CUDA_VISIBLE_DEVICES=0,3 in file run.sh to select GPUs 0 and 3, respectively. I did this and the code seems to enter some kind of deadlock because it does not start training.
Thank you.

[jxzhanggg]

The use of multi-GPU training is basically the same as in https://github.com/NVIDIA/tacotron2.
First create a directory named "logs", then run
python -m multiproc train.py --output_directory=outdir --log_directory=logdir --n_gpus=2 --hparams=distributed_run=True

[agangzz]

Thanks your impressive work.

when I use multi-GPU training, such as
python -m multiproc train.py --output_directory=outdir --log_directory=logdir --n_gpus=2 --hparams=distributed_run=True

I run into the error， as shown below:

Traceback (most recent call last):
File "train.py", line 369, in
args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
File "train.py", line 234, in train
train_loader, valset, collate_fn = prepare_dataloaders(hparams)
File "train.py", line 64, in prepare_dataloaders
drop_last=True, collate_fn=collate_fn)
File "/home/test/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 189, in init
raise ValueError('sampler option is mutually exclusive with '
ValueError: sampler option is mutually exclusive with shuffle

[jxzhanggg]

Hi, as the error message says, when using multi-GPU training, you need to set up shuffle=False in dataloader.