Stage 2 training bug (after joint training)
schnekk opened this issue · 7 comments
Hello, I tried to train LJ speech model from scratch using default config. The training is running fine with some exception below.
Traceback (most recent call last):
File "/home/tan/StyleTTS2/train_second.py", line 666, in main
loss_F0 = F.l1_loss(F0_real, F0_fake) / 10
File "/home/tan/.conda/envs/styletts/lib/python3.8/site-packages/torch/nn/functional.py", line 3230, in l1_loss
expanded_input, expanded_target = torch.broadcast_tensors(input, target)
File "/home/tan/.conda/envs/styletts/lib/python3.8/site-packages/torch/functional.py", line 75, in broadcast_tensors
return _VF.broadcast_tensors(tensors) # type: ignore[attr-defined]
RuntimeError: The size of tensor a (2560) must match the size of tensor b (640) at non-singleton dimension 1
And I got this warning after starting joint training possibly from writing audio output from model.decoder
warning: audio amplitude out of range, auto clipped.
When I look into tensorboard, the losses are diverged after starting joint training and the sample audio after joint training become white noises. How can I fix this?
Training loss
Validation loss
Training log
Sorry for the late reply. I was quite busy recently. Can I see your training config? It looks like your batch_size and batch_percentage may be the cause. Have you checked #81 as well?
Here is the config.
log_dir: "/home/tan/StyleTTS2/checkpoint/lj-test"
first_stage_path: "/home/tan/StyleTTS2/checkpoint/lj-test/first_stage.pth"
save_freq: 10
log_interval: 10
device: "cuda"
epochs_1st: 200 # number of epochs for first stage training (pre-training)
epochs_2nd: 100 # number of peochs for second stage training (joint training)
batch_size: 16
max_len: 400 # maximum number of frames
pretrained_model: ""
second_stage_load_pretrained: true # set to true if the pre-trained model is for 2nd stage
load_only_params: false # set to true if do not want to load epoch numbers and optimizer parameters
F0_path: "/home/tan/StyleTTS2/Utils/JDC/bst.t7"
ASR_config: "/home/tan/StyleTTS2/Utils/ASR/config.yml"
ASR_path: "/home/tan/StyleTTS2/Utils/ASR/epoch_00080.pth"
PLBERT_dir: '/home/tan/StyleTTS2/Utils/PLBERT'
data_params:
train_data: "/home/tan/StyleTTS2/Data/train_list.txt"
val_data: "/home/tan/StyleTTS2/Data/val_list.txt"
root_path: "/home/tan/dataset/LJSpeech-1.1/wavs_24k"
OOD_data: "/home/tan/StyleTTS2/Data/OOD_texts.txt"
min_length: 50 # sample until texts with this size are obtained for OOD texts
preprocess_params:
sr: 24000
spect_params:
n_fft: 2048
win_length: 1200
hop_length: 300
model_params:
multispeaker: false
dim_in: 64
hidden_dim: 512
max_conv_dim: 512
n_layer: 3
n_mels: 80
n_token: 178 # number of phoneme tokens
max_dur: 50 # maximum duration of a single phoneme
style_dim: 128 # style vector size
dropout: 0.2
# config for decoder
decoder:
type: 'istftnet' # either hifigan or istftnet
resblock_kernel_sizes: [3,7,11]
upsample_rates : [10, 6]
upsample_initial_channel: 512
resblock_dilation_sizes: [[1,3,5], [1,3,5], [1,3,5]]
upsample_kernel_sizes: [20, 12]
gen_istft_n_fft: 20
gen_istft_hop_size: 5
# speech language model config
slm:
model: '/home/tan/pretrained/wavlm-base-plus'
sr: 16000 # sampling rate of SLM
hidden: 768 # hidden size of SLM
nlayers: 13 # number of layers of SLM
initial_channel: 64 # initial channels of SLM discriminator head
# style diffusion model config
diffusion:
embedding_mask_proba: 0.1
# transformer config
transformer:
num_layers: 3
num_heads: 8
head_features: 64
multiplier: 2
# diffusion distribution config
dist:
sigma_data: 0.2 # placeholder for estimate_sigma_data set to false
estimate_sigma_data: true # estimate sigma_data from the current batch if set to true
mean: -3.0
std: 1.0
loss_params:
lambda_mel: 5. # mel reconstruction loss
lambda_gen: 1. # generator loss
lambda_slm: 1. # slm feature matching loss
lambda_mono: 1. # monotonic alignment loss (1st stage, TMA)
lambda_s2s: 1. # sequence-to-sequence loss (1st stage, TMA)
TMA_epoch: 50 # TMA starting epoch (1st stage)
lambda_F0: 1. # F0 reconstruction loss (2nd stage)
lambda_norm: 1. # norm reconstruction loss (2nd stage)
lambda_dur: 1. # duration loss (2nd stage)
lambda_ce: 20. # duration predictor probability output CE loss (2nd stage)
lambda_sty: 1. # style reconstruction loss (2nd stage)
lambda_diff: 1. # score matching loss (2nd stage)
diff_epoch: 20 # style diffusion starting epoch (2nd stage)
joint_epoch: 50 # joint training starting epoch (2nd stage)
optimizer_params:
lr: 0.0001 # general learning rate
bert_lr: 0.00001 # learning rate for PLBERT
ft_lr: 0.00001 # learning rate for acoustic modules
slmadv_params:
min_len: 400 # minimum length of samples
max_len: 500 # maximum length of samples
batch_percentage: 0.5 # to prevent out of memory, only use half of the original batch size
iter: 10 # update the discriminator every this iterations of generator update
thresh: 5 # gradient norm above which the gradient is scaled
scale: 0.01 # gradient scaling factor for predictors from SLM discriminators
sig: 1.5 # sigma for differentiable duration modeling
I tried adjusting batch_size to 8 and batch_percentage to 1 to match batch_size = 16 I used in stage 1 training according to #81 but still no luck, the results are the same.
these are the hardware I used
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17 Driver Version: 525.105.17 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-SXM... On | 00000000:03:00.0 Off | 0 |
| N/A 38C P0 51W / 400W | 0MiB / 40960MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA A100-SXM... On | 00000000:41:00.0 Off | 0 |
| N/A 38C P0 53W / 400W | 0MiB / 40960MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 2 NVIDIA A100-SXM... On | 00000000:81:00.0 Off | 0 |
| N/A 39C P0 52W / 400W | 0MiB / 40960MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 3 NVIDIA A100-SXM... On | 00000000:C1:00.0 Off | 0 |
| N/A 39C P0 54W / 400W | 0MiB / 40960MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
and these are the torch version I used
torch==1.11.0+cu113
torchaudio==0.11.0+cu113
Could this be the case? I'll try changing torch version to >= 2 and try training again.
@schnekk did you try with torch==2.0.1+cu117 ?
@melodyze-ai I am currently training with torch==2.1.2+cu118 and waiting for the result.
Okay, it worked now. I tried
- change torch version from
torch==1.11.0+cu113totorch==2.1.2+cu118. - use
batch_size=16at stage1 training. - use
batch_size=8andbatch_percentage=1at stage2 training.