NTT123/vietTTS

High RAM usage acoustic model

lethanhson9901 opened this issue · 3 comments

After MFA training, I've got 39778 Textgrids and 39778 wav file. The issue that when I ran acoustic trainer it cost 14GB RAM and keep increasing. How to fix this? Thank you
Screenshot from 2021-10-27 14-40-13

I've tried to decrease batch size to 16 but it's not work

There are two ways to resolve this issue:

  • use machines with more RAM. Google Colab allows users to select machine with more RAM at Runtime -> Change runtime type
  • modify the following function to prevent it loading all data to memory.

def load_textgrid_wav(data_dir: Path, token_seq_len: int, batch_size, pad_wav_len, mode: str):

How about this solution: shuffle data and divide into 2-3 part and training sequentially each part ?
Does this solution lead to worse result or not ?