Any-to-any and emotion examples

Question

Any-to-any and emotion examples

sleimanitani opened this issue a year ago · 6 comments

I'm trying to replicate your results. How did you create your any-to-any examples? Did you change the voice to text from the "source" audio and use the "reference" audio as a zero-shot reference?

Similarly, in the emotion examples, were those also zero-shot where you just used the file from ESD as a reference to create the reference embedding?

Thanks!

Answer 1 · 2023-08-07T04:38:31.000Z

Any-to-Any was voice conversion. The idea is you provide the text and use the text aligner to get the attention alignment, giving reference audio of the target speaker, and use the alignment and text to "resynthesize" the speech of the target speaker. The idea is carried to the project StyleTTS-VC: https://github.com/yl4579/StyleTTS-VC

The emotion examples are not zero-shot but seen speakers from ESD. It won't be this good if you do zeroshot unfortunately.

Answer 2 · 2023-10-12T11:35:56.000Z

Can you tell me about the zero shot examples? Were the texts already in the training dataset?

Answer 3 · 2023-10-12T15:18:36.000Z

What do you mean by text?

Answer 4 · 2023-10-17T11:37:32.000Z

I mean the text that is producing the speech, was it part of the training set (with the accompanying audio), or was it a new text that wasn't seen in the training?

Answer 5 · 2023-10-17T11:39:54.000Z

for example "The difference in the rainbow depends considerably upon the size of the drops, and the width of the colored band increases as the size of the drops increases.", and it has a "GT" audio. Were those part of the training dataset or not?

Answer 6 · 2023-10-19T06:36:44.000Z

Both the texts and speakers were unseen during training if that is what you mean. The GT audio is just a reference for you to compare the model performance and the ground truth.