Speaker-id
Hello,
First of all, thank you for creating this very useful page.
I have a question about data preparation for training the model. During training, we have a speech sample and a text transcript for each data point. From these, we obtain the speech tokens that represent the label using the s3tokenizer. Additionally, we use the s3tokenizer to create speech tokens for the speech prompt (voice cloning).
I would like to know whether, for each data point during training, you create the prompt speech token using the same speech token label, or whether you use the speaker ID instead—where each data point has the same speaker's speech token but uses a different data sample?
Hi! Theres a section on the Github page explaining in more detail, but in short, its possible to both create the prompt speech token prior to training (by using another sample/uterance from the same ID), or from the data entry itself. Important to note is, if you use the sample itself, you must mask the first X seconds of the clip, which means the audio must be longer than that.
I hope it makes sense, otherwise just reach out again 🙂