The S2ST models have evolved from Translatotron's encoder-decoder architecture to newer designs prioritizing voice conversion and style transfer. Translatotron 2 achieves antispoofing by eliminating the speaker encoder and leveraging cross-lingual TTS for training. Newer textless models utilize semantic units for translation but lose style specificity. To overcome this, recent proposals advocate for the integration of speaker and emotion encoders, while state-of-the-art acoustic unit models enhance expressivity and fidelity in translated speech, showcasing the continuous advancements in speech-to-speech translation technology.
Recent advancements in S2ST models involve removing speaker encoders and developing unit-based systems, enhancing expressivity and capturing richer acoustic and style information.
Translatotron 2 has removed the speaker encoder for antispoofing and uses a cross-lingual TTS model to create speaker-aligned data for training.
While textless models provide good semantic quality, they fall short in preserving vocal style, prompting proposals to integrate speaker and emotion encoders.
State-of-the-art acoustic unit models like EnCodec are improving S2ST through better representation of style, unlocking new potential for vocal style transfer.
Collection
[
|
...
]