The article presents SpeechVerse, a unified architecture that integrates an audio encoder and large language models (LLMs) to improve handling of audio data. The research employs Flan-T5-XL and Mistral-7bInstruct as the primary LLMs, chosen for their instruction-following abilities. However, these models are not inherently designed for safety, prompting fine-tuning to align them towards safer outputs. The research highlights the limitations of popular LLMs like ChatGPT, which do not support audio inputs without additional tuning, showcasing the necessity for advancements in audio-LLM integration.
In our study, we introduce SpeechVerse, a unified SLM architecture consisting of an audio encoder and large language models (LLMs) to tackle limitations in LLMs handling audio data.
The Flan-T5-XL and Mistral-7bInstruct models were selected for their instruction-following capabilities, though we note their training lacks explicit safety measures.
Collection
[
|
...
]