Audio Samples from "Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness"
Abstract: Recent advancements in Natural Language Processing (NLP) have seen Large-scale Language Models (LLMs) excel at producing high-quality text for various purposes. Notably, in Text-To-Speech (TTS) systems, the integration of BERT for semantic token generation has underscored the importance of semantic content in producing coherent speech outputs. Despite this, the specific utility of LLMs in enhancing TTS synthesis remains considerably limited. This research introduces an innovative approach, Llama-VITS, which enhances TTS synthesis by enriching the semantic content of text using LLM. Llama-VITS integrates semantic embeddings from Llama2 with the VITS model, a leading end-to-end TTS framework. By leveraging Llama2 for the primary speech synthesis process, our experiments demonstrate that Llama-VITS matches the naturalness of the original VITS (ORI-VITS) and those incorporate BERT (BERT-VITS), on the LJSpeech dataset, a substantial collection of neutral, clear speech. Moreover, our method significantly enhances emotive expressiveness on the EmoV_DB_bea_sem dataset, a curated selection of emotionally consistent speech from the EmoV_DB dataset, highlighting its potential to generate emotive speech.
the forms of printed letters should be beautiful and that their arrangement on the page should be reasonable and a help to the shapeliness of the letters themselves.
have now come into general use and are obviously a great improvement on the ordinary modern style in use in England, which is in fact the Bodoni type.
It was hoped that their rulers would hire accommodation in the county prisons, and that the inferior establishments would in course of time disappear.
The disgraceful overcrowding had been partially ended. But the same evils of indiscriminate association were still present. There was the old neglect of decency.
wherefore let him that thinketh he standeth, take heed lest he fall, and was full of the most pointed allusions to the culprit.
Ground Truth
ORI-VITS
BERT-VITS w/ [CLS]
BERT-VITS w/ [BERT_TEX]
BERT-VITS w/ [BERT_PHO]
Llama-VITS w/ [AVE]
Llama-VITS w/ [PCA]
Llama-VITS w/ [LAST]
Llama-VITS w/ [EIS_Word]
Llama-VITS w/ [EIS_Sentence]
Llama-VITS w/ [TEX]
Llama-VITS w/ [PHO]
1-hour LJSpeech
Model
the forms of printed letters should be beautiful and that their arrangement on the page should be reasonable and a help to the shapeliness of the letters themselves.
have now come into general use and are obviously a great improvement on the ordinary modern style in use in England, which is in fact the Bodoni type.
It was hoped that their rulers would hire accommodation in the county prisons, and that the inferior establishments would in course of time disappear.
The disgraceful overcrowding had been partially ended. But the same evils of indiscriminate association were still present. There was the old neglect of decency.
wherefore let him that thinketh he standeth, take heed lest he fall, and was full of the most pointed allusions to the culprit.
Ground Truth
ORI-VITS
BERT-VITS w/ [CLS]
BERT-VITS w/ [BERT_TEX]
BERT-VITS w/ [BERT_PHO]
Llama-VITS w/ [AVE]
Llama-VITS w/ [PCA]
Llama-VITS w/ [LAST]
Llama-VITS w/ [EIS_Word]
Llama-VITS w/ [EIS_Sentence]
Llama-VITS w/ [TEX]
Llama-VITS w/ [PHO]
EmoV_DB_bea_sem
Model
He would first hunt up Gregson and begin his work there.
Close beside him gleamed the white fangs of the wolf-dog.
Her words sent a strange chill through Philip.
It was his intention to return to Eileen and her father.
