Prosodic feature extraction and singing voice synthesis with an End-to-end Neural network model sequence - Kočí, Ondřej (2022)
Prosody is an intrinsic aspect of human speech. However, most popular speech synthesizers ignore it or only use its average representation when synthesizing artificial voices. This thesis proposes a new End-to-End model sequence prototype based on the Deep Neural Network architecture. Utilizing Mellotron (Rafael Valle et al. 2019), Tacotron (Yuxuan Wang et al. 2017), WaveGlow (Ryan Prenger et al. 2018) and other neural models, it can synthesize waveforms with prosody by extracting it from a source audio clip and applying it to synthesized audio. Unlike other models, this prototype uses a popular framework Nvidia NeMo for neural module development (Oleksii Kuchaiev et al. 2019). The prototype can synthesize singing voice clips with pitch and rhythm transfer, which allows it to preserve original characteristics of the source speaker. The prototype also works with datasets based solely on speaking utterances, and therefore does not require a singing corpus to train. This prototype serves as a proof of concept for real-time singing voice synthesis with advanced control over prosodic features supplied via a source audio implemented largely using the models provided by the Nvidia NeMo framework.