Intelligible emotional voice conversion with starGAN assisted by DTW and speaker classifier

Human speech carries not only linguistic content and speaker identity but also emotional content. The ability to alter emotional colouring of speech has the potential to enable a variety of tasks such as producing affective speech for intelligent dialogue systems and guiding people in their emotional expression abilities. While performing emotional voice conversion, special focus needs to be given to both the quality of speech (e.g., naturalness, intelligibility) and the perceived emotion of the generated speech. In this study, we propose a method for converting the emotion of speech across multiple emotion categories with a single trained model. Our StarGAN-based model, enhanced by the DTW algorithm and auxiliary speaker classifier, can change the emotion of a given speech signal into 3 emotion classes: angry, happy and sad. When building our model, we determine the loss functions targeting distinct attributes of speech including the authenticity, linguistic information, speaker identity, and emotional expression. We evaluate the performance of our model through objective and subjective evaluation criteria for both audio quality and emotional content of the converted speech. The results show that our method compares favourably with the state-of-the-art method in terms of both speech quality and emotional articulateness.