Unsupervised affective state learning from speech

The conventional paradigm for estimating continuous emotional states from speech, investigated as a regression problem over time, has been widely acknowledged. This thesis introduces a novel methodology that transposes this challenge into the classification domain by learning clusters of affect contours of uniform lengths. Our approach involves a novel joint clustering and classification scheme, wherein each iteration involves clustering affect contours into independent classes. We seek to classify these classes, identifying distinct clusters with observed intra-class sample similarities. The classification structure integrates an audio feature extractor based on a Wav2Vec 2.0 model, followed by a convolutional neural network (CNN). Concurrently, the clustering component processes a segment of the affect contour, employing a convolutional network for dimensionality reduction and subsequent application of k-means clustering. The classification network predicts these generated clusters. The cumulative loss is then propagated to neural networks for weight updates. Empirical findings reveal that the obtained clusters exhibit distinctive and insightful characteristics. Simultaneously, incorporating a regression head into the trained classification network yields competitive audio-only performance on the RECOLA and USC CreativeIT datasets regarding continuous emotion recognition (CER). The results for CER are compared against baselines and existing literature, illustrating the efficacy of our approach. Our results demonstrate that while achieving competitive continuous emotion recognition performance, our approach, fundamentally a classification framework, converts the nature of the well-studied continuous regression problem.