Comparison of feature selection and extraction methods and active learning in voice based emotion recognition systems

Since humans are social beings, emotions stand at the center of their behaviors. We interact and exchange emotions with each other, and so our communication is constructed on the basis of those emotions. One of the most common ways of emotion sharing tool is the voice. Recently, thanks to the improvement in new technologies, voice-based emotion recognition becomes very popular. Basically; the extraction of emotions from audio signals is a supervised machine learning application. Classifiers are trained using extracted features from audio signals. The performance of classifiers is dependent on data preprocessing, selection of features and classifiers and the parameters of classifiers. Obtaining high performed classifiers is the main purpose in emotion recognition systems as in the all machine learning applications. That is why many researches are conducting on phases of classifiers. Data preprocessing phase is one of the most important parts of emotion recognition like all other machine learning processes. Feature extraction and selection methods are one of the important operations that affect the performance of classification in this phase. Feature selection method aims to find the features that provide more successful result among the other features, whereas feature extraction method aims to generate a new feature space or representation of features that outputs higher performance. Nowadays, collecting data and accessing this data has become very easy. In order to use these data in machine learning applications, they need to be categorized by labeling. This labeling operation is conducted by experts or oracle. But labeling of these collected data is very hard and expensive. In many problems, there are a huge amount of unlabeled data and a small amount of labeled training data. Common methods such as active learning are proposed in order to enable the obtained unlabeled data to be used as an enhancement in machine learning efficiently. Active learning is basically processed by adding samples which are selected from unlabeled data according to specified conditions, to training data iteratively and updating model using this new training data. Selecting the most informative examples is a crucial step in active learning. In literature, several methods have been proposed for this process. One of the commonly used methods is selecting the samples that classifier is least certain about it. This method is called uncertainty sampling. The uncertainty method is based on the irregularity of samples. This irregularity is computed as entropy. The more entropy sample has, the more information it contains. An active learning flow is defined as following steps; At the first step, trained model is tested on unlabeled data. The samples where the model least certain of them are determined. The selection of samples can be decided in two ways. First one is a specified threshold value of uncertainty score. The samples which are lower than this limit value are selected as uncertain samples. The second way is that samples are ordered according to uncertainty score and a specific number of samples are selected. The selected samples are the most informative ones in terms of sample diversity. These samples are labeled by asking experts or oracle. At the last stage, the trained model is updated with new labeled samples. The new model is tested on test data and success of the model is expected to increase. By doing so, the cost of labeling is decreased sharply. In this thesis, feature selection and extraction methods are first analyzed using supervised learning methods. In voice-based emotion recognition systems, acoustic features of audio signals are commonly used. These acoustic features can be subcategorized according to information they carry. Several amounts of features and different types of them can be extracted from a single signal. Features that output high performance on classification are aimed to be selected and used in the scope of this experiment. This selection is critically important in terms of accuracy and cost saving of classification. Autoencoder, Relief-F and Chi-Square are used as feature selection and extraction methods. Results of each selection methods and each acoustic feature types are analyzed in comparison and best feature selection method and feature type are proposed. For the experiments; University of Michigan Song and Speech Emotion Dataset (UMSSED) and Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) data sets were used. With the OpenSmile tool, a total of four different types of features were generated from these data sets. These feature types are; Energy and loudness, Harmonic product spectrum (HPS) pitch and voice quality, spectral and Mel frequency cepstral coefficient (MFCC). Two different features have been produced by using the different combinations of these features and in the end, total of six types of features have been obtained. The results of the study indicate that the MFCC type features and the features selected by using the Relief-F method have the most successful results. Relief-F method is based on the distance between feature and class label. The neighbors of the pointed feature are also taken into account of calculation with the class label. These distance calculations are the reason that Relief-F outputs higher performance. Chi-Square outputs close performance result with Relief-F. This method uses a statistical calculation of independence between feature and class label. Both depend on the individual relationship between a feature and class label. This common feature of methods is the reason behind this high performance of classification. At the second part of the thesis, active learning was applied with the information obtained in the first part and the accuracy results of applications on emotion recognition systems are analyzed. The experiment of this part is separated into two sub-experiments according to the usage of data sets. At first experiment, RAVDESS data set is used as training set, UMSSED data set is divided into test and unlabeled data sets. The main purpose of this experiment is to find a solution to the classification of data which contains a small amount of samples. The model which is trained by a huge amount of training data outputs higher performance using active learning. At the other experiment, UMMSED data set is used as training set, RAVDESS is used test and unlabeled data sets. This experiment analyzes the smart selection and efficient usage of samples from a huge amount of unlabeled data in emotion recognition systems. In these experiments, MFCC, Statistical and Is13ComParE types of features are used. Cross validation method is used for evaluation. In both experiments, the trained model is tested on unlabeled data set. The most uncertain samples were selected. Afterward, the labels were learned from experts and the model was updated with this information. These operations were performed iteratively, and at each iteration, a certain number of samples were added to the training set. In each iteration, the change in the performance of the model was observed and the results showed that the performance of classification was increasing. As a future work, in order to improve the performance, other instance selection algorithms can be used in active learning. To sum up, within these experiments; the effect of the feature selection and extraction methods used in the emotion recognition from audio signals is analyzed, and also active learning is applied with the obtained results in the first phase. This study emphasizes the importance of data preprocessing phase of classification and concludes successful results with a remarkable recommends that provide an insight to focus on the other phases of emotion recognition.