Sound event detection on weakly labeled datasets with semi-supervised learning and fusion techniques

Sound Event Detection (SED) is the task of automatically identifying and classifying specific sound events within audio signals. It has a wide range of applications including security systems, automation systems, and audio-based user interactions. However, most of these models come with long training durations and high computational costs. This thesis aims to develop deep learning-based methods for more accurately detecting sound events and their start and end points, primarily from sound recordings obtained from daily life. Unlike similar studies, our research focuses on datasets composed mostly of weakly labeled and unlabeled sounds. To accelerate the training process, we utilize the mean teacher model, which is a technique of semi supervised learning. On the other hand, attention mechanisms enable more effective results by focusing on specific parts within sound signals and through temporal contexts and relationships. In this study, alongside the teacher-student model, the roles of self-attention and multi-head attention mechanisms in sound event detection are thoroughly examined. The effects of attention mechanisms with individual and combined features have been analyzed using low and high-level audio features such as Mel-Frequency Cepstral Coefficients (MFCC), Log Mel-Spectrogram (Log-Mel), Bidirectional Encoder Representation from Audio Transformers (BEATs), Audio Spectrogram Transformer (AST), and Pretrained Audio Neural Networks (PANNs). Our work compares and evaluates the potential of the multi-head attention mechanism, including early and late fusion techniques, against the self-attention mechanism. Our results indicate that combined features with attention mechanisms compared to individual features, significantly improving detection performance. Additionally, even higher performance was achieved by combining features using the early fusion method and integrating the multi-head attention mechanism. This study offers methods to increase the robustness and generalization performance of neural networks in scenarios where labeled training data is scarce.