Annotation consensus networks for improving machine learning with human annotated data

In deep learning applications, particularly in tasks involving subjective judgments such as emotion recognition, medical diagnosis, or content moderation, it is common to collect annotations from multiple human annotators for the same data instances. This practice captures diverse perspectives but often introduces variations or disagreements due to differences in interpretation, expertise, or individual bias. Effectively modeling and reconciling these inconsistencies is a key challenge in building robust and reliable machine learning models. Annotations are typically aggregated using simple methods such as averaging or majority voting. However, these conventional methods do not effectively capture common patterns or collective annotator behavior, and instead dilute valuable information about shared understanding among annotators. To address this problem, this thesis introduces a novel Annotation Consensus Network (ACN) that explicitly models and leverages annotator consensus as a more reliable training signal. ACN extracts a Learned Annotation Consensus (LAC) that aligns with all human annotations, providing improved supervision for machine learning models within an end-to-end training framework. This consensus reduces label variance and enables backbone networks to achieve more accurate and stable learning. In this thesis, ACN is integrated into two widely studied tasks, video summarization and continuous emotion recognition from speech, both of which rely on human-annotated ground truth targets. Extensive experiments demonstrate that ACN significantly improves the quality of training annotations. Models trained with ACN consistently outperform those using traditional annotation aggregation methods, achieving higher accuracy, better generalization, and increased robustness to annotator variability. This thesis highlights that explicitly modeling consensus among annotators is critical for improving deep learning performance. By learning a common representation from multiple annotators, ACN effectively harnesses collective expertise, providing a strong foundation for more accurate and reliable AI systems. This approach is beneficial for any domain facing annotation disagreements, enabling models to better understand and represent common human judgments rather than isolated individual perspectives.