Backchannel prediction in human-robot interaction for engaging agents

This thesis offers a comprehensive investigation into the realm of human-robot interaction (HRI), with a particular focus on the role of non-verbal cues such as smiles, laughs, and head nods in enhancing social engagement and naturalness. Utilizing meticulously annotated multi-modal data from naturalistic dyadic conversations, the research employs advanced machine learning techniques, including Time Delay Neural Networks (TDNNs), Support Vector Machines (SVMs), Long Short-Term Memory networks (LSTMs), and Transformer models. The work rigorously explores the utility of facial information, head movement, and audio features for the continuous detection of laughter, addressing the class imbalance problem through the effective incorporation of bagging techniques. It also delves into the impact of laughter perception and response on engagement in HRI, evaluated through objective and subjective measures in experimental setups featuring robots with laughter-responsive and non-responsive modes. Furthermore, the research presents an audio-visual prediction framework for head-nod and turn-taking events, trained and evaluated on human-human conversational datasets. A comparative approach is employed, using LSTMs as a baseline and Transformer models with cross attention mechanisms as the main proposed method, demonstrating significant improvements in the prediction performance of upcoming candidate smiles and laughs as backchannels. Collectively, these findings significantly advance our understanding of dialog management systems, offering crucial insights into the mechanics of social engagement in HRI and laying a robust foundation for future research.