Modeling and monitoring of engagement and affect in human-computer interaction systems

Social robots are becoming widely used in human-computer interaction (HCI) systems with their artificial intelligence powered skills. Modeling and monitoring of engagement and affective state in the course of social HCI set important problems. In this thesis, we investigate three aspects of these important problems that are monitoring engagement and affective state in HCI settings, and affective talking head generation for more natural HCI applications. Monitoring of engagement has been studied as a part of the joint work on constructing multimodal database of engagement in human-robot interactions (eHRI Database). The second aspect is affective state monitoring in real-time HCI settings. We construct a CNN-GRU architecture to estimate affective state attributes from the speech signal. The proposed affective state estimation model has been trained with RECOLA and CreativeIT databases and tested on the IEMOCAP database. Lastly, we investigate contributions of speech and facial landmarks in talking head generation. We find that landmark based models have better performance compared to speech based models. However, speech can be used as an additional input along with landmarks to improve the performance further.