Konuşma bürününden duygu yüklü insan beden jestleri çok kipli analizi ve sentezi

Conversational agents become more realistic as they utilize affective non-verbal communication, such as the use of gestures, in human computer interaction. Synthesis and animation of gestures to accompany affective verbal communication can help to create more naturalistic conversational agents. In human-to-human communication, speech signal carries rich emotional cues, which are further emphasized by affect-expressive gestures. Speech-driven gesture synthesis can map emotional cues of the speech signal into affect-expressive gestures by modeling complex variability and timing relationships of the joint articulation of speech and gesture. In this thesis, we first introduce a framework for joint analysis of speech prosody and human body gestures towards automatic synthesis and realistic animation of beat gestures from speech prosody and rhythm. Later, we investigate the use of continuous affect attributes, which are activation, valence and dominance, in speech-driven affective synthesis and animation of gestures. We present a statistical framework for multimodal analysis of gesture, speech and affect based on the hidden semi-Markov models, where gestures are representing the states, and speech-prosody and continuous affect attributes are representing the observations of the model. Different structures of the statistical model are evaluated for synthesis and animation of affect-expressive gestures given speech and affect attributes. Evaluations are performed over two multimodal datasets in speaker-dependent and independent settings. Among different statistical structures, the conditional structure, which models observation distributions as prosody given affect, achieved the best performance with objective evaluations for the gesture synthesis and with subjective evaluations for the gesture animation.