Konuşma ile sürülen üst beden hareketlerinin analizi ve sentezi

In this thesis we present a new computational model for natural and believable upper-body gesture synthesis in synchrony with speech using statistical learning techniques over multimodal gesticulation data. The framework consists of four main tasks for: i) unimodal clustering of gesture and intonational phrases, ii) multimodal analysis of gesture and intonational phrases, iii) speech driven gesture synthesis, and iv) gesture animation. The first task consists of unimodal analysis of speech and upper body motion to learn temporal patterns of gesture and speech prosody. Body motion features, which are extracted from multi-channel synchronous video recordings, are used to define gesture phrases with a semi-supervised temporal clustering scheme. On the other hand prosody features, which are extracted from speech input, are used to define intonational phrases with an unsupervised temporal clustering scheme. The second task performs multimodal analysis to learn dependencies between gesture and intonational phrases by utilizing a hidden semi-Markov model (HSMM). Third, we perform gesture synthesis, that is extraction of gesture sequence and gesture durations, given the speech input. The final task is to perform gesture animation, where the synthesized gesture sequence is mapped into body motion sequences to maintain a natural looking animation. The performance of the proposed speech driven gesture synthesis system is tested over our MVGL-MUB Database. Experimental results demonstrate that our system is able to properly discover audiovisual correlations between speech and gesture thus it can synthesize realistic and natural body gestures along with 3D human model animation.