Dynamic 3D Facial Expression Recognition

In this study, dynamic 3D facial expression recognition is addressed by proposing novel landmark-based and appearance-based approaches. As a preliminary work, a set of geometric landmark-based features are extracted from 3D images, followed by sequential forward feature selection (SFFS) and a two-layered support vector machine (SVM), fuzzy SVM classifier to recognize six basic expressions. Experiments conducted on BU-3DFE data set proved that the proposed algorithm outperforms the conventional methods advocating the effectiveness of geometric landmark-based methods. In the second phase, a novel method using time series analysis of landmark-based geometric deformations is proposed for dynamic 3D facial expression recognition. After head pose correction and normalization, a set of multimodal time series are constructed from the local temporal deformations by applying a sliding window averaging on a comprehensive set of geometric landmark-based deformations (point, distance and angle). This stage is interlocked with facial action unit analysis to identify the key points from facial landmarks. Then, neighborhood component analysis feature selection (NCFS) is utilized to discard redundant features. Finally, adaptive cost dynamic time warping (AC-DTW) is applied to classify six prototypic expressions. Experiments on BU-4DFE data set confirmed the effectiveness of the proposed algorithm. In the third phase, an appearance-based dynamic 3D facial expression recognition is proposed using low-rank sparse codes and a novel spatiotemporal region of interest (ROIs) pooling. 12 ROIs are defined using automatically detected and tracked landmarks in by applying a multi-point tracker. LBP-TOP feature descriptors are extracted from cuboids inside spatiotemporal regions of interests in both texture and depth sequences and are fused to form the feature matrix. Sparse codes are obtained using low-rank sparse coding. Finally, hidden-state conditional random fields are employed to classify six basic expressions. Experimental results on BU-4DFE data set verified that proposed method improves the accuracy of dynamic facial expression recognition in comparison to previously proposed approaches. Keywords: Dynamic 3D facial expression recognition; Spatiotemporal analysis; Geometric landmark-based deformations; Time series analysis; Dynamic time warping; Facial landmark detection; Landmark tracking; Sparse Code; Region of interest.