Exploiting representative image patches for facial expression analysis

The human face displays expressions through the contraction of various facial muscles. The Facial Action Coding System (FACS) is a widely accepted standard that describes all visible changes in the face in terms of Action Units (AUs). In this study AUs are examined in three different ways for AU detection: (1) by highlighting the representative features of the AU, (2) by finding the most representative parts of the face and getting rid of the rest, and (3) by finding the most representative patch sizes for each AU detection task. Initially introducing Cascaded Convolutional Networks (CCNN), this study leverages Convolutional Neural Networks (CNN) to highlight the concealed features in early layers where the facial shape is not distorted. CCNN bypasses traditional landmark-based methods, acting as a strong pre-processing step that boosts AU detection rates. For the second part of this study, sparse learning is employed to learn the most active landmarks for each AU, then the active landmark patches are fed to ViT and Perceiver mechanisms (independently). Experiments indicate that using active landmark patches improves the results when compared to using all the landmarks, especially when it is used on more challenging datasets as a support for the attention mechanism to the classifier. Finally, the size of the patches is investigated, and a size-variant patch method is proposed to experiment the most appropriate size for each patch. The results demonstrate that the proposed method improves the performance of the employed models and are further supported by experiments conducted across different datasets.