Video based automatic affect analysis of a child and a therapist during play therapy

With recent developments in machine and deep learning techniques and the advent of big data, computer vision supports many disciplines, including social sciences. Although computer vision is used in social signal processing in psychology and its sub-branches, there is a lack of studies in the field of play therapy. Play therapy is a psychotherapeutic method, and it is a convenient yet challenging field to apply automatic computer analysis techniques due to the extensive range of body poses, the existence of various play activities, and occlusions with people and objects. In this thesis, we investigate an approach to track the affective state of a child during a play therapy session with two diagonal cameras. Moreover, to differentiate the therapist and the child in the therapy room, we introduce a human tracking module. Finally, we provide a web-based affect analysis tool for the field experts to interactively visualize affect over longitudinal data. We conduct experiments on various modalities and their fusions, including text analysis, face analysis and body motion analysis during the therapy session. In this study, we used about 350 hours of therapy videos, containing two million facial expressions. Our results show that the proposed system is promising and yet still open to improvement at different stages.