Affective video summarization

With the availability of video sharing and streaming services, the media consumption behavior of users has moved from TV to the internet resulting in a surge in video content. The video summarization field produces solutions for efficient video representation, retrieval, and browsing to ease the complications caused by video content and traffic surge. Inspection of the new video content shows that user-generated human-centric video production and consumption consolidates most of the surge. Conventional video summarization neglects the human content and treats all video categories similarly. In this thesis, we argue that summarization of human-centric videos requires both understanding human behavior and video summarization. We break down this complex task into serial sub-tasks to understand complex human behavior through emotion recognition and video summarization. First, we represent the human video content by affective states and propose a multi-modal, multi-task learning-based framework estimating affective states from audio-visual input. Along with predicting affective states, we define a novel problem of detecting affective bursts to capture salient regions in the affective contour accurately. We define affective video summarization which focuses on the summarization of human-centric videos and proposes a framework that integrates affective information into the video summarization process. Finally, we presented a dataset referred to as AffWild2-VS annotated for video summarization, enabling joint research on video summarization and emotion recognition.