Spherical vision transformers for audio-visual saliency prediction in 360◦ videos

Saliency prediction aims to model human audio-visual attention mechanisms to highlight the perceptually important regions in the scenes. This problem was first addressed in the literature under three branches based on the scene characteristics: static (for images), dynamic (for videos), and audio-visual saliency prediction. Due to the growing interest in virtual reality (VR), omnidirectional videos (ODVs) that capture the full field-of-view have gained 360◦ saliency prediction importance in computer vision. However, predicting where humans look in 360◦ scenes presents novel challenges, including the representation of 360◦ scenes regarding spherical distortion, high resolution, and the limited amount of annotated data. This thesis proposes a novel vision-transformer-based saliency prediction model named SalViT360 for omnidirectional videos. We introduce a spherical geometry-aware spatio-temporal self-attention mechanism among tangent image representations for effective omnidirectional video understanding. We present a consistency-based unsupervised regularization term for projection-based 360◦ dense-prediction models to reduce artefacts in the predictions after inverse projection. Our approach is the first to employ tangent images for undistorted omnidirectional saliency prediction. Lastly, we propose SalViT360-AV by extending our video saliency prediction model with audio-visual adapters to incorporate mono and spatial audio modalities for a unified 360◦ audiovisual saliency prediction model. Our experimental results on four ODV saliency datasets demonstrate the effectiveness of SalViT360 and SalViT360-AV compared to the state-of-the-art.