Audio-visual saliency in omnidirectional panoramic scenes

As virtual reality devices become more widespread, investigating how humans explore omnidirectional panoramic scenes has become a topic of interest. Several computational models now exist for predicting visual saliency in such environments. However, the existing literature largely ignores auditory cues while estimating saliency maps. To pave the road for more capable models, we investigate how scene attributes and sound categories influence visual exploration. In that manner, how several semantic categories for audio and places influence observer saliency and how such effects differ across different color and audio modulations are investigated by conducting an experiment and creating a novel eye-tracking dataset. Analyses revealed that when viewing 360° videos freely, observer exploration behavior completely changed across different audio categories. The presence of audio, irrespective of whether it contained spatial features, strongly guided this exploration behavior. Color information did not necessarily have a similar effect. Furthermore, the effects of task type over eye fixations were explored by conducting a psychophysics experiment via presenting participants with simple three-dimensional audio-visual stimuli across free viewing, search, and memory recall tasks where the stimuli differed by colors. Overall, analyses revealed that neither of the modulated features had any effect except color contrast levels, hinting that only the task-relevant features play a role in eye fixations in the presence of a task.