Çiz, konuş ve arat: Çok kipli bir video arama sistemi

With the increasing amount of multimedia content available on the web, the focus on video retrieval engines has been shifting from text-based systems to content-based methods that allow indexing and retrieval based on video contents. This trend has sparked a quest for efficient and effective video retrieval systems on large video collections. Most video retrieval systems rely only on hand-crafted features and manual annotations. Motion of the individual objects, the most decisive information conveyed in videos, is usually overlooked in video retrieval. From a user interaction perspective, motion can be given as a query using speech and sketch simultaneously. Speech allows easy specification of content, events and relationships, while sketching brings in spatial expressiveness. Unfortunately, we have insufficient knowledge of how sketching and speech can be used for video retrieval, because there are no existing retrieval systems that support such interaction. In this paper, we describe a Wizard-of-Oz protocol and a set of tools that we have developed to engage users in a sketch- and speech- based video retrieval task. We report how the protocol and the tools fit together to establish an ecologically valid testbed using retrieval of soccer videos as a use case scenario. Using the data collected in the studies, we developed a model capable of interpreting simultaneous speech and sketching to infer the sequence of motions described by a user. The performance results of the model suggest that the protocol and the tools together have the potential to serve as effective means for studying a wide range of multi-modal use cases. Moreover, a video retrieval system was built by integrating the multimodal interpretation model to a database back-end designed for big multimedia collections. The retrieval system was assessed through user evaluation studies. The evaluation results demonstrate that the given query interpretation mechanism and the database system make a good couple for motion-based video retrieval on big video collections.