End-to-end open vocabulary keyword search

Keyword search (KWS) solves the problem of searching written queries in spoken documents, thereby facilitating cataloging of and information retrieval from large archives of speech. Conventional keyword search entails transcribing the archive into text with an automatic speech recognition (ASR) system and then searching queries in the resulting transcriptions. Although this approach has grown into a mature and reliable technology, ASR-based cataloging is a complicated undertaking. In this dissertation, we propose a novel end-to-end neural-network-based KWS paradigm in which we replace the graph decoding and symbol matching of ASR-based systems with a simpler search method based on dot products on the outputs of a pair of encoding neural networks. In our method, documents are transformed into sequences in the output space of one encoder, and queries are projected to the same vector space by another encoder. Search is then conducted by comparing, by means of dot products, the query vector to the vectors representing frames of the document. Locations in the document with high dot product similarity to the query are returned as hits and locations with low dot products are ignored as background. We further adopt a multilingual training strategy which increases the viability of the proposed framework for KWS in languages for which low amounts of training data are available. Finally, we propose a scheme which jointly learns the KWS task of searching for text in speech with a task of searching for text in text, with the latter task allowing us to integrate unpaired text into the proposed KWS model. We conduct in-depth experiments across several languages analyzing the various properties of the proposed KWS method and showing its viability for competitive KWS performance both as an alternative and as a complement to ASR-based KWS.