Deep learning approaches for vocal tract boundary segmentation in rtMRI

Recent advances in real-time Magnetic Resonance Imaging (rtMRI) provide an invaluable tool to study speech articulation. Development of automatic algorithms to detect the landmarks defining the boundaries of the vocal tract (VT) is crucial for a wide range of research, from speech modeling and synthesis to clinical research. In this thesis, we present two effective deep learning approaches for supervised detection and tracking of vocal tract contours in a sequence of rtMRI frames: (1) we propose a fully convolutional network to estimate the VT contour in heatmap regression fashion and (2) we introduce a deep temporal regression network which learns the non-linear mapping from a temporal overlapping fixed-length sequence of rtMRI frames to the corresponding articulatory movements. We as well introduce two post-processing algorithms succeeding the deep models, to further improve the quality of VT contour detection: (i) a novel appearance model based contour refinement to overcome the potential failures of data-driven approaches for highly deformable articulators and (ii) a spatiotemporal stabilization scheme to stabilize the estimated contours in space and time by removing the spatial outliers and temporal jitter. The proposed VT contour tracking models are trained and evaluated over the large audiovisual USC-TIMIT dataset. Performance evaluation is carried out using various objective assessment metrics for the spatial error and temporal stability of the contour landmarks in comparison with several baseline approaches from the recent literature. Results indicate significant improvements with the proposed methods over the state-of-the-art baselines. In addition, we develop a graphical user interface (GUI) for the analysis of the rtMRI data, integrated with various attributes including automatic segmentation of the VT boundaries using the proposed contour estimation methods and calculation of tract variables and cross-sectional distance.