Isolated sign language recognition using deep learning architectures

Sign languages are visual languages, which are the main communication medium used by hearing and speech impaired individuals in their daily life. Thanks to the computer recognition of the signs transmitted over many channels, individuals with hearing and speech disabilities will be able to communicate naturally with both other individuals and machines. In this thesis, sign language recognition was carried out through isolated sign language videos using deep learning. In the study, first experiments were carried out to determine the data augmentation and preprocessing parameters using the "general" subset of the BosphorusSign dataset. Then, as a result of experiments using various deep learning models, a suitable model for sign language recognition was determined. After that, a set of studies were conducted using different data modalities. The performances of data modalities that extracted to express various channels in sign language were evaluated by themselves and with various combinations. In this way, the most suitable data modality combination to be used for a multimodal sign language recognition has been obtained. Lastly, a multimodal sign language recognition model is proposed which uses the parameters and data modalities obtained by the experiments. The proposed model takes a total of 6 different data streams as input in RGB, joint and optical flow modalities. The features extracted from the data streams are combined and transferred to the deep learning-based classifier layers with the help of a fusion mechanism in the model. The holistic sign language recognition model, trained with an end-to-end method, provided 89.3% accuracy which is the highest performance seen in the data set used. The proposed multimodal sign language recognition model has strong potential to improve sign language recognition performance.