Development of an end-to-end sign language hand gesture recognition system using novel deep learning methods

Sign languages serve as the primary communication tools for individuals with hearing impairments. Developing systems to support these languages is essential due to the challenges such individuals face when interacting with people who are not proficient in sign language (SL). To address this, the present study leverages advancements in Artificial Intelligence (AI). The research involves four integrated phases to develop a Human–Computer Interaction-Based Sign Language Recognition System (HCISLRS). These phases include designing a novel graphical user interface for Sign Language System (G-SLS) with a proposed segmentation method, creating a new dataset for Turkish Sign Language Alphabet (TSLA), developing innovative learning methods, and testing the system in real time. G-SLS uses combined segmentation techniques to detect objects, specifically hands, in images. This method, employed in G-SLS, supports both dataset creation and hand gesture recognition. Following this, a dataset comprising visual samples of TSLA is compiled. The dataset is imbalanced, introducing variations in class distribution, gesture types, backgrounds, user-to-camera distances, and lighting conditions. These variations contribute to the dataset's complexity and enhance its relevance for real-world applications. Subsequently, this research also develops novel deep learning methods: TSLA networks (TSLAnets) and Vision Transformer of Turkish Sign Language Alphabet (ViTSLA). These methods demonstrate high performance across multiple evaluation metrics on various dataset classes. Moreover, a comprehensive comparison is conducted between the different developed methods, established deep learning architectures, and models from previous studies. To assess the generalization ability of the proposed methods, they are also evaluated on external datasets. This ensures that the performance is not dataset-specific but consistent across different data distributions. These comparisons affirm the effectiveness of the innovative methods, demonstrating superior performance measures, faster computation, and fewer parameters compared to other models.