Real-time recognition of turkish sign language expression using deep learning

This study focuses on the use of MediaPipe-based skeleton data extraction and deep learning models to enable the real-time recognition of Turkish Sign Language (TİD) expressions. The primary objective is to facilitate more effective communication between individuals who do not know sign language and TİD users, thereby promoting the social inclusion of hearing-impaired individuals. In this context, an original dataset consisting of 446 words was developed based on the Turkish Ministry of National Education's TİD Course Teaching Material and named TurkSign446. This dataset includes both static and dynamic signs and was constructed to be comprehensive and device-independent. Skeleton images were obtained using the MediaPipe Holistic library by capturing keypoints from the body, hands, and face. The dataset was split using the hold-out method into 70% for training, 10% for validation, and 20% for testing. On this dataset, various baseline models such as LSTM, Bi-LSTM, GRU, and CNN were trained, along with hybrid architectures including CNN+GRU, CNN+LSTM, CNN+BiLSTM, GRU+LSTM and, GRU+BiLSTM. The highest test accuracy of 97.72% was achieved with the CNN+GRU model. This model was also tested in real time with two participants—one included in the training data and one excluded. In test scenarios involving 10 repetitions, overall success rates of 85.50% and 80.26% were recorded for the seen and unseen participants, respectively. Additionally, for each word, 30-frame skeleton sequences were transformed into different visual representations using grid, optical flow, and horizontal concatenation methods. Based on these representations, ResNet18 and CBAM+ResNet18 architectures (the latter incorporating an attention mechanism) were trained. The highest recognition accuracy 97.30% was obtained using the CBAM+ResNet18 model with optical flow data. In conclusion, the TurkSign446 dataset and the deep learning-based approaches implemented in this study provide a significant foundation for real-time recognition of Turkish Sign Language. The results indicate that the system is applicable for both academic research and practical deployments. In future work, it is planned to expand the dataset and further optimize hybrid models to recognize more complex sign language expressions.