Alphabet and dynamic word recognition in Turkish sign language with machine learning algorithms

Computers and smart interfaces, which have become a part of social life, make life easier by using them effectively, especially in recognizing signs and movements, apart from sound and image. In this study, it has been studied within the framework of easy communication between hearing and speech impaired people who use sign language and other individuals and a person who does not know sign language can understand this language. Due to the technological trend towards portable systems today, LMC and Kinect devices have been used in sign recognition studies. The LMC device has been especially preferred because it can be embedded or directly integrated into other smart interfaces, especially computers and mobile devices, its instantaneous scanning speed and sensitivity of hand and finger movements, and its openness to development. On the other hand, Kinect was used as a support in cases where the LMC was insufficient. In this study, two-handed static finger alphabet and dynamic word recognition systems of Turkish sign language (TSL) were studied by using LMC and Kinect device. The study consists of 8 stages: preprocessing, monitoring, collection of image frames, feature extraction, feature selection and extraction, size reduction, training and testing. The study consists of four applications. The first application was carried out on static finger alphabet recognition using LMC. In the second application, a dynamic word recognition system was designed with LMC. For this application, a dataset was created by using 4 signers for 50 dynamic words prepared by considering their similarities and differences. In the third application, the dynamic word recognition system was studied using the Kinect device. The fourth application is a dynamic word recognition system with different lengths and durations using LMC+Kinect devices. The histogram and temporal feature extraction used in this application make it easier to enter the classifier by equalizing the size of the datasets, and also reduce the size of the data. From the datasets obtained from these applications, new datasets were obtained by using feature selection algorithm, feature extraction methods and PCA, LDA and PCA+LDA dimension reduction methods. Using these datasets, signal recognition performance was analyzed with traditional machine learning methods, neural network and ELM based classifiers. ELM architectures were used for the first time as a classifier in a sign language recognition system in this study. 5 different architectures of ELM, which offers robust and stable generalization ability in terms of recognition performance, and their unique learning methods were tested and the results were compared. The performance test of the TSL recognition system proposed in this study was carried out using the 10-fold cross-validation method. Based on the performance metrics obtained, it has been observed that ML-KELM, one of the ELM-based architecture and machine learning methods, maintains the performance rate in all datasets, gives the highest performance rate and has a stable structure in terms of performance.