Development of a system to use IMU and sEMG sensor data in communication with hearing impaired individuals

This study was planned considering that information technologies can be used effectively to enable hearing impaired individuals to communicate with other members of the society. For this purpose, a unique software that enables the recognition of words in Sign Language (SL) with two arms, which can be used without reducing the quality of life of the person, has been realized. Sign language recognition systems using different technologies (such as cameras, armbands and gloves) have been developed in the literature. However, most of the studies remain in theory rather than being a practical solution that hearing impaired individuals can use in daily life. The developed system uses surface electromyography (sEMG) and inertial measurement unit (IMU) data of two arms instead of typical vision-based systems. A software that can run on different operating systems (Android, Windows) using digital signal processing and machine learning methods specific to the study has been developed. Instead of using the Myo Armband SDK, the software uses its own Bluetooth services for better performance and does not require a device-specific dongle. It enables smoother and real-time communication by detecting motion instead of a fixed time for each signal. For the test, a dataset consisting of 80 words determined by considering the frequency of use in daily life was created. A comprehensive feature extraction process was performed for the proposed system. The recognition performance is tested with different classifiers and parameters and the results are compared. To test the validity of the proposed system and the accuracy of the obtained results, 10-fold cross validation method is used. As a result of the tests of training with different classification methods, RandomForest and Ibk algorithms achieved the highest success with 99.88% accuracy, while J48 algorithm achieved the lowest success with 89.75% accuracy. The study presented to the focus group was found comfortable, aesthetic and usable by all participants. It is thought that the system developed with this study will provide a great convenience in the lives of disabled individuals.