New feature approaches based on spatial lip points in visual-based lip reading applications

As a social being, human beings often communicate with people by talking in order to meet their needs. The act of speaking takes place as a result of the joint use of both sight and hearing. While the sounds are produced during the speech, the forms of the lip can be clearly observed. Lip reading is the technique of understanding speech by analyzing the movement of the lips, face and tongue in cases where the voice is not heard or distorted. Visual speech information plays an important role in automatic lip reading, especially when the sound is distorted or inaccessible. Despite the success of audio-image-based lip reading, visual-only lip reading is a very difficult problem due to difficulties in distinguishing sounds with similar lip movements. In this study, some new attribute approaches are presented in order to increase the success rate in visual only lip reading applications. In this study, two separate data sets were used in speaker-independent and speaker-dependent prediction applications. These data sets; AVLetters2, in which 26 letters in the Latin alphabet are repeated seven (7) times by five (5) speakers, and AVDigits, in which the 10 digits 0-9 are repeated nine (9) times by six (6) speakers. First of all, the facial elements and lips are separated and the lip borders are marked with 20 points. Later, the attribute approaches based on these spatial points, named Center-Euclidean-Distance (CED), Symmetric-Euclidean-Distance (SED) and Neighbor-Points-Angles (NPA), are applied to classifiers. Finally, using the classification algorithms named K-Nearest Neighbor algorithm (KNN), Random Forest (RF), Support Vector Machine (SVM), lip reading analysis was performed from video images to determine 26 characters and 10 numbers. As a result of the analysis, the best success results were found to be 45.934% for the AVLetters2 data set with the RF-CED method and 67.407% for the AVDigits data set using the KNN-CED method. When compared to other visual-only studies on these data sets, it was seen that quite high and successful results were obtained.