Sign language recognition using dino and heatmap techniques

Deaf people mainly use sign language as their primary method of communication. However, learning sign language can be challenging because it requires considerable time and effort, particularly for those with hearing ability. Sign Language Recognition (SLR) involves identifying and translating individual signs into corresponding spoken language words, known as glosses. SLR approaches help communication problems between the deaf and hearing communities. This study introduced an approach that combines self-supervised learning Distillation with No Labels (DINO) with heatmap-based 3-dimensional Convolutional Neural Network (3D CNN) cue representation techniques to improve recognition accuracy. The proposed approach uses DINO to focus on learning visual cues from hands and faces through self-supervised learning. Heatmap-based modeling is applied to extract upper body posture and movement. The approach includes the proposed CueSeqNet model for body part-specific cue processing and the proposed HeatPose2D/3D model, which applies CNNs to interpret joint and limb heatmaps. An introduced mixed modeling strategy, CueHeat2D/3D, combines these cues for improved recognition. We tested our models with the isolated Turkish SLR dataset BosphorusSign22k. The proposed model achieves 91.71% classification accuracy. Our study shows that self-supervised learning DINO with a 3D heatmap gives competitive results in isolated SLR.