Multi-cue skeleton-based sign language recognition

Sign languages (SL) have developed into comprehensive and natural languages that rely on multiple visual articulators to convey linguistic meaning. Their role serves as the communication medium for Deaf communities globally, facilitating rich and complex interactions among signers. Automatic Sign Language Recognition (ASLR) is crucial to lower the communication barrier between signers and non-signers. Sign languages combine manual (e.g., hand shapes and gestures) and non-manual (e.g., facial expressions) visual SL cues to express meaning. However, the asynchronous and multi-cue characteristics present unique challenges for ASLR. This dissertation addresses Isolated Sign Language Recognition (SLR) by leveraging Spatio-Temporal Graph Convolutional Networks (ST-GCNs) and LSTM-based fusion to model visual cues. Our Multi-Cue LSTM framework merges manual and non-manual visual SL cues at each time step, improving recognition on the BosphorusSign22k and AUTSL datasets. We further explore domain-driven hand graph topologies within ST-GCNs, and demonstrate that domain-guided connections among hand skeleton joints effectively capture subtle sign details, outperforming larger full-body graph topologies. Moreover, we utilize self-attention and cross-attention mechanisms to improve interpretability in single- and multi-cue scenarios. While self-attention identifies essential temporal segments within a visual SL cue, cross-attention learns the dynamic interactions between manual and non-manual cues. We analyze attention scores to show how individual articulators complementarily influence accurate predictions. Our contributions highlight the benefits of multi-cue integration, domain-driven hand skeleton configurations, and attention-based interpretability in advancing isolated SLR and improving the robustness of SLR systems.