REAL-TIME MALAYSIAN SIGN LANGUAGE SENTENCE-LEVEL GESTURE RECOGNITION ON ANDROID USING GRU-CTC MODEL
Loading...
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
UNIVERSITI MALAYSIA SARAWAK
Abstract
Description
The unique structure of sign language sentences, relying on transitions between gestures rather than isolated words, renders traditional word-based or fingerspelling methods inefficient for natural communication. Despite the growing need for accessibility tools, a significant gap exists in real-time recognition systems for Malaysian Sign Language (BIM), particularly at the sentence level, and in the availability of portable mobile translator applications. This paper presents a vision-based gesture recognition system for real-time, sentence-level BIM translation using a Gated Recurrent Unit (GRU) combined with Connectionist Temporal Classification (CTC) to handle temporal dependencies and variable-length sequences. The model was evaluated using 5-fold cross-validation, achieving an average WER of 0.0275 with a standard deviation of 0.0173, and a final test WER of 0.0132 after full-dataset training, confirming its strong generalization capability. Integrated into an Android mobile application, the system was tested in real-time on unseen users, achieving a 69% accuracy and average WER at 0.30, highlighting its potential for practical deployment while also revealing challenges related to inter-user variation and gesture similarity. This work contributes toward bridging the accessibility gap for the Malaysian Deaf community through a portable, vision-based real-time translation solution.