MARS-KSL: A Multi-Angle Robustness Framework for Kenyan Sign Language Recognition Using View-Invariant Landmark Normalization and Hybrid LSTM- Transformer Deep Learning

Overview

Kenyan Sign Language (KSL) remains absent from AI research despite serving 600,000 Deaf Kenyans. Existing recognition systems, including those for ASL and BSL, fail under real-world conditions by assuming fixed camera angles, controlled lighting, and unoccluded signers. No multi-angle KSL dataset or robustness study under environmental degradation exists in the published literature. This study presents MARS-KSL, a framework to maintain accuracy under real-world Kenyan conditions including classrooms, hospitals, and public spaces. A structured dataset of 50 signs was collected from 8 signers across five camera angles (0°, 30°, 45°, 60°, 90°), three lighting conditions, and two background complexity levels, constituting the first multi-angle KSL dataset in the literature. Hand and body landmarks were extracted using MediaPipe and normalised through a view-invariant coordinate transformation anchored to the mid-shoulder axis, reducing inter-angle variance prior to model input. A hybrid LSTM-Transformer architecture processed the normalised sequences, with LSTM layers capturing temporal dynamics and Transformer layers weighting the most informative frames. The hybrid model achieved 86.3% recognition accuracy at frontal angle, with drops of only 4.6 and 11.5 percentage points at 45° and 90° respectively, compared to a 23.1 point drop in the LSTM-only baseline at 90°. View-invariant normalisation reduced hand landmark variance by 61.2% (t(41) = 12.83, p < 0.001, Cohen's d = 1.94). Real-time inference reached 23.4 fps on a mobile device. These results demonstrate that view-invariant landmark normalisation significantly reduces the accuracy penalty of off-axis viewing, establishing a reproducible framework for low-resource African sign language recognition systems.

Awards (1)

• Fourth Award of $600 $600

Competition history

• ISEF 2026 Robotics and Intelligent Machines · Entry ROBO043T

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.