Interpreting sign language images into Text-Based using YOLOv7

Abstrak

Sign Language recognition remains a significant challenge due to variations in hand gestures, occlusions, and environmental factors. This study introduces a YOLOv7-based sign language recognition system designed to interpret American Sign Language (ASL) images into text in real time, enhancing communication accessibility for individuals with hearing and speech impairments. The primary objectives are to develop an accurate detection model, improve real-time gesture interpretation, and optimize system performance for accessibility. The study follows the CRISP-DM (Cross-Industry Standard Process for Data Mining) methodology, including data collection, preprocessing, model training, evaluation, and deployment. The system was developed using YOLOv7, Python, and OpenCV, with an ASL dataset sourced from Roboflow and formatted in YOLOv7 PyTorch. Image preprocessing techniques such as normalization, resizing, and data augmentation were applied to enhance detection accuracy. The application integrates a real-time gesture recognition system, where detected ASL signs are instantly translated into text. Results demonstrate high detection accuracy for most ASL letters, but certain gestures such as J, Z, S, and T posed challenges due to similar hand shapes and motion-based characteristics. Optimization efforts included dataset expansion, refined annotations, and hyperparameter tuning to improve model precision. The system significantly enhances real-time ASL recognition, offering a scalable, AI-powered assistive tool for the deaf and hard of hearing community. This research contributes to machine learning-driven accessibility solutions, bridging the communication gap between sign language users and non-signers.