Abstract

Understanding sign language can facilitate communication between deaf people and hearing people. Nevertheless, a considerable communication gap still exists between sign language users and those who are unfamiliar with its gestures. To address this challenge, the present study combines sensor technology with machine learning (ML) algorithms to develop an AI-enabled wearable glove (AIEWG) for real-time American Sign Language (ASL) recognition and text translation. Two MPU6050 inertial measurement units and three flex sensors were used to capture finger-bending movements, hand motions, and wrist orientations. The proposed system recognizes individual letters (A–Z) and expressions such as "hello," "help," "thank you," "me," and "please." The collected sensor readings were preprocessed for machine learning analysis to interpret static gestures. Several ML and DL models were employed, among which the Particle Swarm Optimization (PSO)-tuned Extra Trees classifier (EXT) demonstrated superior performance, with precision and F1-score values of 99.87%, an AUC-ROC of 99.9%, and recall and accuracy of 99.86%. For real-time deployment, the AIEWG captures sensor data, which are then classified using a PSO-tuned EXT model deployed on an ESP32. The trained model is embedded in the firmware as a C-compatible (.h) file to enable real-time inference in the Arduino IDE. The system operates at a sampling frequency of 100 Hz, with each gesture captured in a continuous 3-second window. The recognized gestures are converted into text and transmitted via Wi-Fi to a web server, where they are displayed through an accessible graphical user interface (GUI). This enables uninterrupted interaction between deaf users and people who are unfamiliar with sign language.

Keywords

American Sign Language, Static Gestures, Wearable Glove, Machine Learning, Graphical User Interface,

Downloads

Download data is not yet available.

References

  1. B. Jain, M. Chandna, A. Dasgupta, K. Bansal, Sign Language Recognition: Current State of Knowledge and Future Directions. In 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT) IEEE, (2023) 1–7. https://doi.org/10.1109/ICCCNT56998.2023.10307879
  2. A. Filipowska, W. Filipowski, P. Raif, M. Pieniążek, J. Bodak, P. Ferst, K. Pilarski, S. Sieciński, R.J. Doniec, J. Mieszczanin, E. Skwarek, K. Bryzik, M. Henkel, M. Grzegorzek, Machine Learning-based Gesture Recognition Glove: Design and Implementation. Sensors, 24(18), (2024) 6157. https://doi.org/10.3390/s24186157
  3. R. Tchantchane, H. Zhou, S. Zhang, G. Alici, A Review of Hand Gesture Recognition Systems Based on Noninvasive Wearable Sensors. Advanced Intelligent Systems, 5(10), (2023) 2300207. https://doi.org/10.1002/aisy.202300207
  4. İ. Umut, Ü. C. Kumdereli, Novel Wearable System to Recognize Sign Language in Real Time. Sensors, 24(14), (2024) 4613. https://doi.org/10.3390/s24144613
  5. J. Kim, S. Mastnik, E. André, EMG-based hand gesture recognition for realtime biosignal interfacing. in Proceedings of the 13th International Conference on Intelligent User Interfaces, (2008) 30–39. https://doi.org/10.1145/1378773.1378778
  6. J. Wu, L. Sun, R. Jafari, A Wearable System for Recognizing American Sign Language in Real-Time Using IMU and Surface EMG Sensors. IEEE Journal of Biomedical and Health Informatics, 20(5), (2016) 1281-1290. https://doi.org/10.1109/JBHI.2016.2598302
  7. B.G. Lee, S.M. Lee, Smart Wearable Hand Device for Sign Language Interpretation System with Sensors Fusion. IEEE Sensors Journal, 18(3), (2018) 1224-1232. https://doi.org/10.1109/JSEN.2017.2779466
  8. P.N. Huu, T. Phung Ngoc, Hand Gesture Recognition Algorithm Using SVM and HOG Model for Control of Robotic System. Journal of Robotics, 2021(1), (2021) 3986497. https://doi.org/10.1155/2021/3986497
  9. A. Kasapbaşi, A.E.A. Elbushra, O. Al-Hardanee, A. Yilmaz, DeepASLR: A CNN based human computer interface for American Sign Language Recognition for Hearing-Impaired Individuals. Computer Methods and Programs in Biomedicine Update, 2, (2022) 100048. https://doi.org/10.1016/j.cmpbup.2021.100048
  10. H. Alsolai, L. Alsolai, F.N. Al-Wesabi, M. Othman, M. Rizwanullah, A.A. Abdelmageed, Automated Sign Language Detection and Classification using Reptile Search Algorithm with Hybrid Deep Learning. Heliyon, 10(1), (2024) e23252. https://doi.org/10.1016/j.heliyon.2023.e23252
  11. T.H. Noor, A. Noor, A.F. Alharbi, A. Faisal, R. Alrashidi, A.S. Alsaedi, G. Alharbi, T. Alsanoosy, A. Alsaeedi, Real-Time Arabic Sign Language Recognition Using a Hybrid Deep Learning Model. Sensors, 24(11), (2024) 3683. https://doi.org/10.3390/s24113683
  12. I.A. Adeyanju, O.O. Bello, M.A. Adegboye, Machine Learning Methods for Sign Language Recognition: A Critical Review and Analysis. Intelligent Systems with Applications, 12, (2021) 200056. https://doi.org/10.1016/j.iswa.2021.200056
  13. J. Qi, L. Ma, Z. Cui, Y. Yu, Computer Vision-based Hand Gesture Recognition for Human-Robot Interaction: A Review. Complex & Intelligent Systems, 10(1), (2024) 1581-1606. https://doi.org/10.1007/s40747-023-01173-6
  14. M.M. Rahman, A. Uzzaman, F. Khatun, M. Aktaruzzaman, N. Siddique, A Comparative Study of Advanced Technologies and Methods in Hand Gesture Analysis and Recognition Systems. Expert Systems with Applications, 266, (2025) 125929. https://doi.org/10.1016/j.eswa.2024.125929
  15. M. Alaghband, H.R. Maghroor, I. Garibay, A Survey on Sign Language Literature. Machine Learning with Applications, 14, (2023) 100504. https://doi.org/10.1016/j.mlwa.2023.100504
  16. R. Kaur, G. Karmakar, F. Xia, M. Imran, Deep Learning: Survey of Environmental and Camera Impacts on Internet of Things Images. Artificial Intelligence Review, 56(9), (2023) 9605-9638. https://doi.org/10.1007/s10462-023-10405-7
  17. K.R. Pyun, K. Kwon, M.J. Yoo, K.K. Kim, D. Gong, W.H. Yeo, S. Han, H. Ko, Machine-Learned Wearable Sensors for Real-Time Hand-Motion Recognition: Toward Practical Applications. National Science Review, 11(2), (2024) nwad298. https://doi.org/10.1093/nsr/nwad298
  18. Y. Liu, X. Jiang, X. Yu, H. Ye, C. Ma, W. Wang, Y. Hu, A Wearable System for Sign Language Recognition Enabled by a Convolutional Neural Network. Nano Energy, 116, (2023) 108767. https://doi.org/10.1016/j.nanoen.2023.108767
  19. M. Daviran, A. Maghsoudi, R. Ghezelbash, Optimized AI-MPM: Application of PSO for Tuning the Hyperparameters of SVM and RF Algorithms. Computers & Geosciences, 195, (2025) 105785. https://doi.org/10.1016/j.cageo.2024.105785
  20. M.A.A. Faisal, F.F. Abir, M.U. Ahmed, M.A.R. Ahad, Exploiting Domain Transformation and Deep Learning for Hand Gesture Recognition using a Low-Cost Dataglove. Scientific Reports, 12(1), (2022) 21446. https://doi.org/10.1038/s41598-022-25108-2
  21. F. Wen, Z. Zhang, T. He, C. Lee, AI Enabled Sign Language Recognition and VR Space Bidirectional Communication Using Triboelectric Smart Glove. Nature communications, 12(1), (2021) 5378. https://doi.org/10.1038/s41467-021-25637-w
  22. S. Maharjan, S. Shrestha, S. Fernando, (2024) Sign Language Detection and Translation Using Smart Glove. In Innovative Computing and Communications, A.E. Hassanien, S. Anand, A. Jaiswal, P. Kumar, Eds., Singapore: Springer Nature Singapore, 1021, 535–554. https://doi.org/10.1007/978-981-97-3591-4_41
  23. T. Kanwal, S. Altaf, R.M. Yousaf, K. Sattar, A Smart Glove-Based System for Dynamic Sign Language Translation Using LSTM Networks. Engineering Proceedings, 118(1), (2025) 45. https://doi.org/10.3390/ECSA-12-26530