Hand Gesture Recognition: Sign to Voice System (S2V)

Hand gesture is one of the typical methods used in sign language for non-verbal communication. It is most commonly used by people who have hearing or speech problems to communicate among themselves or with normal people. Various sign language systems have been developed by manufacturers around the globe but they are neither flexible nor cost-effective for the end users. This paper presents a system prototype that is able to automatically recognize sign language to help normal people to communicate more effectively with the hearing or speech impaired people. The Sign to Voice system prototype, S2V, was developed using Feed Forward Neural Network for two-sequence signs detection. Different sets of universal hand gestures were captured from video camera and utilized to train the neural network for classification purpose. The experimental results have shown that neural network has achieved satisfactory result for sign-to-voice translation.

Authors:

• Oi Mean Foong
• Tan Jung Low
• Satrio Wibowo

Keywords:

• Hand gesture detection
• neural network
• signlanguage
• sequence detection.

References:

[1] Noor Saliza Mohd Salleh, Jamilin Jais, Lucyantie Mazalan, Roslan
Ismail, Salman Yussof, Azhana Ahmad, Adzly Anuar, and Dzulkifli
Mohamad, "Sign Language to Voice Recognition: Hand Detection
Techniques for Vision-Based Approach," Current Developments in
Technology-Assisted Education, FORMATEX 2006, vol. 2, pp.967-972.
[2] C. Neider, J. Kegel, D. MacLaughlin, B. Bahan, and R.G. Lee, The
syntax of American sign language. Cambridge: The MIT Press, 2000.
[3] M. J. Jones, and J. M. Rehg, "Statistical Color Models with Application
to skin Detection," International Journal of Computer Vision, Jan. 2002,
vol. 46, no.1, pp. 81-96.
[4] D. Saxe, and R. Foulds, "Automatic Face and Gesture Recognition,"
IEEE International Conference on Automatic Face and Gesture
Recognition, Sept. 1996, pp. 379-384.
[5] E.J. Ong, and R. Bowden, "A Boosted Classifier Tree for Hand Shape
Detection," Sixth IEEE International Conference on Automatic Face and
Gesture Recognition (FGR 2004), IEEE Computer Society, 2004, pp.
889-894.
[6] C.S. Lee, J.S. Kim, G. T. Park, W. Jang, and Z.N. Bien, "Implementation
of Real-time Recognition System for Continuous Korean Sign Language
(KSL) mixed with Korean Manual Alphabet (KMA)," Journal of the
Korea Institute of Telematics and Electronics, 1998, vol. 35, no.6, pp.
76-87.
[7] Gonzalez R., and R. Woods, Digital Image Processing. Addison Wesley,
1992.
[8] Boyle R., and R. Thomas, Computer Vision: A First Course. Blackwell
Scientific Publications, 1988.
[9] Davies E., Machine Vision: Theory, Algorithms and Practicalities.
Academic Press, 1990.
[10] X.L. Teng, B. Wu, W.Yu, and C.Q. Liu, "A Hand Gesture Recognition
System Based on Local Linear Embedding," Journal of Visual
Languages and Computing, vol. 16, Elsevier Ltd., 2005, pp. 442 - 454.
[11] W.W. Kong, and S.Ranganath, "Signing Exact English (SEE): Modeling
and Recognition," The Journal of Pattern Recognition Society, vol. 41,
Elsevier Ltd., 2008, pp. 1638 -1652.
[12] Y.H.Lee, and C.Y. Tsai, "Taiwan Sign Language (TSL) Recognition
based on 3D Data and Neural Networks," Expert Systems with
Applications, Elsevier Ltd., 2007, pp. 1-6.