Hand Gesture Recognition Based on Combined Features Extraction

Hand gesture is an active area of research in the vision community, mainly for the purpose of sign language recognition and Human Computer Interaction. In this paper, we propose a system to recognize alphabet characters (A-Z) and numbers (0-9) in real-time from stereo color image sequences using Hidden Markov Models (HMMs). Our system is based on three main stages; automatic segmentation and preprocessing of the hand regions, feature extraction and classification. In automatic segmentation and preprocessing stage, color and 3D depth map are used to detect hands where the hand trajectory will take place in further step using Mean-shift algorithm and Kalman filter. In the feature extraction stage, 3D combined features of location, orientation and velocity with respected to Cartesian systems are used. And then, k-means clustering is employed for HMMs codeword. The final stage so-called classification, Baum- Welch algorithm is used to do a full train for HMMs parameters. The gesture of alphabets and numbers is recognized using Left-Right Banded model in conjunction with Viterbi algorithm. Experimental results demonstrate that, our system can successfully recognize hand gestures with 98.33% recognition rate.

Authors:

• Mahmoud Elmezain
• Ayoub Al-Hamadi
• Bernd Michaelis

Keywords:

• Gesture Recognition
• Computer Vision & Image Processing
• Pattern Recognition.

References:

[1] X. Deyou, A Network Approach for Hand Gesture Recognition in Virtual
Reality Driving Training System of SPG, International Conference ICPR,
pp. 519-522, 2006.
[2] M. Elmezain, A. Al-Hamadi, and B. Michaelis, Real-Time Capable
System for Hand Gesture Recognition Using Hidden Markov Models in
Stereo Color Image Sequences, The Journal of WSCG, Vol. 16, No. 1,
pp. 65-72, 2008.
[3] M. Elmezain, A. Al-Hamadi, J. Appenrodt, and B. Michaelis, A Hidden
Markov Model-Based Continuous Gesture Recognition System for
Hand Motion Trajectory, International Conference on Pattern Recognition
(ICPR) pp. 1-4, 2008.
[4] M. Elmezain, A. Al-Hamadi, and B. Michaelis, A Novel System for
Automatic Hand Gesture Spotting and Recognition in Stereo Color Image
Sequences, The Journal of WSCG, Vol. 17, No. 1, pp. 89-96, 2009.
[5] E. Holden, R. Owens, and G. Roy, Hand Movement Classification Using
Adaptive Fuzzy Expert System, The Journal of Expert Systems, Vol. 9(4),
pp. 465-480, 1996.
[6] N. Liu, and B. C. Lovell, MMX-accelerated Real-Time Hand Tracking
System, In IVCNZ, pp. 381-385, 2001.
[7] T. Nobuhiko, S. Nobutaka, and S. Yoshiaki, Extraction of Hand Features
for Recognition of Sign Language Words, In International Conference of
VI, pp. 391-398, 2002.
[8] D. Comaniciu, V. Ramesh, and P. Meer, Kernel-Based Object Tracking,
The IEEE Transactions on Pattern Analysis and Machine Intelligence
(PAMI), Vol. 25, pp. 564-577, 2003.
[9] N. P. Vassilia and G. M. Konstantinos, On Feature Extraction and
Sign Recognition for Greek Sign Language, International Conference on
Artificial Intelligence and Soft Computer, pp. 93-98, 2003.
[10] Y. Ho-Sub, S. Jung, J. B. Young, and S. Y. Hyun, Hand Gesture
Recognition using Combined Features of Location, Angle and Velocity,
Journal of Pattern Recognition, Vol. 34(7), pp. 1491-1501, 2001.
[11] L. Nianjun, C. L. Brian, J. K. Peter, and A. D. Richard Model Structure
Selection & Training Algorithms for a HMM Gesture Recognition
System,International Workshop IWFHR, pp. 100-105, 2004.
[12] D. B. Nguyen, S. Enokida, and E. Toshiaki, Real-Time Hand Tracking
and Gesture Recognition System, In GVIP Conf., pp. 362-368, 2005.
[13] D. Comaniciu, V. Ramesh, and P. Meer, Real-Time Tracking of Non-
Rigid Objects Using Mean Shift, In Conference CVPR, pp. 1-8, 2000.
[14] G. Welch, and G. Bishop, An Introduction to the Kalman Filter, In
Technical Report, University of North Carolina at Chapel Hill, pp. 95-
041, 1995.
[15] R. R. Lawrence, A Tutorial on Hidden Markov Models and Selected
Applications in Speech Recognition, Proceeding of the IEEE, Vol. 77(2),
pp. 257-286, 1989.
[16] S. Mitra, and T. Acharya, Gesture Recognition: A Survey, IEEE Transactions
on Systems, MAN, and Cybernetics, pp. 311-324, 2007.
[17] M. Elmezain, A. Al-Hamadi, S. S. Pathan, and B. Michaelis, Spatio-
Temporal Feature Extraction-Based Hand Gesture Recognition for Isolated
American Sign Language and Arabic Numbers. IEEE Symposium
on ISPA, pp. 254-259, 2009.
[18] M. Elmezain, A. Al-Hamadi, G. Krell, S. El-Etriby, and B. Michaelis,
Gesture Recognition for Alphabets from Hand Motion Trajectory Using
Hidden Markov Models, The IEEE International Symposium on Signal
Processing and Information Technology, pp. 1209-1214, 2007.
[19] T. Kanungo, D. M. Mount, N. Netanyahu, C. Piatko, R. Silverman,
and A. Y. Wu, An Efficient k-means Clustering Algorithm: Analysis and
Implementation, IEEE Transaction on PAMI, Vol. 24, pp. 881-892, 2002.
[20] C. Ding and X. He, K-means Clustering via Principal Component
Analysis, International Conference on ML, pp. 225-232, 2004.
[21] R. Niese, A. Al-Hamadi, and B. Michaelis, A Novel Method for 3D
Face Detection and Normalization, Journal of Multimedia, Vol. 2, pp.
1-12, 2007.
[22] S. Khalid, U. Ilyas, S. Sarfaraz, and A. Ajaz, ABhattacharyya Coefficient
in Correlation of Gary-Scale Objects, The Journal of Multimedia, Vol. 1,
pp. 56-61, 2006.