Gesture Recognition by Data Fusion of Time-of-Flight and Color Cameras
In the last years numerous applications of Human-
Computer Interaction have exploited the capabilities of Time-of-
Flight cameras for achieving more and more comfortable and precise
interactions. In particular, gesture recognition is one of the most active
fields. This work presents a new method for interacting with a virtual
object in a 3D space. Our approach is based on the fusion of depth
data, supplied by a ToF camera, with color information, supplied
by a HD webcam. The hand detection procedure does not require
any learning phase and is able to concurrently manage gestures of
two hands. The system is robust to the presence in the scene of
other objects or people, thanks to the use of the Kalman filter for
maintaining the tracking of the hands.
[1] T. Oggier, M. Lehmann, K. R., M. Schweizer, M. Richter, P. Metzler,
G. Lang, F. Lustenberger, and N. Blanc, "An all-solid-state optical range
camera for 3D real-time imaging with sub-centimeter depth resolution
(SwissRanger)," in Proceeding of SPIE Vol. 5249, 2003, pp. 634-645.
[2] A. Kolb, E. Barth, R. Koch, and R. Larsen, "Time-of-Flight Cameras
in Computer Graphics," Computer Graphics Forum, vol. 29, no. 1, pp.
141-159, 2010.
[3] P. Dondi, L. Lombardi, and M. Porta, "Human-Computer Interaction
through Time-of-Flight and RGB cameras," in Proceedings of ICIAP
2011, 16th International Conference on Image Analysis and Processing,
vol. 2. Springer, September 2011, pp. 89-98.
[4] R. Reulke, "Combination of distance data with high resolution images,"
in Proceedings of IEVM06, Image Engeeniring and Vision Metrology,
2006.
[5] S. Ghobadi, O. Loepprich, K. Hartmann, and O. Loffeld, "Hand segmentation
using 2D/3D images," in Proceedings of Image and Vision
Computing 07, December 2007, pp. 64-69.
[6] S. E. Ghobadi, O. E. Loepprich, F. Ahmadov, J. Bernshausen, K. Hartmann,
and O. Loffeld, "Real time hand based robot control using 2D/3D
images," in Proceedings of the 4th International Symposium on Advances
in Visual Computing, Part II, ser. ISVC -08. Berlin, Heidelberg:
Springer-Verlag, 2008, pp. 307-316.
[7] P. Breuer, C. Eckes, and S. Mller, "Hand gesture recognition with a
novel IR time-of-flight range camera: a pilot study," in Proceedings
of 3rd International Conference on Computer vision/computer graphics
collaboration techniques (MIRAGE-07), 2007, pp. 247-260.
[8] Z. Li and R. Jarvis, "Visual interpretation of natural pointing gestures
in 3d space for human-robot interaction," in Proceedings of Control Automation
Robotics Vision (ICARCV), 2010 11th International Conference
on, December 2010, pp. 2513-2518.
[9] A. Treskunov, S. Kim, and S. Marti, "Range camera for simple behind
display interaction," in Proceedings of MVA2011 IAPR Conference on
Machine Vision Applications, Nara, Japan, June 2011, pp. 160-163.
[10] E. Kollorz, J. Penne, J. Hornegger, and A. Barke, "Gesture recognition
with a time of flight camera," Int. J. Intell. Syst. Technol. Appl., vol. 5,
pp. 334-343, November 2008.
[11] M. B. Holte, T. B. Moeslund, and P. Fihl, "View invariant gesture
recognition using the csem swissranger sr-2 camera," Int. J. Intell. Syst.
Technol. Appl., vol. 5, pp. 295-303, November 2008.
[12] M. Van den Bergh and L. Van Gool, "Combining RGB and ToF cameras
for real-time 3D hand gesture interaction," in Applications of Computer
Vision (WACV), 2011 IEEE Workshop on, January 2011, pp. 66-72.
[13] M. Haker, M. Bhme, T. Martinetz, and E. Barth, "Deictic gestures
with a time-of-flight camera," in Proceedings of Gesture in Embodied
Communication and Human-Computer Interaction 8th International
Gesture Workshop, GW 2009, S. Kopp and I. Wachsmuth, Eds., January
2009, pp. 110-121.
[14] T. Oggier, B. Bttgen, F. Lustenberger, G. Becker, B. Regg, and A. Hodac,
"Swissranger SR3000 and first experiences based on miniaturized 3DTOF
cameras," in Proceedings, 1st Range Imaging Research Day.
Springer, September 2005, pp. 97-108.
[15] N. Haubner, U. Schwanecke, R. Drner, S. Lehmann, and J. Luderschmidt,
"Recognition of Dynamic Hand Gestures with Time-of-Flight
Cameras," in Proceedings of ITG/GI Workshop on Self-Integrating
Systems for Better Living Environments 2010 (Sensyble Workshop),
2010, pp. 33-39.
[16] S. Soutschek, J. Penne, J. Hornegger, and J. Kornhuber, "3-D gesturebased
scene navigation in medical imaging applications using timeof-
flight cameras," in Proceedings of Computer Vision and Pattern
Recognition Workshops, 2008. CVPRW -08. IEEE Computer Society
Conference on, June 2008, pp. 1-6.
[17] J. Penne, S. Soutschek, L. Fedorowicz, and J. Hornegger, "Robust
real-time 3D time-of-flight based gesture navigation," in Proceedings
of Automatic Face Gesture Recognition, 2008. FG -08. 8th IEEE
International Conference on, September 2008, pp. 1-2.
[18] P. Dondi and L. Lombardi, "Fast real-time segmentation and tracking of
multiple subjects by time-of-flight camera," in Proceedings of VISAPP
2011, 6th International Conference on Computer Vision Theory and
Applications, March 2011, pp. 582-587.
[1] T. Oggier, M. Lehmann, K. R., M. Schweizer, M. Richter, P. Metzler,
G. Lang, F. Lustenberger, and N. Blanc, "An all-solid-state optical range
camera for 3D real-time imaging with sub-centimeter depth resolution
(SwissRanger)," in Proceeding of SPIE Vol. 5249, 2003, pp. 634-645.
[2] A. Kolb, E. Barth, R. Koch, and R. Larsen, "Time-of-Flight Cameras
in Computer Graphics," Computer Graphics Forum, vol. 29, no. 1, pp.
141-159, 2010.
[3] P. Dondi, L. Lombardi, and M. Porta, "Human-Computer Interaction
through Time-of-Flight and RGB cameras," in Proceedings of ICIAP
2011, 16th International Conference on Image Analysis and Processing,
vol. 2. Springer, September 2011, pp. 89-98.
[4] R. Reulke, "Combination of distance data with high resolution images,"
in Proceedings of IEVM06, Image Engeeniring and Vision Metrology,
2006.
[5] S. Ghobadi, O. Loepprich, K. Hartmann, and O. Loffeld, "Hand segmentation
using 2D/3D images," in Proceedings of Image and Vision
Computing 07, December 2007, pp. 64-69.
[6] S. E. Ghobadi, O. E. Loepprich, F. Ahmadov, J. Bernshausen, K. Hartmann,
and O. Loffeld, "Real time hand based robot control using 2D/3D
images," in Proceedings of the 4th International Symposium on Advances
in Visual Computing, Part II, ser. ISVC -08. Berlin, Heidelberg:
Springer-Verlag, 2008, pp. 307-316.
[7] P. Breuer, C. Eckes, and S. Mller, "Hand gesture recognition with a
novel IR time-of-flight range camera: a pilot study," in Proceedings
of 3rd International Conference on Computer vision/computer graphics
collaboration techniques (MIRAGE-07), 2007, pp. 247-260.
[8] Z. Li and R. Jarvis, "Visual interpretation of natural pointing gestures
in 3d space for human-robot interaction," in Proceedings of Control Automation
Robotics Vision (ICARCV), 2010 11th International Conference
on, December 2010, pp. 2513-2518.
[9] A. Treskunov, S. Kim, and S. Marti, "Range camera for simple behind
display interaction," in Proceedings of MVA2011 IAPR Conference on
Machine Vision Applications, Nara, Japan, June 2011, pp. 160-163.
[10] E. Kollorz, J. Penne, J. Hornegger, and A. Barke, "Gesture recognition
with a time of flight camera," Int. J. Intell. Syst. Technol. Appl., vol. 5,
pp. 334-343, November 2008.
[11] M. B. Holte, T. B. Moeslund, and P. Fihl, "View invariant gesture
recognition using the csem swissranger sr-2 camera," Int. J. Intell. Syst.
Technol. Appl., vol. 5, pp. 295-303, November 2008.
[12] M. Van den Bergh and L. Van Gool, "Combining RGB and ToF cameras
for real-time 3D hand gesture interaction," in Applications of Computer
Vision (WACV), 2011 IEEE Workshop on, January 2011, pp. 66-72.
[13] M. Haker, M. Bhme, T. Martinetz, and E. Barth, "Deictic gestures
with a time-of-flight camera," in Proceedings of Gesture in Embodied
Communication and Human-Computer Interaction 8th International
Gesture Workshop, GW 2009, S. Kopp and I. Wachsmuth, Eds., January
2009, pp. 110-121.
[14] T. Oggier, B. Bttgen, F. Lustenberger, G. Becker, B. Regg, and A. Hodac,
"Swissranger SR3000 and first experiences based on miniaturized 3DTOF
cameras," in Proceedings, 1st Range Imaging Research Day.
Springer, September 2005, pp. 97-108.
[15] N. Haubner, U. Schwanecke, R. Drner, S. Lehmann, and J. Luderschmidt,
"Recognition of Dynamic Hand Gestures with Time-of-Flight
Cameras," in Proceedings of ITG/GI Workshop on Self-Integrating
Systems for Better Living Environments 2010 (Sensyble Workshop),
2010, pp. 33-39.
[16] S. Soutschek, J. Penne, J. Hornegger, and J. Kornhuber, "3-D gesturebased
scene navigation in medical imaging applications using timeof-
flight cameras," in Proceedings of Computer Vision and Pattern
Recognition Workshops, 2008. CVPRW -08. IEEE Computer Society
Conference on, June 2008, pp. 1-6.
[17] J. Penne, S. Soutschek, L. Fedorowicz, and J. Hornegger, "Robust
real-time 3D time-of-flight based gesture navigation," in Proceedings
of Automatic Face Gesture Recognition, 2008. FG -08. 8th IEEE
International Conference on, September 2008, pp. 1-2.
[18] P. Dondi and L. Lombardi, "Fast real-time segmentation and tracking of
multiple subjects by time-of-flight camera," in Proceedings of VISAPP
2011, 6th International Conference on Computer Vision Theory and
Applications, March 2011, pp. 582-587.
@article{"International Journal of Information, Control and Computer Sciences:59762", author = "Piercarlo Dondi and Luca Lombardi and Marco Porta", title = "Gesture Recognition by Data Fusion of Time-of-Flight and Color Cameras", abstract = "In the last years numerous applications of Human-
Computer Interaction have exploited the capabilities of Time-of-
Flight cameras for achieving more and more comfortable and precise
interactions. In particular, gesture recognition is one of the most active
fields. This work presents a new method for interacting with a virtual
object in a 3D space. Our approach is based on the fusion of depth
data, supplied by a ToF camera, with color information, supplied
by a HD webcam. The hand detection procedure does not require
any learning phase and is able to concurrently manage gestures of
two hands. The system is robust to the presence in the scene of
other objects or people, thanks to the use of the Kalman filter for
maintaining the tracking of the hands.", keywords = "Gesture recognition, human-computer interaction,Time-of-Flight camera.", volume = "5", number = "11", pages = "1377-6", }
