Affective Robots: Evaluation of Automatic Emotion Recognition Approaches on a Humanoid Robot towards Emotionally Intelligent Machines
One of the main aims of current social robotic research
is to improve the robots’ abilities to interact with humans. In order
to achieve an interaction similar to that among humans, robots
should be able to communicate in an intuitive and natural way
and appropriately interpret human affects during social interactions.
Similarly to how humans are able to recognize emotions in other
humans, machines are capable of extracting information from the
various ways humans convey emotions—including facial expression,
speech, gesture or text—and using this information for improved
human computer interaction. This can be described as Affective
Computing, an interdisciplinary field that expands into otherwise
unrelated fields like psychology and cognitive science and involves
the research and development of systems that can recognize and
interpret human affects. To leverage these emotional capabilities
by embedding them in humanoid robots is the foundation of
the concept Affective Robots, which has the objective of making
robots capable of sensing the user’s current mood and personality
traits and adapt their behavior in the most appropriate manner
based on that. In this paper, the emotion recognition capabilities
of the humanoid robot Pepper are experimentally explored, based
on the facial expressions for the so-called basic emotions, as
well as how it performs in contrast to other state-of-the-art
approaches with both expression databases compiled in academic
environments and real subjects showing posed expressions as well
as spontaneous emotional reactions. The experiments’ results show
that the detection accuracy amongst the evaluated approaches differs
substantially. The introduced experiments offer a general structure
and approach for conducting such experimental evaluations. The
paper further suggests that the most meaningful results are obtained
by conducting experiments with real subjects expressing the emotions
as spontaneous reactions.
[1] Mayer JD, Salovey P, Caruso DR, Mayer Salovey Caruso Emotional
Intelligence Test (MSCEIT) users manual, 2.0. Toronto, Canada: MHS
Publishers, 2002.
[2] J. Liu, A. Harris, N. Kanwisher, Stages of processing in face perception:
an meg study, Nat Neurosci, vol. 5, pp. 910916, 09 2002.
[3] Klaus R. Scherer, Mayer Salovey Caruso Emotional Intelligence Test
(MSCEIT) users manual, v. 44, 695-729 Social Science Information,
2005.
[4] E. Kennedy-Moore, J. Watson, Expressing Emotion: Myths, Realities, and
Therapeutic Strategies. Emotions and social behavior, Guilford Press,
1999.
[5] P. Ekman, Universals and Cultural Differences in Facial Expressions of
Emotion. University of Nebraska Press, 1971.
[6] P. Ekman, W. V. Friesen, and J. C. Hager, The facial action coding system,
in Research Nexus eBook, 2002.
[7] P. Lucey, J. F. Cohn, T. Kanade, J. M. Saragih, Z. Ambadar, and I. A.
Matthews, The extended cohn-kanade dataset (CK+): A complete dataset
for action unit and emotion-specified expression, in IEEE Conference on
Computer Vision and Pattern Recognition, CVPR Workshops 2010, San
Francisco, CA, USA, 13-18 June, 2010, pp. 94–101, 2010.
[8] Challenges in representation learning: Facial expression recognition
challenge, https://www.kaggle.com/c/challenges-in-representationlearning-
facial-expression-recognition-challenge (Last accessed: in April
2018)
[9] M. J. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, Coding facial
expressions with gabor wavelets in 3rd International Conference on Face
& Gesture Recognition (FG’98), Nara, Japan, pp. 200–205, 1998.
[10] The Third Emotion Recognition in The Wild (EmotiW) 2015 Grand
Challenge, http://cs.anu.edu.au/few/emotiw2015.html (Last accessed:
April 2018)
[11] Z. Yu and C. Zhang, Image based static facial expression recognition
with multiple deep network learning in ICMI’ 15 Proceedings of the 2015
ACM on International Conference on Multimodal Interaction, Seattle,
WA, USA, pp. 435–442, 2015.
[12] B.-K. Kim, J. Roh, S.-Y. Dong, and S.-Y. Lee, Hierarchical committee
of deep convolutional neural networks for robust facial expression
recognition in J. Multimodal User Interfaces, vol. 10, no. 2, pp. 173–189,
2016.
[13] G. Levi and T. Hassner, Emotion recognition in the wild via
convolutional neural networks and mapped binary patterns in ICMI’ 15
Proceedings of the 2015 ACM on International Conference on Multimodal
Interaction, Seattle, WA, USA, pp. 503–510, 2015.
[14] Y. Lv, Z. Feng, and C. Xu, Facial expression recognition via deep
learning, in SMARTCOMP, IEEE Computer Society, pp. 303–308, 2014.
[15] T. Ahsan, T. Jabid, and U.-P. Chong, Facial expression recognition using
local transitional pattern on gabor filtered facial images, IETE Technical
Review, vol. 30, no. 1, pp. 47–52, 2013.
[16] A. Gudi, Recognizing semantic features in faces using deep learning,
CoRR, vol. abs/1512.00743, 2015.
[17] E. Correa, A. Jonker, M. Ozo, and R. Stolk, Emotion recognition using
deep convolutional neural networks., 2016.
[18] M. Hashemian, H. Moradi, and M. S. Mirian, How is his/her mood:
A question that a companion robot may be able to answer, in Social
Robotics: 8th International Conference, ICSR 2016, Kansas City, MO,
USA, November 1-3, 2016 Proceedings (A. Agah, J.-J. Cabibihan, A. M.
Howard, M. A. Salichs, and H. He, eds.), pp. 274–284, Springer
International Publishing, 2016.
[19] M. M. A. de Graaf, S. Ben Allouch, and J. A. G. M. van Dijk,
What makes robots social?: A user’s perspective on characteristics
for social human-robot interaction, in Proceedings of Social Robotics:
7th International Conference, ICSR 2015, Paris, France, pp. 184–193,
Springer International Publishing, 2015.
[20] A. Meghdari, M. Alemi, A. G. Pour, and A. Taheri, Spontaneous
human-robot emotional interaction through facial expressions, in Social
Robotics: 8th International Conference, ICSR 2016, Kansas City, MO,
USA, November 1-3, 2016 Proceedings (A. Agah, J.-J. Cabibihan, A. M.
Howard, M. A. Salichs, and H. He, eds.), (Cham), pp. 351–361, Springer
International Publishing, 2016.
[21] U. Hess and R. E. Kleck, Differentiating emotion elicited and deliberate
emotional facial expressions, European Journal of Social Psychology,
vol. 20, no. 5, pp. 369–385, 1990.
[22] M. Hirose, T. Takenaka, H. Gomi and N. Ozawa, Humanoid robot,
Journal of the Robotics Society of Japan, vol. 15, no. 7, pp. 983–985,
1997.
[23] K. Hirai, M. Hirose, Y. Haikawa and T. Takenaka, The Honda
humanoid robot: development and future perspective, Industrial Robot:
An International Journal, vol. 26, no. 4, pp. 260–266, 1999.
[24] P. Ekman, J.C. Hager, W.V. Friesen, The symmetry of emotional and
deliberate facial actions, Psychophysiology, 18: 101-106, 1981.
[25] A. Schaefer, F. Nils, X. Sanchez, and P. Philippot, Assessing the
effectiveness of a large database of emotion-eliciting films: A new
tool for emotion researchers, Cognition and Emotion, vol. 24, no. 7,
pp. 1153–1172, 2010.
[26] Aldebaran (Softbank Robotics), Pepper robot,
https://www.ald.softbankrobotics.com/en/robots/pepper (Last accessed:
April 2018)
[27] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin,
S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga,
S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden,
M. Wicke, Y. Yu, and X. Zheng, Tensorflow: A system for large-scale
machine learning in 12th USENIX Symposium on Operating Systems
Design and Implementation (OSDI 16), pp. 265–283, 2016. [28] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, ImageNet:
A Large-Scale Hierarchical Image Database in IEEE Computer Vision
and Pattern Recognition (CVPR), 2009.
[29] Softbank Robotics, ALMood Module,
http://doc.aldebaran.com/2-4/naoqi/core/almood.html (Last accessed:
April 2018)
[30] OMRON Corporation, Facial Expression Estimation Technology,
https://www.omron.com/media/press/2012/10/e1023.html (Last accessed:
April 2018)
[31] Google Inc., Cloud Vision API, https://cloud.google.com/vision/ (Last
accessed: April 2018)
[32] Microsoft Corporation, Emotion API,
https://azure.microsoft.com/en-us/services/cognitive-services/emotion/
(Last accessed: April 2018)
[33] Kairos AR, Inc.,Human Analytics, https://www.kairos.com/features (Last
accessed: April 2018)
[1] Mayer JD, Salovey P, Caruso DR, Mayer Salovey Caruso Emotional
Intelligence Test (MSCEIT) users manual, 2.0. Toronto, Canada: MHS
Publishers, 2002.
[2] J. Liu, A. Harris, N. Kanwisher, Stages of processing in face perception:
an meg study, Nat Neurosci, vol. 5, pp. 910916, 09 2002.
[3] Klaus R. Scherer, Mayer Salovey Caruso Emotional Intelligence Test
(MSCEIT) users manual, v. 44, 695-729 Social Science Information,
2005.
[4] E. Kennedy-Moore, J. Watson, Expressing Emotion: Myths, Realities, and
Therapeutic Strategies. Emotions and social behavior, Guilford Press,
1999.
[5] P. Ekman, Universals and Cultural Differences in Facial Expressions of
Emotion. University of Nebraska Press, 1971.
[6] P. Ekman, W. V. Friesen, and J. C. Hager, The facial action coding system,
in Research Nexus eBook, 2002.
[7] P. Lucey, J. F. Cohn, T. Kanade, J. M. Saragih, Z. Ambadar, and I. A.
Matthews, The extended cohn-kanade dataset (CK+): A complete dataset
for action unit and emotion-specified expression, in IEEE Conference on
Computer Vision and Pattern Recognition, CVPR Workshops 2010, San
Francisco, CA, USA, 13-18 June, 2010, pp. 94–101, 2010.
[8] Challenges in representation learning: Facial expression recognition
challenge, https://www.kaggle.com/c/challenges-in-representationlearning-
facial-expression-recognition-challenge (Last accessed: in April
2018)
[9] M. J. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, Coding facial
expressions with gabor wavelets in 3rd International Conference on Face
& Gesture Recognition (FG’98), Nara, Japan, pp. 200–205, 1998.
[10] The Third Emotion Recognition in The Wild (EmotiW) 2015 Grand
Challenge, http://cs.anu.edu.au/few/emotiw2015.html (Last accessed:
April 2018)
[11] Z. Yu and C. Zhang, Image based static facial expression recognition
with multiple deep network learning in ICMI’ 15 Proceedings of the 2015
ACM on International Conference on Multimodal Interaction, Seattle,
WA, USA, pp. 435–442, 2015.
[12] B.-K. Kim, J. Roh, S.-Y. Dong, and S.-Y. Lee, Hierarchical committee
of deep convolutional neural networks for robust facial expression
recognition in J. Multimodal User Interfaces, vol. 10, no. 2, pp. 173–189,
2016.
[13] G. Levi and T. Hassner, Emotion recognition in the wild via
convolutional neural networks and mapped binary patterns in ICMI’ 15
Proceedings of the 2015 ACM on International Conference on Multimodal
Interaction, Seattle, WA, USA, pp. 503–510, 2015.
[14] Y. Lv, Z. Feng, and C. Xu, Facial expression recognition via deep
learning, in SMARTCOMP, IEEE Computer Society, pp. 303–308, 2014.
[15] T. Ahsan, T. Jabid, and U.-P. Chong, Facial expression recognition using
local transitional pattern on gabor filtered facial images, IETE Technical
Review, vol. 30, no. 1, pp. 47–52, 2013.
[16] A. Gudi, Recognizing semantic features in faces using deep learning,
CoRR, vol. abs/1512.00743, 2015.
[17] E. Correa, A. Jonker, M. Ozo, and R. Stolk, Emotion recognition using
deep convolutional neural networks., 2016.
[18] M. Hashemian, H. Moradi, and M. S. Mirian, How is his/her mood:
A question that a companion robot may be able to answer, in Social
Robotics: 8th International Conference, ICSR 2016, Kansas City, MO,
USA, November 1-3, 2016 Proceedings (A. Agah, J.-J. Cabibihan, A. M.
Howard, M. A. Salichs, and H. He, eds.), pp. 274–284, Springer
International Publishing, 2016.
[19] M. M. A. de Graaf, S. Ben Allouch, and J. A. G. M. van Dijk,
What makes robots social?: A user’s perspective on characteristics
for social human-robot interaction, in Proceedings of Social Robotics:
7th International Conference, ICSR 2015, Paris, France, pp. 184–193,
Springer International Publishing, 2015.
[20] A. Meghdari, M. Alemi, A. G. Pour, and A. Taheri, Spontaneous
human-robot emotional interaction through facial expressions, in Social
Robotics: 8th International Conference, ICSR 2016, Kansas City, MO,
USA, November 1-3, 2016 Proceedings (A. Agah, J.-J. Cabibihan, A. M.
Howard, M. A. Salichs, and H. He, eds.), (Cham), pp. 351–361, Springer
International Publishing, 2016.
[21] U. Hess and R. E. Kleck, Differentiating emotion elicited and deliberate
emotional facial expressions, European Journal of Social Psychology,
vol. 20, no. 5, pp. 369–385, 1990.
[22] M. Hirose, T. Takenaka, H. Gomi and N. Ozawa, Humanoid robot,
Journal of the Robotics Society of Japan, vol. 15, no. 7, pp. 983–985,
1997.
[23] K. Hirai, M. Hirose, Y. Haikawa and T. Takenaka, The Honda
humanoid robot: development and future perspective, Industrial Robot:
An International Journal, vol. 26, no. 4, pp. 260–266, 1999.
[24] P. Ekman, J.C. Hager, W.V. Friesen, The symmetry of emotional and
deliberate facial actions, Psychophysiology, 18: 101-106, 1981.
[25] A. Schaefer, F. Nils, X. Sanchez, and P. Philippot, Assessing the
effectiveness of a large database of emotion-eliciting films: A new
tool for emotion researchers, Cognition and Emotion, vol. 24, no. 7,
pp. 1153–1172, 2010.
[26] Aldebaran (Softbank Robotics), Pepper robot,
https://www.ald.softbankrobotics.com/en/robots/pepper (Last accessed:
April 2018)
[27] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin,
S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga,
S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden,
M. Wicke, Y. Yu, and X. Zheng, Tensorflow: A system for large-scale
machine learning in 12th USENIX Symposium on Operating Systems
Design and Implementation (OSDI 16), pp. 265–283, 2016. [28] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, ImageNet:
A Large-Scale Hierarchical Image Database in IEEE Computer Vision
and Pattern Recognition (CVPR), 2009.
[29] Softbank Robotics, ALMood Module,
http://doc.aldebaran.com/2-4/naoqi/core/almood.html (Last accessed:
April 2018)
[30] OMRON Corporation, Facial Expression Estimation Technology,
https://www.omron.com/media/press/2012/10/e1023.html (Last accessed:
April 2018)
[31] Google Inc., Cloud Vision API, https://cloud.google.com/vision/ (Last
accessed: April 2018)
[32] Microsoft Corporation, Emotion API,
https://azure.microsoft.com/en-us/services/cognitive-services/emotion/
(Last accessed: April 2018)
[33] Kairos AR, Inc.,Human Analytics, https://www.kairos.com/features (Last
accessed: April 2018)
@article{"International Journal of Mechanical, Industrial and Aerospace Sciences:77439", author = "Silvia Santano Guillén and Luigi Lo Iacono and Christian Meder", title = "Affective Robots: Evaluation of Automatic Emotion Recognition Approaches on a Humanoid Robot towards Emotionally Intelligent Machines", abstract = "One of the main aims of current social robotic research
is to improve the robots’ abilities to interact with humans. In order
to achieve an interaction similar to that among humans, robots
should be able to communicate in an intuitive and natural way
and appropriately interpret human affects during social interactions.
Similarly to how humans are able to recognize emotions in other
humans, machines are capable of extracting information from the
various ways humans convey emotions—including facial expression,
speech, gesture or text—and using this information for improved
human computer interaction. This can be described as Affective
Computing, an interdisciplinary field that expands into otherwise
unrelated fields like psychology and cognitive science and involves
the research and development of systems that can recognize and
interpret human affects. To leverage these emotional capabilities
by embedding them in humanoid robots is the foundation of
the concept Affective Robots, which has the objective of making
robots capable of sensing the user’s current mood and personality
traits and adapt their behavior in the most appropriate manner
based on that. In this paper, the emotion recognition capabilities
of the humanoid robot Pepper are experimentally explored, based
on the facial expressions for the so-called basic emotions, as
well as how it performs in contrast to other state-of-the-art
approaches with both expression databases compiled in academic
environments and real subjects showing posed expressions as well
as spontaneous emotional reactions. The experiments’ results show
that the detection accuracy amongst the evaluated approaches differs
substantially. The introduced experiments offer a general structure
and approach for conducting such experimental evaluations. The
paper further suggests that the most meaningful results are obtained
by conducting experiments with real subjects expressing the emotions
as spontaneous reactions.", keywords = "Affective computing, emotion recognition, humanoid
robot, Human-Robot-Interaction (HRI), social robots.", volume = "12", number = "6", pages = "584-9", }
