A Real Time Set Up for Retrieval of Emotional States from Human Neural Responses
Real time non-invasive Brain Computer Interfaces have a significant progressive role in restoring or maintaining a quality life for medically challenged people. This manuscript provides a comprehensive review of emerging research in the field of cognitive/affective computing in context of human neural responses. The perspectives of different emotion assessment modalities like face expressions, speech, text, gestures, and human physiological responses have also been discussed. Focus has been paid to explore the ability of EEG (Electroencephalogram) signals to portray thoughts, feelings, and unspoken words. An automated workflow-based protocol to design an EEG-based real time Brain Computer Interface system for analysis and classification of human emotions elicited by external audio/visual stimuli has been proposed. The front end hardware includes a cost effective and portable Emotiv EEG Neuroheadset unit, a personal computer and a set of external stimulators. Primary signal analysis and processing of real time acquired EEG shall be performed using MATLAB based advanced brain mapping toolbox EEGLab/BCILab. This shall be followed by the development of MATLAB based self-defined algorithm to capture and characterize temporal and spectral variations in EEG under emotional stimulations. The extracted hybrid feature set shall be used to classify emotional states using artificial intelligence tools like Artificial Neural Network. The final system would result in an inexpensive, portable and more intuitive Brain Computer Interface in real time scenario to control prosthetic devices by translating different brain states into operative control signals.
[1] R.W. Picard, E. Vyzas and J. Healey, "Toward Machine Emotional Intelligence: Analysis of Affective Physiological State”, IEEE T. Pattern Anal., vol. 23, pp.1175-1191, 2001.
[2] M. A. Khalilzadeh, S. M. Homam, S. A. Hosseini and V. Niazmand, "Qualitative and Quantitative Evaluation of Brain Activity in Emotional Stress”, Iranian Journal of Neurology, vol.8 (28), pp. 605-618, 2010.
[3] K. Schaaff and T. Schultz, "Towards an EEG-Based Emotion Recognizer for Humanoid Robots”, in The 18th IEEE International Symposium on Robot and Human Interactive Communication, Toyama, Japan. 2009: pp. 792-796.
[4] D. P. Subha, P.K. Joseph, R. Acharya and C.M. Lim, "EEG Signal Analysis: A Survey”, Journal of Medical Systems, vol. 34(2), pp. 195-212, 2010.
[5] S.A. Hosseini, "Classification of Brain Activity in Emotional States Using HOS Analysis”, International Journal of Image, Graphics and Signal Processing, vol. 4(1), pp. 21-27, 2012.
[6] K. Takahashi and A. Tsukaguchi, "Remarks on Emotion Recognition from Multi-Modal Biopotential Signals”, Proceeding of the IEEE International Workshop on Robot and Human Interactive Communication, Sept. 20-22, 2004, IEEE Xplore, Japan, pp: 95-100
[7] C. T. Yuen, W. S. San, J. H. Ho and M. Rizon, "Effectiveness of Statistical Features for Human Emotions Classification using EEG Biosensors,” Research Journal of Applied Sciences, Engineering and Technology, vol. 5(21), pp. 5083-5089, 2013.
[8] M. Murugappan, R. Nagarajan and S. Yaacob, "Comparison of different wavelet features from EEG signals for classifying human emotions”, Proceeding of IEEE Symposium on Industrial Electronics and Applications, 2009, pp: 836-841, Kuala Lumpur.
[9] E. D. Ubeyli, "Combined Neural Network Model Employing Wavelet Coefficients for EEG Signal Classification”, Digit. Signal. Process, vol. 19, pp.297-308, 2009.
[10] M. Murugappan, R. Nagarajan and S. Yaacob, "Combining Spatial Filtering and Wavelet Transform for Classifying Human Emotions Using EEG Signals”, J. Med. Biol. Eng., vol. 31(1), pp.45-51, 2010.
[11] K. Schaaff and T. Schultz, "Towards Emotion Recognition from Electroencephalographic Signals”, Proc. of 3rd International Conference on Affective Computing and Intelligent Interaction (ACII) and Workshops, pp. 1-6, 2009.
[12] Y. P. Lin, C. H. Wang, T. L. Wu, S. K. Jeng and J. H. Chen, "EEG-Based Emotion Recognition in Music Listening: A Comparison of Schemes for Multiclass Support Vector Machine”, Proceeding of ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing, 2009, pp.489- 492, Taipei.
[13] Mikhail M, Ayat K.E, "Using Minimal Number of Electrodes for Emotion Detection Using Brain Signals Produced from a New Elicitation Technique”, Int. J. Autonomous and Adaptive Communications Systems, vol. 6(1), pp. 80-97, 2013.
[14] K.E. Ko, H. C. Yang and K. B. Sim, "Emotion Recognition Using EEG Signals with Relative Power Values and Bayesian Network”, International Journal of Control, Automation & Systems, vol. 7(5), pp.865-870, 2009.
[15] N. Jatupaiboon, S. Panngum, and P. Israsena, "Real-Time EEG-Based Happiness Detection System”, Hindawi Publishing Corporation: The Scientific World Journal Volume, Article ID 618649, 12 pages, 2013,
[16] K. C. Chua, V. Chandran and R. Acharya, "Application of Higher Order Statistics/Spectra in Biomedical Signals - A Review”, Journal of Medical Engineering and Physics, vol. 32(7), pp. 679-689, 2010.
[17] P.C. Petrantonakis and L. J. Hadjileontiadis, "EEG-Based Emotion Recognition Using Hybrid Filtering and Higher Order Crossings”, Proc. of 3rd International Conference on Affective Computing and Intelligent Interaction (ACII) and Workshops: IEEE, 2009, pp. 1-6, Amsterdam.
[18] P.C. Petrantonakis and L. J. Hadjileontiadis, "Emotion Recognition from Brain Signals Using Hybrid Adaptive Filtering and Higher Order Crossings Analysis”, IEEE Transactions on Affective Computing, vol. 1(2), pp. 81-97, 2010.
[19] P.C. Petrantonakis and L. J. Hadjileontiadis, "Emotion Recognition from EEG Using Higher Order Crossings”, IEEE T. Inf. Technol., vol. 14, pp. 186-197, 2010.
[20] Y. Liu, O. Sourina and M.K. Nguyen, "Real-Time EEG-Based Human Emotion Recognition and Visualization” Proceeding of International Conference on Cyberworlds, pp. 262-269, 2010.
[21] S. A. Hosseini, M. A. Khalilzadeh, M. B. Naghibi-Sistani and V. Niazmand, "Higher Order Spectra Analysis of EEG Signals in Emotional Stress States”, ITCS '10 Proceedings of Second International Conference on Information Technology and Computer Science, IEEE Computer Society, 2010, pp. 60-63, Washington, DC, USA.
[22] P. J. Lang, M. M. Bradley, and B. N. Cuthbert, "International Affective Picture System (IAPS): Affective Ratings of Pictures and Instruction Manual”, the Center for Research in Psychophysiology, University of Florida, Gainesville, FL, USA, 2005.
[23] M. M. Bradley and P. J. Lang, "International Affective Digitized Sounds (IADS): Stimuli, Instruction Manual and Affective Ratings”, the Center for Research in Psychophysiology, University of Florida, Gainesville, FL, USA, 1999.
[24] D. O. Bos, "EEG-Based Emotion Recognition”, [online] http://hmi.ewi. utwente.nl/verslagen/capita-selecta/CS-Oude_Bos-Danny.pdf, 2006.
[25] K. H. Kim, S. W. Bang and S. R. Kim, "Emotion Recognition System Using Short-Term Monitoring of Physiological Signals”, Medical and Biological Engineering and Computing, Vol. 42, pp. 419-427, 2004.
[26] Emotiv Website: http://www.emotiv.com/, Accessed 26 January, 2014.
[27] Neurosky Website: http://store.neurosky.com/products/mindwave-1, Accessed 26 January, 2014.
[28] XwaveWebsite: http://www.plxdevices.com/product_info.php?id= WAVESPORT, Accessed 27 Januaryy, 2014.
[29] Interaxon Muse Website: http://www.interaxon.ca/muse/, Accessed 27 Januaryy, 2014.
[30] R. Oostenveld, P. Fries, E. Maris, and J. M. Schoffelen, "FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data”, Computational Intelligence and Neuroscience Vol. 2011 (2011), Article ID 156869, 9 pages.
[31] A. Delorme and S. Makeig, "EEGLAB: An Open Source Toolbox for Analysis of Single-Trial EEG Dynamics”, J. Neurosci. Methods, vol. 134, pp. 9–21, Mar. 2004.
[32] G. Schalk, et al., "BCI2000: A General-Purpose Brain-Computer Interface (BCI) System”, IEEE Transactions on Biomedical Engineering, vol. 51(6), pp. 1034-1043, 2004.
[33] C. R. Pernet, N. Chauveau, C. Gaspar, and G. A. Rousselet, "LIMO EEG: A Toolbox for Hierarchical LInear MOdeling of ElectroEncephaloGraphic Data”, Computational Intelligence and Neuroscience, vol. 2011, Article ID 831409, 11 pages, 2011.
[34] C. Vidaurre, T. H. Sander, and A. Schlögl, "BioSig: The Free and Open Source Software Library for Biomedical Signal Processing,” Comput. Intell. Neurosci., pp. 935364, 2011.
[35] F. S. Bao, X. Liu, and C. Zhang, "PyEEG: An Open Source Python Module for EEG/MEG Feature Extraction,” Comput. Intell.Neurosci., pp. 406391, 2011.
[36] Openvibe website: http://openvibe.inria.fr/ Accessed 27 January, 2014.
[37] J. L. Andreassi, Psychophysiology: Human Behavior and Physiological Response, fourth ed. Lawrence Erlbaum Associates, Hillsdale, New Jersey, 2000.
[38] H. Suk and S. W. Lee, "A Novel Bayesian Framework for Discriminative Feature Extraction in Brain-Computer Interfaces”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35 (2), pp. 286-299, February 2013.
[39] A. C. Wanpracha, Y. Fan and C.S. Rajesh, "On the Time Series K-nearest Neighbor Classification of Abnormal Brain Activity”, IEEE T. Syst. Man Cy., vol.37, pp.1005-1016, 2007.
[40] C. J. Lin and M. H. Hsieh, "Classification of Mental Task from EEG Data Using Neural Networks Based On Particle Swarm Optimization”, Neurocomputing, vol.72, pp.1121– 1130, 2009.
[41] V. Srinivasan, C. Eswaran, and N. Sriraam, "Approximate Entropy-Based Epileptic EEG Detection Using Artificial Neural Networks,” IEEE Transactions on Information Technology in Biomedicine, vol. 11(3), May 2007.
[1] R.W. Picard, E. Vyzas and J. Healey, "Toward Machine Emotional Intelligence: Analysis of Affective Physiological State”, IEEE T. Pattern Anal., vol. 23, pp.1175-1191, 2001.
[2] M. A. Khalilzadeh, S. M. Homam, S. A. Hosseini and V. Niazmand, "Qualitative and Quantitative Evaluation of Brain Activity in Emotional Stress”, Iranian Journal of Neurology, vol.8 (28), pp. 605-618, 2010.
[3] K. Schaaff and T. Schultz, "Towards an EEG-Based Emotion Recognizer for Humanoid Robots”, in The 18th IEEE International Symposium on Robot and Human Interactive Communication, Toyama, Japan. 2009: pp. 792-796.
[4] D. P. Subha, P.K. Joseph, R. Acharya and C.M. Lim, "EEG Signal Analysis: A Survey”, Journal of Medical Systems, vol. 34(2), pp. 195-212, 2010.
[5] S.A. Hosseini, "Classification of Brain Activity in Emotional States Using HOS Analysis”, International Journal of Image, Graphics and Signal Processing, vol. 4(1), pp. 21-27, 2012.
[6] K. Takahashi and A. Tsukaguchi, "Remarks on Emotion Recognition from Multi-Modal Biopotential Signals”, Proceeding of the IEEE International Workshop on Robot and Human Interactive Communication, Sept. 20-22, 2004, IEEE Xplore, Japan, pp: 95-100
[7] C. T. Yuen, W. S. San, J. H. Ho and M. Rizon, "Effectiveness of Statistical Features for Human Emotions Classification using EEG Biosensors,” Research Journal of Applied Sciences, Engineering and Technology, vol. 5(21), pp. 5083-5089, 2013.
[8] M. Murugappan, R. Nagarajan and S. Yaacob, "Comparison of different wavelet features from EEG signals for classifying human emotions”, Proceeding of IEEE Symposium on Industrial Electronics and Applications, 2009, pp: 836-841, Kuala Lumpur.
[9] E. D. Ubeyli, "Combined Neural Network Model Employing Wavelet Coefficients for EEG Signal Classification”, Digit. Signal. Process, vol. 19, pp.297-308, 2009.
[10] M. Murugappan, R. Nagarajan and S. Yaacob, "Combining Spatial Filtering and Wavelet Transform for Classifying Human Emotions Using EEG Signals”, J. Med. Biol. Eng., vol. 31(1), pp.45-51, 2010.
[11] K. Schaaff and T. Schultz, "Towards Emotion Recognition from Electroencephalographic Signals”, Proc. of 3rd International Conference on Affective Computing and Intelligent Interaction (ACII) and Workshops, pp. 1-6, 2009.
[12] Y. P. Lin, C. H. Wang, T. L. Wu, S. K. Jeng and J. H. Chen, "EEG-Based Emotion Recognition in Music Listening: A Comparison of Schemes for Multiclass Support Vector Machine”, Proceeding of ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing, 2009, pp.489- 492, Taipei.
[13] Mikhail M, Ayat K.E, "Using Minimal Number of Electrodes for Emotion Detection Using Brain Signals Produced from a New Elicitation Technique”, Int. J. Autonomous and Adaptive Communications Systems, vol. 6(1), pp. 80-97, 2013.
[14] K.E. Ko, H. C. Yang and K. B. Sim, "Emotion Recognition Using EEG Signals with Relative Power Values and Bayesian Network”, International Journal of Control, Automation & Systems, vol. 7(5), pp.865-870, 2009.
[15] N. Jatupaiboon, S. Panngum, and P. Israsena, "Real-Time EEG-Based Happiness Detection System”, Hindawi Publishing Corporation: The Scientific World Journal Volume, Article ID 618649, 12 pages, 2013,
[16] K. C. Chua, V. Chandran and R. Acharya, "Application of Higher Order Statistics/Spectra in Biomedical Signals - A Review”, Journal of Medical Engineering and Physics, vol. 32(7), pp. 679-689, 2010.
[17] P.C. Petrantonakis and L. J. Hadjileontiadis, "EEG-Based Emotion Recognition Using Hybrid Filtering and Higher Order Crossings”, Proc. of 3rd International Conference on Affective Computing and Intelligent Interaction (ACII) and Workshops: IEEE, 2009, pp. 1-6, Amsterdam.
[18] P.C. Petrantonakis and L. J. Hadjileontiadis, "Emotion Recognition from Brain Signals Using Hybrid Adaptive Filtering and Higher Order Crossings Analysis”, IEEE Transactions on Affective Computing, vol. 1(2), pp. 81-97, 2010.
[19] P.C. Petrantonakis and L. J. Hadjileontiadis, "Emotion Recognition from EEG Using Higher Order Crossings”, IEEE T. Inf. Technol., vol. 14, pp. 186-197, 2010.
[20] Y. Liu, O. Sourina and M.K. Nguyen, "Real-Time EEG-Based Human Emotion Recognition and Visualization” Proceeding of International Conference on Cyberworlds, pp. 262-269, 2010.
[21] S. A. Hosseini, M. A. Khalilzadeh, M. B. Naghibi-Sistani and V. Niazmand, "Higher Order Spectra Analysis of EEG Signals in Emotional Stress States”, ITCS '10 Proceedings of Second International Conference on Information Technology and Computer Science, IEEE Computer Society, 2010, pp. 60-63, Washington, DC, USA.
[22] P. J. Lang, M. M. Bradley, and B. N. Cuthbert, "International Affective Picture System (IAPS): Affective Ratings of Pictures and Instruction Manual”, the Center for Research in Psychophysiology, University of Florida, Gainesville, FL, USA, 2005.
[23] M. M. Bradley and P. J. Lang, "International Affective Digitized Sounds (IADS): Stimuli, Instruction Manual and Affective Ratings”, the Center for Research in Psychophysiology, University of Florida, Gainesville, FL, USA, 1999.
[24] D. O. Bos, "EEG-Based Emotion Recognition”, [online] http://hmi.ewi. utwente.nl/verslagen/capita-selecta/CS-Oude_Bos-Danny.pdf, 2006.
[25] K. H. Kim, S. W. Bang and S. R. Kim, "Emotion Recognition System Using Short-Term Monitoring of Physiological Signals”, Medical and Biological Engineering and Computing, Vol. 42, pp. 419-427, 2004.
[26] Emotiv Website: http://www.emotiv.com/, Accessed 26 January, 2014.
[27] Neurosky Website: http://store.neurosky.com/products/mindwave-1, Accessed 26 January, 2014.
[28] XwaveWebsite: http://www.plxdevices.com/product_info.php?id= WAVESPORT, Accessed 27 Januaryy, 2014.
[29] Interaxon Muse Website: http://www.interaxon.ca/muse/, Accessed 27 Januaryy, 2014.
[30] R. Oostenveld, P. Fries, E. Maris, and J. M. Schoffelen, "FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data”, Computational Intelligence and Neuroscience Vol. 2011 (2011), Article ID 156869, 9 pages.
[31] A. Delorme and S. Makeig, "EEGLAB: An Open Source Toolbox for Analysis of Single-Trial EEG Dynamics”, J. Neurosci. Methods, vol. 134, pp. 9–21, Mar. 2004.
[32] G. Schalk, et al., "BCI2000: A General-Purpose Brain-Computer Interface (BCI) System”, IEEE Transactions on Biomedical Engineering, vol. 51(6), pp. 1034-1043, 2004.
[33] C. R. Pernet, N. Chauveau, C. Gaspar, and G. A. Rousselet, "LIMO EEG: A Toolbox for Hierarchical LInear MOdeling of ElectroEncephaloGraphic Data”, Computational Intelligence and Neuroscience, vol. 2011, Article ID 831409, 11 pages, 2011.
[34] C. Vidaurre, T. H. Sander, and A. Schlögl, "BioSig: The Free and Open Source Software Library for Biomedical Signal Processing,” Comput. Intell. Neurosci., pp. 935364, 2011.
[35] F. S. Bao, X. Liu, and C. Zhang, "PyEEG: An Open Source Python Module for EEG/MEG Feature Extraction,” Comput. Intell.Neurosci., pp. 406391, 2011.
[36] Openvibe website: http://openvibe.inria.fr/ Accessed 27 January, 2014.
[37] J. L. Andreassi, Psychophysiology: Human Behavior and Physiological Response, fourth ed. Lawrence Erlbaum Associates, Hillsdale, New Jersey, 2000.
[38] H. Suk and S. W. Lee, "A Novel Bayesian Framework for Discriminative Feature Extraction in Brain-Computer Interfaces”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35 (2), pp. 286-299, February 2013.
[39] A. C. Wanpracha, Y. Fan and C.S. Rajesh, "On the Time Series K-nearest Neighbor Classification of Abnormal Brain Activity”, IEEE T. Syst. Man Cy., vol.37, pp.1005-1016, 2007.
[40] C. J. Lin and M. H. Hsieh, "Classification of Mental Task from EEG Data Using Neural Networks Based On Particle Swarm Optimization”, Neurocomputing, vol.72, pp.1121– 1130, 2009.
[41] V. Srinivasan, C. Eswaran, and N. Sriraam, "Approximate Entropy-Based Epileptic EEG Detection Using Artificial Neural Networks,” IEEE Transactions on Information Technology in Biomedicine, vol. 11(3), May 2007.
@article{"International Journal of Medical, Medicine and Health Sciences:65191", author = "Rashima Mahajan and Dipali Bansal and Shweta Singh", title = "A Real Time Set Up for Retrieval of Emotional States from Human Neural Responses", abstract = "Real time non-invasive Brain Computer Interfaces have a significant progressive role in restoring or maintaining a quality life for medically challenged people. This manuscript provides a comprehensive review of emerging research in the field of cognitive/affective computing in context of human neural responses. The perspectives of different emotion assessment modalities like face expressions, speech, text, gestures, and human physiological responses have also been discussed. Focus has been paid to explore the ability of EEG (Electroencephalogram) signals to portray thoughts, feelings, and unspoken words. An automated workflow-based protocol to design an EEG-based real time Brain Computer Interface system for analysis and classification of human emotions elicited by external audio/visual stimuli has been proposed. The front end hardware includes a cost effective and portable Emotiv EEG Neuroheadset unit, a personal computer and a set of external stimulators. Primary signal analysis and processing of real time acquired EEG shall be performed using MATLAB based advanced brain mapping toolbox EEGLab/BCILab. This shall be followed by the development of MATLAB based self-defined algorithm to capture and characterize temporal and spectral variations in EEG under emotional stimulations. The extracted hybrid feature set shall be used to classify emotional states using artificial intelligence tools like Artificial Neural Network. The final system would result in an inexpensive, portable and more intuitive Brain Computer Interface in real time scenario to control prosthetic devices by translating different brain states into operative control signals.
", keywords = "Brain Computer Interface (BCI), Electroencephalogram (EEG), EEGLab, BCILab, Emotiv, Emotions, Interval features, Spectral features, Artificial Neural Network, Control applications.", volume = "8", number = "3", pages = "129-6", }
