A NOVEL APPROACH OF HUMAN EMOTIONAL ESTIMATION SYSTEM VIA MULTISENSORY DATASETS FUSION METHODS

Felix Yustian Setiono

Abstract


Human emotional understanding has become one of the most rapidly emerging fields developed over the years since the emotional state is created and influenced by several neither easily measured and nor predicted factors. The proposed novel approach is based on the mixture of the processed acceleration data of several designed activities and processed heart rate data under several emotional states, recorded by the application of`smart band devices. Since the processed acceleration and heart rate data are running on different dimensionalities and recorded by different types of sensors, some machine learning-based methods applied to fusing these two kinds of data. The results of the data fusion methods are defined as the new fused datasets for Multi-Layer Perceptron and Support Vector Machines classifiers based on a selected criterion. At last, the accuracy results of the selected classifiers running under the defined machine-learning-based fusion dataset compared to each other to find out the effectiveness of the proposed fusion method.

Keywords


datasets fusion algorithm, emotional state estimation, human-robot interaction, heart rate variability, machine learning

Full Text:

PDF

References


Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology, “Heart rate variability: Standards of measurement, physiological interpretation, and clinical use”, European Heart Journal, vol. 17, pp. 354-381, 1996.

Tiying Cui, “Spectrum Analysis of Heart Rate Variability (HRV)”, September 2013 (student paper).

Wilhelm von Rosenberg et al, “Resolving Ambiguities in the LF/HF Ratio: LF-HF Scatter Plots for the Categorization of Mental and Physical Stress from HRV”, Frontiers in Physiology, vol. 8, article 360 (June 2017).

Shafer, Fred and J.P. Ginsberg. “An Overview of Heart Rate Variability Metrics and Norms”, in Frontiers in Public Health 2017, vol. 5, pp. 258 (September 2017)

Kuusela T. “Methodological aspects of heart rate variability analysis”, in Heart Rate Variability (HRV) Signal Analysis, CRC Press, pp. 9-42 (2009)

Sung-Nien Yu and Shu-Feng Chen, “Emotion State Identification Based on Heart Rate Variability and Genetic Algorithm”, in Proc. 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Milan, 2015, pp. 538-541.

Youseop Jo et al, “Emotion Recognition Through Cardiovascular Response in Daily Life Using KNN Classifier”, Advances in Computer Science and Ubiquitous Computing, Lecture Note in Electrical Engineering 474, pp. 1451-1456 (2018).

SiliconLabs BlueGiga BLED112-v1 Product Page, https://www.silabs.com/products/wireless/bluetooth/bluetooth-low-energy-modules/bled112-bluetooth-smart-dongle (January 2019)

Empatica E4 Product Page, https://www.empatica.com/research/e4/ (January 2019)

Empatica BLE Server for Windows (Beta) Product Page, http://developer.empatica.com/windows-ble-server.html (January 2019)

Gari D. Clifford, “Signal Processing Methods for Heart Rate Variability”, Ph.D. dissertation, Dept Engineering Science, Oxford University, 2002.

Mickaël Ménard et al, “Emotion Recognition Based on Heart Rate and Skin Conductance”, in Proc. 2nd International Conference on Physiological Computing System, Angers France, 2015, pp. 26-32.

J. De Jonckheere et al. “Heart rate variability analysis as an index of emotion regulation processes”: Interest of the Analgesia Nociception Index (ANI)”, in Proc. 34th Annual International Conference of the IEEE EMBS, San Diego, California USA, 2012, pp. 3432-3435.

Han-Wen Guo et al, “Heart Rate Variability Signal Features for Emotion Recognition by using Principal Component Analysis and Support Vector Machine”, in Proc. 16th International Conference on Bioinformatics and Bioengineering, Taichung Taiwan, 2012, pp. 274-277.

Nam T. Nguyen et al. “A Potential Approach for Emotion Prediction Using Heart Rate Signal”, in Proc 9th International Conference on Knowledge and System Engineering, pp. 221-226

Rhenan Bartels, “Pythonic package for Heart Rate Variability Analysis version 0.2.3”, https://github.com/rhenanbartels/hrv (January 2019)

Rakshit, Raj, V.R. Reddy, P. Deshpande. “Emotion Detection and Recognition using HRV Features Derived from Photoplethysmogram Signal”, in Proc. of the 2nd Workshop on Emotion Representations and Modelling for Companion Systems ERM4CT 2016, Tokyo Japan, 2016, article no.2.

Chin, Zi Ha, et al. “Daily Activities Classification on Human Motion Primitives Detection Dataset”, in Computational Science and Technology, Lecture Notes in Electrical Engineering 481, Springer Nature, Singapore, 2019, pp. 117-125.

Brouwer, Anne-Marie, et al. “Improving Real-Life Estimates of Emotion Based on Heart Rate: A Perspective on Taking Metabolic Heart Rate Into Account”, in Frontiers in Human Neuroscience, vol. 12, article 284 (July 2018)

Sun, Kook Yo, et al. “Neural Network-Based Emotion Using Heart Rate Variability and Skin Resistance”, in Proc. of Advanced in Natural Computation part 1, ICNC 2005, Changsha, China, pp. 818-824.

Elwenspoek, M. and Wiegerink, R. "Mechanical Microsensors". New York: Springer, 1993, pp. 132-145

Doscher, James. "Accelerometer Design and Applications". Norwood, MA: Analog Devices, 2005, pp. 61

Empatica. "How is the acceleration data formatted in E4 connect?", https://support.empatica.com/hc/en-us/articles/202028739-How-is-the-acceleration-data-formatted-in-E4-connect- (July 20, 2018)

Hu, Li-Yu, et al. "The distance function effect on k-nearest neighbor classification for medical datasets." SpringerPlus 5.1 (2016): 1-9.

Starner, Thad, Joshua Weaver, and Alex Pentland. "Real-time american sign language recognition using desk and wearable computer based video." IEEE Transactions on pattern analysis and machine intelligence 20.12 (1998): 1371-1375.

Arthur, David, and Sergei Vassilvitskii. “k-means++: The advantages of careful seeding” Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, Society for Industrial and Applied Mathematics (2007)

Altman, Naomi S. "An introduction to kernel and nearest-neighbor non parametric regression." The American Statistician 46.3 (1992): 175-185.

Piryonesi, S. Madeh, and Tamer E. El-Diraby. "Role of Data Analytics in Infrastructure Asset Management: Overcoming Data Size and Quality Problems." Journal of Transportation Engineering, Part B: Pavements 146.2 (2020): 04020022.

Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. "The elements of statistical learning: data mining, inference, and prediction". Springer Science & Business Media, 2009.

Sergei Lebedev, et al. "Hidden Markov Model (HMM) Learning Library for Python (textit{hmmlearn})", https://github.com/hmmlearn/hmmlearn (October 2020)

Dorland, William A. "Dorland's Illustrated medical dictionary". Saunders, 1981.

Schmidt, A; Thews, G (1989). "Autonomic Nervous System". In Janig, W (ed.). Human Physiology (2 ed.). New York, NY: Springer-Verlag. pp. 333–370.

"Allostatic load notebook: Parasympathetic Function". the Wayback Machine - 1999, MacArthur research network, UCSF

Russell, Peter (2013). "Biology - Exploring the Diversity of Life". Toronto: Nelson Education. p. 946. ISBN 978-0-17-665133-6.

Betts, J. Gordon, et al. "Anatomy & Physiology (2013)." Open Stax College & Rice University.

Onose, G., et al. "Integrative emphases on intimate, intrinsic propensity/pathological processes–causes of self recovery limits and also, subtle related targets for neuroprotection/pleiotropicity/multimodal actions, by accessible therapeutic approaches–in spinal cord injuries." Journal of medicine and life 3.3 (2010): 262.




DOI: http://dx.doi.org/10.26623/elektrika.v16i2.10624

Refbacks

  • There are currently no refbacks.


Office: Electrical Engineering Lecturer Room, 7th Floor of Menara USM Building, Universitas Semarang, Jalan Soekarno - Hatta, Tlogosari, Semarang - Central Java Tel:(024) 6702757 Fax: (024) 6702272, Email: elektrika@usm.ac.id

View My Stats

Creative Commons License
This work is licensed under a  Creative Commons Attribution 4.0 International License.