3D HMM-based Facial Expression Recognition using Histogram of Oriented Optical Flow
DOI:
https://doi.org/10.14738/tmlai.36.1661Keywords:
Human Computer Interaction (HCI), Facial Expression Recognition, Feature Extraction, Optical Flow, Hidden Markov Model, HMM, Three -dimensional Hidden Markov Model, 3D HMM.Abstract
In this paper, we propose a 3D HMM (Three-dimensional Hidden Markov Models) approach to recognizing human facial expressions and associated emotions. Human emotion is usually classified by psychologists into six categories: Happiness, Sadness, Anger, Fear, Disgust and Surprise. Further, psychologists categorize facial movements based on the muscles that produce those movements using a Facial Action Coding System (FACS). We look beyond pure muscle movements and investigate facial features – brow, mouth, nose, eye height and facial shape – as a means of determining associated emotions. Histogram of Optical Flow is used as the descriptor for extracting and describing the key features, while training and testing are performed on 3D Hidden Markov Models. Experiments on datasets show our approach is promising and robust.
References
(1) Bashyal, Shishir, and Ganesh K. Venayagamoorthy. 2008. Recognition of facial expressions using gabor wavelets and learning vector quantization. Engineering Applications of Artificial Intelligence 21
(7): 1056-64.
(2) Bay, Herbert, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. 2008. Speeded-up robust features (SURF). Computer Vision and Image Understanding 110 (3): 346-59.
(3) Bouchaffra, Djamel. 2010. Conformation-based hidden markov models: Application to human face identification. Neural Networks, IEEE Transactions on 21 (4): 595-608.
(4) Bouchaffra, Djamel, and Abbes Amira. 2008. Structural hidden markov models for biometrics: Fusion of face and fingerprint. Pattern Recognition 41 (3): 852-67.
(5) Bruhn, Andrés, Joachim Weickert, and Christoph Schnörr. 2005. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision 61 (3): 211-31.
(6) Chaudhry, Rizwan, Arunkumar Ravichandran, Georg Hager, and René Vidal. 2009. Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. Paper presented at Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, .
(7) Cohen, Ira, Nicu Sebe, Ashutosh Garg, Lawrence S. Chen, and Thomas S. Huang. 2003. Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding 91 (1): 160-87.
(8) Donoser, Michael, and Horst Bischof. 2006. Efficient maximally stable extremal region (MSER) tracking. Paper presented at Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, .
(9) Draper, Bruce A., Kyungim Baek, Marian Stewart Bartlett, and J. Ross Beveridge. 2003. Recognizing faces with PCA and ICA. Computer Vision and Image Understanding 91 (1): 115-37.
(10) Ekman, Paul, and Wallace V. Friesen. 1977. Facial action coding system.
(11) Elramsisi, AM, MA Zohdy, and NK Loh. 1991. A joint frequency-position domain structure identification of nonlinear discrete-time systems by neural networks. Automatic Control, IEEE Transactions on 36 (5): 629-32.
(12) Hamm, Jihun, Christian G. Kohler, Ruben C. Gur, and Ragini Verma. 2011. Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders. Journal of Neuroscience Methods 200 (2): 237-56.
(13) Horn, Berthold K., and Brian G. Schunck. 1981. Determining optical flow. Paper presented at 1981 Technical symposium east, .
(14) Jafri, Rabia, and Hamid R. Arabnia. 2009. A survey of face recognition techniques. Journal of Information Processing Systems 5 (2): 41-68.
(15) Jitén, Joakim, and Bernard Merialdo. 2007. Video modeling using 3D hidden markov model. Paper presented at VISAPP (1), .
(16) Joshi, Dhiraj, Jia Li, and James Z. Wang. 2006. A computationally efficient approach to the estimation of two-and three-dimensional hidden markov models. Image Processing, IEEE Transactions on 15 (7): 1871-
(17) Kanade, Takeo, Jeffrey F. Cohn, and Yingli Tian. 2000. Comprehensive database for facial expression analysis. Paper presented at Automatic Face and Gesture Recognition, 2000. Proceedings. Fourth IEEE International Conference on, .
(18) Karray, Fakhreddine, Milad Alemzadeh, Jamil Abou Saleh, and Mo Nours Arab. 2008. Human-computer interaction: Overview on state of the art.
(19) Khan, Aftab Ali, and MA Zohdy. 1997. A genetic algorithm for selection of noisy sensor data in multisensor data fusion. Paper presented at American Control Conference, 1997. Proceedings of the 1997, .
(20) Ledoux, H., and CM Gold. 2005. Interpolation as a tool for the modelling of three-dimensional geoscientific datasets. Paper presented at Proceedings ISPRS
(21) Lien, James J., Takeo Kanade, Jeffrey F. Cohn, and Ching-Chung Li. 1998. Automated facial expression recognition based on FACS action units. Paper presented at Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on, .
(22) Lien, Jenn-Jier James. 1998. Automatic Recognition of Facial Expressions using Hidden Markov Models and Estimation of Exprssion Intensity.
(23) Lowe, David G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60 (2): 91-110.
(24) Lucas, Bruce D., and Takeo Kanade. 1981. An iterative image registration technique with an application to stereo vision. Paper presented at IJCAI, .
(25) Mahoor, Mohammad H., Steven Cadavid, Daniel S. Messinger, and Jeffrey F. Cohn. 2009. A framework for automated measurement of the intensity of non-posed facial action units. Paper presented at Computer Vision and Pattern Recognition Workshops, 2009. CVPR Workshops 2009. IEEE Computer Society Conference on, .
(26) Matas, Jiri, Ondrej Chum, Martin Urban, and Tomás Pajdla. 2004. Robust wide-baseline stereo from maximally stable extremal regions. Image and Vision Computing 22 (10): 761-7.
(27) Moreno, Plinio, Alexandre Bernardino, and José Santos-Victor. 2005. Gabor parameter selection for local feature detection. In Pattern recognition and image analysis., 11-19Springer.
(28) Pinto, Sílvia Cristina Dias, Jesús P. Mena-Chalco, Fabrício Martins Lopes, Luiz Velho, and Roberto Marcondes Cesar Jr. 2011. 3D facial expression analysis by using 2D and 3D wavelet transforms. Paper presented at Image Processing (ICIP), 2011 18th IEEE International Conference on, .
(29) Praseeda Lekshmi, V., and M. Sasikumar. 2009. Analysis of facial expression using gabor and SVM. International Journal of Recent Trends in Engineering 1 (2): 1-43.
(30) Rabiner, Lawrence R. 1989. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE 77 (2): 257-86.
(31) Sánchez, A., José V. Ruiz, Ana Belén Moreno, Antonio S. Montemayor, Javier Hernández, and Juan José Pantrigo. 2011. Differential optical flow applied to automatic facial expression recognition. Neurocomputing 74 (8): 1272-82.
(32) Schmidt, Miriam, Martin Schels, and Friedhelm Schwenker. 2010. A hidden markov model based approach for facial expression recognition in image sequences. In Artificial neural networks in pattern recognition., 149-160Springer.
(33) Shan, Caifeng, Shaogang Gong, and Peter W. McOwan. 2009. Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision Computing 27 (6): 803-16.
(34) Sirovich, Lawrence, and Michael Kirby. 1987. Low-dimensional procedure for the characterization of human faces. Josa a 4 (3): 519-24.
(35) Soyel, Hamit, and Hasan Demirel. 2012. Localized discriminative scale invariant feature transform based facial expression recognition. Computers & Electrical Engineering 38 (5): 1299-309.
(36) Tian, Ying-li, Takeo Kanade, and Jeffrey F. Cohn. 2002. Evaluation of gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity. Paper presented at Automatic Face and Gesture Recognition, 2002. Proceedings. Fifth IEEE International Conference on, .
(37) ———. 2001. Recognizing action units for facial expression analysis. Pattern Analysis and Machine Intelligence, IEEE Transactions on 23 (2): 97-115.
(38) Tsai, Ching-Chih, You-Zhu Chen, and Ching-Wen Liao. 2009. Interactive emotion recognition using support vector machine for human-robot interaction. Paper presented at Systems, Man and Cybernetics, 2009. SMC 2009. IEEE International Conference on, .
(39) Turk, Matthew, and Alex P. Pentland. 1991. Face recognition using eigenfaces. Paper presented at Computer Vision and Pattern Recognition, 1991. Proceedings CVPR'91., IEEE Computer Society Conference on, .
(40) Uddin, Md Zia, JJ Lee, and T-S Kim. 2009. An enhanced independent component-based human facial expression recognition from video. Consumer Electronics, IEEE Transactions on 55 (4): 2216-24.
(41) Valstar, Michel, and Maja Pantic. 2006. Fully automatic facial action unit detection and temporal analysis. Paper presented at Computer Vision and Pattern Recognition Workshop, 2006. CVPRW'06. Conference on, .
(42) Wu, Jian, Zhiming Cui, Victor S. Sheng, Pengpeng Zhao, Dongliang Su, and Shengrong Gong. 2013. A comparative study of SIFT and its variants. Measurement Science Review 13 (3): 122-31.
(43) Yamato, Junji, Jun Ohya, and Kenichiro Ishii. 1992. Recognizing human action in time-sequential images using hidden markov model. Paper presented at Computer Vision and Pattern Recognition, 1992. Proceedings CVPR'92., 1992 IEEE Computer Society Conference on, .
(44) Zohdy, M., D. Bouchaffra, and J. Quinlan. 2001. Optimal mapping from chromosome space to feature space for solving sequential pattern recognition problems. Paper presented at Circuits and Systems, 2001. MWSCAS 2001. Proceedings of the 44th IEEE 2001 Midwest Symposium on, .
