1. INTRODUCTION

In Japan, society is aging at an unprecedentedly rapid pace, the average age of the population has been increasing, and these trends are expected to continue into the future. We have therefore been studying ways of applying information technology to improving the medical treatment provided to elderly people with psychiatric disorders.

One of our current studies is aimed at analyzing the facial expressions of persons in a moving image captured by webcams. In this study, we developed a real-time facial expression horizontal asymmetry analysis and visualization system. Unlike current and previously used methods [1,2] in which the moving image data are first saved in a Personal Computer (PC) and then analyzed, the moving image in our method is captured by a webcam and analyzed in real-time by using Open Source Computer Vision (OpenCV) image-processing software [3] and a previously proposed feature parameter (facial expression intensity) [1,2] that is based on the mouth area. The visualization method proposed in this paper is based on our reported method [4] which involves creating a graph from the OpenCV image data for analyzing the relationship between the facial expression intensity and time. This visualization is performed concurrently with the facial expression intensity measurement.

2. PROPOSED SYSTEM AND METHOD

2.2. Mouth Area Extraction

First, moving image data are changed from RGB to YCbCr image data, after which, by using OpenCV, the face area is extracted from the changed image as a rectangular shape, and the lower 40% portion of the face area is standardized. Next, the mouth area is extracted from that area. The reason the mouth area was selected for facial expression analysis is because it is where the differences between neutral and happy facial expressions appear most distinctly. An example of a face image and the extracted mouth area image is shown in Figure 1.

Figure 1

Moving image frame and extracted mouth area.

In this study, the mouth area is separated into left and right half regions, and the facial expression intensity of each region is separately measured. An example of a mouth area separated into the left and right half regions is shown in Figure 2.

Figure 2

The mouth area separated in the left and right half regions.

3. EXPERIMENT

In this study, we used a Dell OPTIPLEX 780 PC equipped with Intel Core 2 Duo E8400 3.0 GHz central processing units and 4.0 GB of random access memory. The Microsoft Windows 7 Professional operating system (OS) was installed on the PC and Microsoft Visual C++ 2008 Express Edition was used as the development language.

The newly proposed system display is shown in Figure 3. We performed evaluation experiments for two test subjects (hereafter Subjects A and B) under the three conditions listed below. Furthermore, the message “SMILE, PLEASE!” was flashed on the display screen at points 20 and 40s from the system start.

1. 1.

   Subjects maintained neutral faces for about 60s.

2. 2.

   Subjects intentionally smiled when the above message was displayed.

3. 3.

   Subjects raised the right side of their mouth when the above message was first displayed, and raised the left side of their mouth when the above message was displayed a second time.

Figure 3

Display while new system is in operation.

The experimental results are shown in the following figures (Figures 4–16). Arrows in each graph indicate the max point of the facial expression intensity.

Figure 4

Facial expression intensity of the whole mouth under experimental condition 1 (upper: Subject A, lower: Subject B).

Figure 5

Reference frame under experimental condition 1 (left: Subject A, right: Subject B).

Figure 6

Facial expression intensity of the right half region of the mouth under experimental condition 1 (upper: Subject A, lower: Subject B).

Figure 7

Facial expression intensity of the left half region of the mouth under experimental condition 1 (upper: Subject A, lower: Subject B).

Figure 8

Facial expression intensity of the whole mouth under experimental condition 2 (upper: Subject A, lower: Subject B).

Figure 9

Facial expression intensity of the right half of the mouth under experimental condition 2 (upper: Subject A, lower: Subject B).

Figure 10

Facial expression intensity of the left half region under experimental condition 2 (upper: Subject A, lower: Subject B).

Figure 11

Facial expression intensity of the whole mouth under experimental condition 3 (upper: Subject A, lower: Subject B).

Figure 12

Mouth area of point A1 (left) and point B1 (right).

Figure 13

Facial expression intensity of the right half of the mouth under experimental condition 3 (upper: Subject A, lower: Subject B).

Figure 14

Right half mouth area of point A2 (left) and point B2 (right).

Figure 15

Facial expression intensity of the left half region under experimental condition 3 (upper: Subject A, lower: Subject B).

Figure 16

Left half mouth area of point A3 (left) and point B3 (right).

The experimental results obtained thus far suggest that this system could be useful for situations where real-time reaction to facial expressions is one of the important factors. Upon examining the experimental results, some problems were discovered. Facial expression intensities on the left half of the subject’s faces during neutral face recording were always higher than those of the right half, and the reason for this discrepancy is unclear.

4. CONCLUSION

We proposed a real-time system for facial expression horizontal asymmetry analysis and visualization and performed a number of evaluation experiments. It is believed that this system could prove useful in the treatment of psychiatric ailments such as depressive disorder and dementia, in addition to providing treatment options for persons with half side body paralysis.

CONFLICTS OF INTEREST

There is no conflicts of interest.

ACKNOWLEDGMENTS

The authors would like to thank Professor J. Narumoto of Kyoto Prefectural University of Medicine for his valuable support and helpful advice in the course of this research. We would also like to thank Mr. K. Nishimura, a student of the Graduate School of Kyoto Prefectural University, for his cooperation in the experiments. This research was supported by COI STREAM of the Ministry of Education, Culture, Sports, Science, and Technology of Japan.

Authors Introduction

Mr. Ryoichi Shimada

He received his B.S. and M.S. degrees from Kyoto Prefectural University in 2015, 2017, respectively. He works at JFE Systems, Inc.

Dr. Taro Asada

He received his B.S., M.S. and PhD degrees from Kyoto Prefectural University in 2002, 2004 and 2010, respectively. He works as an Associate Professor at the Graduate School of Life and Environmental Sciences of Kyoto Prefectural University. His current research interests are human interface and image processing. HIS, IIEEJ member.

Dr. Yasunari Yoshitomi

He received his B.E, M.E. and PhD degrees from Kyoto University in 1980, 1982 and 1991, respectively. He works as a Professor at the Graduate School of Life and Environmental Sciences of Kyoto Prefectural University. His specialties are applied mathematics and physics, informatics environment, intelligent informatics. IEEE, HIS, ORSJ, IPSJ, IEICE, SSJ, JMTA and IIEEJ member.

Dr. Masayoshi Tabuse

He received his M.S. and PhD degrees from Kobe University in 1985 and 1988 respectively. From June 1992 to March 2003, he had worked in Miyazaki University. Since April 2003, he has been in Kyoto Prefectural University. He works as a Professor at the Graduate School of Life and Environmental Sciences of Kyoto Prefectural University. His current research interests are machine learning, computer vision and natural language processing. IPSJ, IEICE and RSJ member.