Journal of Robotics, Networking and Artificial Life

Volume 8, Issue 1, June 2021, Pages 47 - 51

Music Recommendation System Driven by Physiological Evaluation Appearing in Fingertip Skin Temperature

Authors
Mayuka Wada1, Taro Asada2, Yasunari Yoshitomi2, *, Masayoshi Tabuse2
1Analysis Business Unit, The First Division, Shimadzu System Development Corp., 1 Tokudaiji-cho, Nishinokyo, Nakagyo-ku, Kyoto, Japan
2Graduate School of Life and Environmental Sciences, Kyoto Prefectural University, 1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
*Corresponding author. Email: yoshitomi@kpu.ac.jp
Corresponding Author
Yasunari Yoshitomi
Received 19 October 2020, Accepted 15 April 2021, Available Online 31 May 2021.
DOI
10.2991/jrnal.k.210521.011How to use a DOI?
Keywords
Music recommendation system; music therapy; peripheral skin temperature
Abstract

In recent years, music therapy has been performed with the aim at recovering the cognitive function of elderly people. In the previously reported system, it was necessary to input the subjective evaluation of the user to determine the next recommended song. In this study, variations in fingertip skin temperature were used as input instead of subjective evaluation, focusing on evaluation of emotion expressed as physiological response through variations in fingertip skin temperature. Our system can be used with no support for users.

Copyright
© 2021 The Authors. Published by Atlantis Press B.V.
Open Access
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

In recent years, against the backdrop of increasingly aging societies in developed countries such as Japan, activities using music for the elderly (“music therapy” [1,2]) have been used with the aim of improving recognition ability. We previously developed music recommendation systems for music therapy for older people [3,4].

However, in the previously reported music recommendation system [3], it was necessary to input the subjective evaluation of the user (“I want to listen to it again” or “I don’t want to listen to it again”) into the recommended music in order to determine the next recommended song. Therefore, in the case of a user who has difficulty in responding by himself/herself, such as a dementia patient, the support of a caregiver may be required, and the problem of securing human resources hinders the use of our previously reported system [3].

In this study, we focused on the evaluation of emotions by measuring peripheral skin temperature [2], developed a method for determining recommended songs based on the physiological evaluation of the user, and evaluated our system through the experiments.

2. RELATED STUDIES

We now briefly explain the systems used in our previous study [3] and another study [5], which we build upon to evaluate emotions in this study.

2.1. Previously Reported System for Music Recommendation

The method for music recommendation used in this study is based on a previously reported methods [3]. Similar to the previously reported music recommendation system [3], our system is a hybrid system for improving recognition ability using collaborative filtering and impression words [6]. See the paper [3] for details on the music recommendation module in our system. See the paper [7] for details on the music recommendation using impression words in our system.

2.2. Evaluation of Emotions by Measuring Peripheral Skin Temperature

In the study by Watanabe et al. [5], the subject watched a horror video and an environmental video alternately, and the peripheral skin temperatures of the subjects viewing the videos were measured. Three peripheral parts, the front of auricle, the nose, and the back of the right index finger were measured, and temperature changes were compared and examined. No significant change was observed in the skin temperature on the front of the auricle and on the nose. However, the skin temperature on the back of the fingers of the right hand decreased when viewing the horror video and increased when viewing the environmental video. From this result, the influence of autonomic nerve activity appears to be reflected strongly in the peripheral skin temperature of the fingertip as a change in blood flow due to vasoconstriction.

3. CONSTRUCTION OF A PHYSIOLOGICAL EVALUATION METHOD FOR MUSIC

A physiological evaluation method for music, which measures the peripheral skin temperatures of subjects listening to music, will be described. In this study, we selected the pad of the left index finger as the peripheral part for measuring temperature.

3.1. Changes in Peripheral Skin Temperature while Listening to Music

To evaluate emotions by measuring peripheral skin temperature in a music recommendation system, we investigated the changes in peripheral skin temperature of subjects listening to music.

When subject A (a female in her 20s) listened to her favorite song (“Canon,” composed by Johann Pachelbel) and least favorite song (“Mystery III -Heterophony for Swirl Butoh -,” composed by Akira Nishimura), the peripheral skin temperature tended to rise while listening to her favorite song and decreased while listening to her least favorite song (Figures 1 and 2) [6].

Figure 1

Changes in peripheral skin temperature when listening to music (Favorite song).

Figure 2

Changes in peripheral skin temperature when listening to music (Least favorite song).

3.2. Determination of Threshold for Changes in Peripheral Skin Temperature

Similar to the previously reported music recommendation system [3], the system used in this study determines the next recommended song by inputting an evaluation value (“1” if you want to listen to the song again, or “0” if you don’t want to listen to the song again) into a computer. In this study, we determined the threshold value for dividing “1” when “physiologically preferable” and “0” when “physiologically unfavorable” based on the change in peripheral skin temperature.

Based on the results of the preliminary experiment, 0.4°C (the temperature difference between the maximum temperature and the subsequent minimum temperature) and 180s (the elapsed time since the subject had begun listening to music) were set as the threshold values [3].

3.3. Binarization of Physiological Evaluation

Using the two thresholds described in Subsection 3.2, the music being listened to was judged to be “Physiologically favorable (1)” or “Physiologically unfavorable (0)” based on the temperature change of the peripheral skin when listening to the music. Figure 3 shows the judgment processing flow for the physiological evaluation of music. In Figure 3, the elapsed time from the start of measurement is indicated by “time,” and the measured temperature at that time is indicated by “temp.” The maximum temperature from the start of measurement to that point is indicated as “max,” and the difference between “max” and “temp” is indicated as “dif.”

Figure 3

Flowchart for binarizing physiological evaluation [6].

4. EXPERIMENTAL EVALUATION

4.1. Conditions

The music recommendation system [6] used in this study is an improved version of the previously reported system [3]. The improved system can support multiple music databases, and is aimed at the elderly. It was reported that elderly people tend to choose songs that are the subject of elementary school music textbooks as familiar songs (favorite songs) [8]. Therefore, we selected a CD described as an anthology of older songs enjoyed by older people with dementia, and selected 52 songs on the CD that were also included in a music textbook database for elementary schools as well as 58 popular songs [9]. There were a total of 10 subjects: four females in their 20s (Subjects A–D), five males in their 20s (Subjects E–I), and one male in his 60s (Subject J). For programming, we used Visual C++ 6.0 (Microsoft) on a PC (Dell Optiplex790, CPU: Intel Corei7-2600 3.40 GHz, main memory: 4.00 GB, and OS: Windows 7 Professional, Microsoft) for the experiment. We used a digital temperature and relative humidity sensor SHT11 [10] to measure temperature.

4.2. Results and Discussion

4.2.1. Concordance rate

Table 1 shows the results of the concordance rate between the physiological evaluation and the subjective evaluation. The concordance rate of all experimental results was relatively high, with an average of 70.9%. Individually, the average for children’s song was 67.3%, while that for popular songs was 74.5%. However, since there is a large difference in the concordance rate between subjects (Table 1), improvements such as reviewing the threshold are considered necessary. In addition, depending on the subject, the concordance rate of either the children’s song or the popular song is high, and the concordance rate of the other is low. Therefore, it would appear that there are some problems that cannot be solved only by adjusting the threshold value.

Subject A B C D E F G H I J Total
Children’s songs Number of recommended songs 15 15 15 15 15 15 15 15 15 15 150
Evaluation value 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
Physiological evaluation 8 7 5 10 6 9 6 9 7 8 9 6 8 7 10 5 8 7 11 4 78 72
Subjective evaluation 9 6 8 7 14 1 9 6 8 7 5 10 7 8 11 4 5 10 13 2 89 61
Number of concordances: Individual 8 6 3 5 6 1 5 5 4 4 5 6 7 7 8 2 2 4 11 2 59 42
Number of concordances: All 14 8 7 10 8 11 14 10 6 13 101
Concordance rate (%) 93.3 53.3 46.7 66.7 53.3 73.3 93.3 66.7 40 86.7 67.3
Popular songs Number of recommended songs 8 11 10 10 9 8 8 8 9 12 93
Evaluation value 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
Physiological evaluation 3 5 3 8 9 1 8 2 6 3 1 7 3 5 6 2 6 3 10 2 55 38
Subjective evaluation 3 5 6 5 10 0 8 2 2 7 2 6 5 3 7 1 5 4 9 3 57 36
Number of concordances: Individual 3 5 1 3 9 0 7 1 2 3 1 6 2 2 6 1 4 2 9 2 44 25
Number of concordances: All 8 4 9 8 5 7 4 7 6 11 69
Concordance rate (%) 100 36.4 90 80 55.6 87.5 50 87.5 66.7 91.7 74.5
Table 1

Experimental results [6]

4.2.2. Correct answer rate of music recommendation

Table 2 shows a comparison of the percentage of correct answers for the song recommendation results. The percentage of correct answers for the music recommendations was relatively low. The probable reason for this was that many of the subjects were relatively young, not elderly people who were targeted by the music database used in this study. As an example, a male in his 60s (Subject J) had a relatively high percentage of correct answers (73.3–83.3%) in both the physiological and subjective evaluations. Figure 4 shows the change in peripheral skin temperature for the song that Subject J judged to be his “favorite song” in both the physiological and subjective evaluations. The peripheral skin temperature gradually increased over time (Figure 4).

Subject A B C D E F G H I J Total
Children’s songs Physiological evaluation (%) 53.3 33.3 40 40 46.7 60 53.3 66.7 53.3 73.3 52
Subjective evaluation (%) 60 53.3 93.3 60 53.3 33.3 46.7 73.3 33.3 86.7 59.3
Popular songs Physiological evaluation (%) 37.5 27.3 90 80 66.7 12.5 37.5 75 66.7 83.3 57.6
Subjective evaluation (%) 37.5 54.5 100 80 22.2 25 62.5 87.5 55.6 75 60
Table 2

Recommendation accuracy [6]

Figure 4

Changes of subject J in peripheral skin temperature when listening to music (Favorite song).

Subject J (a male in his 60s) had a high rate of accuracy for both children’s songs and popular songs. From this, the selection of the music database is considered to be one of the factors that contributed to the low recommendation accuracy for one of the subjects who were in their 20s. In future research, we plan to conduct experiments that focus on the elderly.

5. CONCLUSION

By focusing on the evaluation of emotions by measuring the peripheral skin temperature, we have developed a music recommendation system that inputs subjective evaluations using peripheral skin temperature. This system can be used with no support for users. The concordance rate between the physiological evaluation and the subjective evaluation was relatively high, with an average of 70.9% in the present experiments. The concordance rate varied depending on the subject and the music genre. One of the factors is that the threshold is set for one subject. In addition to music, there are many factors that affect peripheral skin temperature, such as room temperature and the physical condition of the subject. In the present experiments, the 90% subjects were 20s. In future research, we plan to conduct experiments that focus on the elderly, reviewing the threshold in our system.

CONFLICTS OF INTEREST

The authors declare they have no conflicts of interest.

ACKNOWLEDGMENT

We would also like to thank the subjects of our experiments for their cooperation.

AUTHORS INTRODUCTION

Ms. Mayuka Wada

She received her B.S. degree from Kyoto Prefectural University in 2019. She works at Shimadzu System Development Corp.

Dr. Taro Asada

He received his B.S., M.S. and PhD degrees from Kyoto Prefectural University in 2002, 2004 and 2010, respectively. He works as an Associate Professor at the Graduate School of Life and Environmental Sciences of Kyoto Prefectural University. His current research interests are human interface and image processing. He is a member of HIS and IIEEJ.

Dr. Yasunari Yoshitomi

He received his B.E., M.E. and PhD degrees from Kyoto University in 1980, 1982 and 1991, respectively. He works as a Professor at the Graduate School of Life and Environmental Sciences of Kyoto Prefectural University. His specialties are applied mathematics and physics, informatics environment, intelligent informatics. He is a member of IEEE, HIS, ORSJ, IPSJ, IEICE, SSJ, JMTA and IIEEJ.

Dr. Masayoshi Tabuse

He received his M.S. and PhD degrees from Kobe University in 1985 and 1988 respectively. From June 1992 to March 2003, he had worked in Miyazaki University. Since April 2003, he has been in Kyoto Prefectural University. He works as a Professor at the Graduate School of Life and Environmental Sciences of Kyoto Prefectural University. His current research interests are machine learning, computer vision and natural language processing. He is a member of IPSJ, IEICE and RSJ.

REFERENCES

[5]T Watanabe, M Okubo, and T Kuroda, Evaluation of emotion using peripheral skin temperature-effectiveness of the back of finger measurement, JJME, Vol. 36, 1998, pp. 637. (in Japanese).
[6]M Wada, T Asada, Y Yoshitomi, and M Tabuse, Music recommendation system using peripheral skin temperature as input data, Human Interface Society, in Proceedings of the Human Interface Symposium (Kyoto, Japan, 2019), pp. 736-739. (in Japanese).
[7]T Kumamoto and K Ohta, Design of scales to represent user’s impressions of a music piece for use in a music-retrieval system, IPSJ SIG Notes, 2001-NL-147, 2002, pp. 35-40. (in Japanese).
[8]T Takahashi, Research report on songs familiar to people advanced in years, J. Jpn. Music Ther. Assoc., Vol. 15, 1997, pp. 68-75. (in Japanese).
[9]T Akahoshi, Good old anthology enjoyable for people advanced in years and troubled with dementia, Kirara shobo, Tokyo, 2009. (in Japanese).
Journal
Journal of Robotics, Networking and Artificial Life
Volume-Issue
8 - 1
Pages
47 - 51
Publication Date
2021/05/31
ISSN (Online)
2352-6386
ISSN (Print)
2405-9021
DOI
10.2991/jrnal.k.210521.011How to use a DOI?
Copyright
© 2021 The Authors. Published by Atlantis Press B.V.
Open Access
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - JOUR
AU  - Mayuka Wada
AU  - Taro Asada
AU  - Yasunari Yoshitomi
AU  - Masayoshi Tabuse
PY  - 2021
DA  - 2021/05/31
TI  - Music Recommendation System Driven by Physiological Evaluation Appearing in Fingertip Skin Temperature
JO  - Journal of Robotics, Networking and Artificial Life
SP  - 47
EP  - 51
VL  - 8
IS  - 1
SN  - 2352-6386
UR  - https://doi.org/10.2991/jrnal.k.210521.011
DO  - 10.2991/jrnal.k.210521.011
ID  - Wada2021
ER  -