1. SENSIBILITY AS A DYNAMICAL SYSTEM

Our sensibility is a dynamical system in multi-dimensional sensory space, where each dimension corresponds to the sort of sensibility such as visual, audio, tactile, and so on. Music is the time change of sound, and the movie is the time change of vision. Tactile changes over time have not considered so far, but we have proposed a Tactile Score that represents tactile changes over time. On the other hand, the visual-spatial changes in a painting can also be transformed into temporal changes. When measuring spatial change, spatial movement occurs with or without the amount of sensitivity from point p₁ to p₂. In this paper, the path from p₁ to p₂ is ignored, and the focus is on the speed of travel. The movement speed is fast when glancing at the painting. On the other hand, if we want to appreciate the picture carefully, the movement speed is slow. Therefore, even spatial differences are described as temporal changes in sensibility.

Our sensory systems are composed of differentiators, and we perceive sensory stimulations through the partial differentiating sensory stimulations.

In order to calculate the partial differentiation of sensibility, we must set a measurable quantity of the sensibility. We can use a not only quantitative measure such as brightness of the picture, loudness of sound but also can use a quality measure such as the value of Semantic Differential (SD) method for measuring impressions or emotion such as beautiful, fear, happy.

The most crucial concept of the partial differentiation of sensibility measures the differences in time or/and space of sensibility; it shares the concept of partial differentiation in mathematics, but this is not a mathematical concept, and a mathematical equation of sensibility is not required; in calculating partial differentiation of sensibility, we use differentiable measure.

In the following, the notation of partial differential is borrowed. The reason is that sensibility is generally multivariable, and we discuss its temporal changes. Therefore, the temporal change of sensibility and the partial differential have similar conceptual meanings. We emphasize that the notation of partial derivatives in this paper does not claim that the amount of sensitivity is linear or continuous, not that it is a mathematical partial derivative.

2. THE PARTIAL DIFFERENTIATION OF SENSIBILITY

In general, sensitivity consists of cross-talks of multiple sensibilities. For example, music is affected not only by the pitch described in a music score but also by the time variation of timbre and volume. Here, the duration is the duration of the same pitch sound.

In case, we define that music is represented by four variables of pitch, duration, timbre, and volume, music and M have time changes of four sensibilities;

2.1. An Example of Partial Differentiation of Sensibility - Music

We derive the partial differentiation of sensibility, about the time change of the volume at the beginning of Beethoven’s Symphony No. 5. Figure 1 shows the result of calculation.

Figure 1

Sensitivity deviation of music in the volume. The beginning of Beethoven Symphony No. 5 1st movement. The vertical axis shows the volume in deci Bell (dB), time bin is 1/44, 100 s.

Next, we show an example of a partial differentiation about the pitch of the same music in Figure 2.

Figure 2

Sensitivity deviation of music in the pitch. The beginning of Beethoven Symphony No. 5 1st movement. The vertical axis shows the volume in deci Bell (dB), time bin is 60/108 s for half notes (Ludwig van Beethoven, who composed this music, indicate it).

Not limited to this example, in the case of music, the partial differentiation of volume is the change in sound pressure that we call the waveform. And the partial differentiation for pitch becomes a musical score.

3. TACTILE SCORE

Sensitivity can be differentiated, as shown in the previous section. Obtaining differentiation is the discretization, and the discretization enables sensibility to compute digitally.

For example, a cheerful hello and a not-cheerful hello have the same character information. However, the sensibility of these two greetings is different. This difference is called the difference in wording. The characterization of “wording” has been done by frequency [1]. We characterize the “wording” by the sensitivity partial differentiation concerning volume.

We show an example, the audio samples of saying “oha yoh gozai masu (good morning in Japanese)” as “cheerful” and “not-cheerful”. In this example, partial derivative of sensitivity, the time bin of the audio data was 0.01 s, and the maximum and minimum volumes were divided into 36 containers.

To explain the result of partial differentiation, we use a staff notation, where the pitch indicates loudness. Weak sound pressure is described as high sound, and strong sound pressure is described as low sound. The sound data was converted to musical instruments digital interface format and written in staff notation using score generation software (Figure 3).

Figure 3

Partial differential of sensitivity. (a) A lively “Good morning” voice sample, (b) “Good morning” with no energy.

The discretized result of the partial differential of sensitivity is called the tactile bit (T bit), and a staff notation describes T-bit is Tactile Score [2]. Tactile Score was proposed by Suzuki et al. to represent time-varying somatosensory stimuli such as massages.

4. MEDIA TRANSFORMATION SYSTEM BY USING T-BIT AND TACTILE SCORE

Partial differentiation of sensibility can be used to rationalize media conversion while maintaining sensibility. We converted the dancer’s body movements into vibrotactile (Figure 4).

Figure 4

Media transformation system: by partial differentiation of sensibility of physical movements dance, we obtain T-bit and Tactile Score. And by changing the amplitude of vibrations, we can transform a Tactile Score into a vibrotactile sensation.

There are many elements in dance, but only body movements are partially distinguished. Only the speed of movement of the body is extracted from the dance. And all other elements are ignored.

We used OpenPose [3] for the motion capture of the body movements of three-dimensional coordinates of three points: wrist, ankle, and elbow. The speed and acceleration of the center of gravity position of wrist, ankle and elbow are discretized into five levels and converted into Tactile Scores. Then, the amplitude of vibration changed according to the Tactile Score. We call this system as Media Transformation System (MTS).

4.1. Case Report 1 – SHOKKAKU 2019

SHOKKAKU is a Japanese word, which means tactile sense. SHOKKAKU 2019 was an event on science and arts relating to tactile sense and held at Kyoto University Graduate School of Human and Environmental Studies (Figure 5).

Figure 5

From SHOKKAKU 2019, SHOKKAKU 2019 was held at Kyoto University Graduate School of Human and Environmental Studies from June 29 to 30, 2019. In the center of the photo is Norihito Ishii performing his work of Butoh “empty garden”. The mask worn by the audience is a Haptic Vibration Display. The iron plate at the top center of the photo is the Thunder Sheet, which is vibrated and makes vibration sounds.

We used MTS and transform body-movements of dance into tactile sense and created a performance artwork. This artwork was produced through collaboration with a Butoh dancer, Norihito Ishii (San-Kai-Jyuku Company) and a composer Eric Maestri.

We used a C960 webcam (eMeet Co., Ltd.) to shot live video for motion capture, video data send to PC (MacBook Pro), and OpenPose analyzed motion on MAX MSP (Cycling’74 Co., Ltd.). MTS generated a tactile score and transformed to sound data. By using an audio mixer, Eric added sound effect, then sound source for vibration was generated. The vibration sound source was distributed as vibration tactile sensation from four thunder sheets (made by Kolberg, 100 × 200 cm) arranged around the audience and 40 Haptic Vibration Displays (HVDs), worn by the audience (Figure 6).

Figure 6

Floor layout of dance performance by using MTS.

We have originally developed the HVD to display vibration haptics through multiple sound exciters; a sound exciter (Dayton Audio Co., Ltd.) is an oscillator that responds to frequencies below the mid-range of the input audio signal. Thunder Sheet is a percussion instrument used to play the sound of thunder, as used in Richard Strauss’s Alpine Symphony. A Thunder sheet is usually played by hitting with a mallet, but we attached a sound exciter to the Thunder Sheet and vibrated. In the venue, four Thunder sheets were surrounded the audience and every audience wore a HVD. Hence, every audience had tactile sense transformed from dance’s movement directly from a HVD and in-directly from Thunder sheets.

5. FINAL REMARK

Sensibility information is converted to T-bit by partial differentiation of sensibility. T-bit makes it easier to handle sensitivity information, which was previously difficult to drive in a unified manner. Information technology in the 20th century has specialized in logic and data processing, but T-bit allows sensibility information to be integrated with conventional information technology.

CONFLICTS OF INTEREST

The author declares no conflicts of interest.

ACKNOWLEDGMENTS

This research supported by SPIRITZ, Kyoto University and the author would like to thank, for the executive producer of SHOKKAKU 2019, Professor Masaaki Sakagami (Kyoto University).

AUTHOR INTRODUCTION

Dr. Yasuhiro Suzuki

He is an Associate Professor of Graduate School of Informatics, Nagoya University, Japan. He graduated from Japan Advanced Institute of Science and Technology in 1995, received Doctor degree of Informatics from Kyoto University in 2001.