1. INTRODUCTION

When those who need visual assistance, such as aged or visually impaired, go out, or even for those absorbed in a mobile-phone while walking, there is a risk of colliding with obstacles, in particular, the pedestrians around him/her. To avoid the collision, automatic pedestrian detection using a computer vision system may be of great help. Most of the conventional pedestrian detection methods are a combination of Histograms of Oriented Gradients (HOG) and SVM [1], or they use LiDAR which can measure the distance to an object with high accuracy. However, these methods may fail in avoiding the collision, because they are normally used for vehicles and judge only the existence of the pedestrians in front of an ego-car. The information on walk direction of a detected pedestrian is necessary to avoid collision.

This paper proposes a method of detecting a pedestrian and judging its walk direction around a person who may need visual assistance. The method uses a MY VISION system which performs processing of images/videos provided from an ego-camera mounted on a user’s body or a head to obtain useful visual information from them. The system therefore functions as a substitute for a user’s eyes and a brain.

The method initially extracts an area having optical flow on the fed image. It then adopts Multiple Scale Cell (MSC)-HOG [2], an improved version of the original HOG [1], to know if a pedestrian is in the area, and, if it exists, it uses Histograms of Optical Flow (HOF) [3,4] to determine the walk direction of the pedestrian.

The proposed method is introduced in Section 2. Performed experiment is presented in Section 3. Discussion is given in Section 4, and the paper is concluded in Section 5.

2. PROPOSED METHOD

2.2. MSC-HOG Feature

MSC-HOG features expanded from HOG features are used as the features for pedestrian detection. In the MSC-HOG feature, the position and size of cells are variable, and the cells are arranged along a human (edge intensity) model. One of the advantages of MSC-HOG feature over the existent HOG features is that, since a cell is not fixed to a single size, the cells that match the characteristics of a human body shape, e.g., a vertically long cell which may fit a human leg, is able to be placed. Then more effective features are derived from an image for human detection. Figure 1 shows an example of the cells arranged for human detection using MSC-HOG features.

Figure 1

Shape and distribution of the cells in MSC-HOG features: (a) a human model, (b) an example of cell layout (red rectangles).

2.2.1. Human model

A large number of human images are collected in order to create a human model. A brightness gradient image is created from the collected image, and an average of all the brightness gradient images is calculated to create a human model. An example of a human model is given in Figure 1a.

2.2.2. Calculation of the feature

1. (i)

   Calculation of a gradient vector

   The gradient intensity m(x, y) and direction θ(x, y) are calculated for each pixel in a cell by Equations (1–4).

   m(x,y)=fx(x,y)2+fy(x,y)2 (1)

   θ(x,y)=tan-1fy(x,y)fx(x,y) (0°≤θ(x,y)<180°) (2)

   fx(x,y)=f(x+1,y)-f(x,y) (3)

   fy(x,y)=f(x,y+1)-f(x,y) (4)

   Here, f(x, y) is the luminance value of an input image: f_x(x, y) and f_y(x, y) are the first derivatives of f(x, y) in the x- and the y-direction, respectively.

2. (ii)

   Gradient histogram

   Using the gradient intensity calculated by Equation (1), a gradient histogram is created with each cell. The gradient direction (0° ≤ θ(x, y) < 180°) is separated into nine directions by 20°, and the gradient vector of each pixel is assigned to one of the nine directions.

   The gradient intensity of each pixel in a cell is added with each direction, and a histogram showing the frequency in each direction is created. The histogram value hist(j) in each direction in a cell is calculated by

   histj=∑(x,y)∈celli∧dir(x,y)=jm(x,y) (5)

   Here cell_i represents the ith cell in a detection window.

3. (iii)

   MSC-HOG features

   Using Equation (5), a histogram of each cell is calculated. For the ith cell, a nine-dimensional feature vector ci′=(hij)=(histj)i  (j=1, 2, ..., 9) defined by Equation (6). Vector ci′ is normalized using Equation (7) to obtain c_i whose L2 norm is 1. Finally, the feature vectors of all the N cells in a detection window are combined into a single MSC-HOG feature vector x_HOG by Equation (8).

   ci′=(hi,1,hi,2,…,hi,9) (6)

   ci=ci′‖ci′‖2 (7)

   xHOG=(c1,c2,…,cN) (8)

2.3. Histograms of Optical Flow

Histograms of optical flow feature is used for recognizing a pedestrian’s walk direction. HOF is a feature of optical flow vectors in a local area. An area of interest is separated into some overlapping blocks: a block is again separated into some cells with no overlap: a cell is represented by a histogram of optical flow vectors (referred to as flow vectors hereafter) which each pixel in the cell has.

A flow vector is calculated using two successive frames by TV-L1 Optical Flow [4] which produces dense optical flow. The constraint condition of TV-L1 Optical Flow is shown in Equation (9). The first term of Equation (9) is a constraint on the direction of a flow vector, whereas the second term is a constraint on the difference in luminance value between frames.

∫Ω|∇u(x)|dx+λ∫Ω|I1(x+u(x))-I0(x)|dx →min(w.r.t. u(x)) (9)

Here, I₀ and I₁ are two successive images; u(x) is a displacement vector at x; λ is a weight; and x ∈ Ω.

The calculation is repeated until the sum of these terms is minimized with respect to u ≡ u(x). Then u provides a flow vector.

A flow vector is characterized by its intensity and direction. Then the flow vectors in a cell can be represented by a histogram like HOG. In the proposed method, the flow direction θ (0° ≤ θ < 360°) is separated into 18 directions by 20°. Then the histogram has 18 bins. Let us denote the magnitude of bin j (j = 1, 2, …, 18) in the histogram of cell i (i = 1, 2, …, N) in block k (k = 1, 2, …, K) by (hist_j)_ki. Then its value is calculated by Equations (1) and (5). Let us denote a 18-dimensional vector representing a histogram of cell i in block k by a_ki = (hist_j)_ki = (h₁, h₂, …, h₁₈)_ki. Then we have a 18N-dimensional vector bk′=(ak1,ak2, …, akN) corresponding to block k. It is normalized by

bk=bk′‖bk′‖2 (10)

Finally, an overall 18NK-dimensional optical flow vector x_flow representing the flow feature of an interested area is defined by the following:

xflow=(b1,b2,…,bK) (11)

An example of a visualized TV-L1 Optical Flow image is shown in Figure 2. From successive images (a) and (b), a flow image (c) is obtained. Figure 2d depicts cells and blocks arrangement on the flow image.

Figure 2

A flow image between two successive regions of interest: (a) image at time t, (b) image at t + Δ, (c) visualized flow, (d) an example of cell (black rectangle) and block (red rectangle) arrangement.

2.4. Extracting a Candidate Pedestrian Area

Candidate pedestrian areas are extracted using TV-L1 Optical Flow. First, a flow vector is calculated from two successive image frames. Second, the flow angle and intensity at each pixel are visualized using the HSV color system to create a flow image. Third, the flow image is first-order differentiated in the x-direction. This is to separate the pedestrian flow from the camera flow and to generate an edge image of the flow. Finally, the edge image is integrated into a single region by morphological closing process. A candidate area is then obtained in the form of a bounding box containing the area. The procedure is illustrated in Figure 3.

Figure 3

Finding a candidate pedestrian area: (a) image at time t, (b) image at t + Δ, (c) a flow image, (d) a flow edge image, (e) a candidate pedestrian area.

3. EXPERIMENTAL RESULTS

3.3. Results

3.3.1. Experiment 1

Table 1 shows the confusion matrix of the pedestrian recognition experiment. The evaluation values are as follows: Re = 0.933, Pre = 0.914, F = 0.923, Rec = 0.934.

		Predicted class
		Positive	Negative
Actual class	Positive	2101	151
	Negative	198	2802

Table 1

Confusion matrix of Exp_1

3.3.2. Experiment 2

In this experiment, several parameters are chosen as follows:

1. (i)

   Number of pedestrians: 1, 2, 3 in a video.

2. (ii)

   Walk speed of a user and pedestrians: normal, fast. Note that a user (an ego-camera holder) also walks.

3. (iii)

   Walk directions of pedestrians: from front left, front, front right to the user.

Experiments were performed with the combination of above parameters. The result on the case of a single pedestrian is shown in Table 2. All the three cases of (i) are put together in Table 3.

	Front left	Front	Front right	Average
Det	0.817	0.737	0.898	0.817
Acc	0.674	0.473	0.573	0.573

Table 2

Pedestrian detection rate and walk direction recognition rate in the case of a single pedestrian in a video

No. pedestrians	Det	Acc
1	0.817	0.573
2	0.756	0.599
3	0.583	0.617

Table 3

Average pedestrian detection rate and walk direction recognition rate to the number of pedestrians in a video

Some experimental results are given by images in Figure 4, in which (a) shows the case of a single pedestrian walking from front left to a user. A blue box surrounding him indicates that his walk direction is from front left to the user, i.e., the direction is correctly recognized. Three pedestrians’ case is given in Figure 4b, in which a red box represents a walk direction from front right to the user, whereas a green box is that from front to the user. The walk directions are almost correctly recognized.

Figure 4

Experimental results: (a) case of a single pedestrian, (b) case of three pedestrians.

4. DISCUSSION

In this paper, a method was proposed of detecting pedestrians and judging their walk directions by a MY VISION system which a user holds. The system detects pedestrians from a video captured by an ego-camera of a user. It is supposed to be used in a future by those who need visual assistance in a daily life. This claims the importance of the study. The performed experiments gave expecting results as shown in Figure 4. Some issues, however, need to be further investigated to improve the method.

In Exp_1, the pedestrian detection rate was 93.4% which is reasonable. INRIA Person Dataset mostly provides standing person images and not walking person images. So authors’ own dataset containing human walk images was added in the training data. As one may know, a walking motion has two main patterns; a legs spread pattern and a legs closed pattern. The latter occurs when both legs cross and it is similar to a standing pattern. The training data prepared in Exp_1 is therefore stands to reason. The amount of the training data, however, needs to be increased to raise the detection rate more.

As shown in Table 2, the average pedestrian detection rate is higher than 80%, but the average walk direction recognition rate is still in the lower level. This means that the flow vector, x_flow, has not extracted directional information reliably from a flow image yet. The value γ, the threshold of Equation (12), may be further examined.

Table 2 also shows that frontal direction is difficult to judge its direction compared to other directions. This is because the frontal direction provides smaller flow vectors than diagonal direction. Instability of direction recognition in this case is seen in Figure 4b. Spacing between successive image frames in the calculation of a flow vector needs further study.

As seen in Table 3, the detection rate of a pedestrian, Det, is decreasing as the number of pedestrians increase. This is understandable, because it is not very simple to find candidate pedestrian areas when pedestrians cross or overlap each other. Tracking of moving objects needs to be considered in addition to the optical flow analysis. On the other hand, walk direction recognition rate, Acc, is not very different among the three cases.

As for (ii), walk speed, no particular difference was observed in the results. This may be because there was not a significant difference between the used normal and fast speed. Larger difference in walk speed between an approaching pedestrian and a user may be advantageous in segmenting a candidate pedestrian area from a flow image.

5. CONCLUSION

In this paper, we proposed a pedestrian detection method including walking direction recognition using a MY VISION system. For this purpose, the MSC-HOG feature and the HOF feature by TV-L1 Optical Flow were introduced. The experiments on pedestrian detection and walk direction recognition using the proposed method gave expecting results. The issues for further study include parameter tuning in optical flow calculation and pedestrian detection, employment of moving object tracking, along with increasing training data in designing classifiers. One of the application fields of the proposed method may be the third eye system for those who need visual assistance such as elderly or visually impaired to avoid collision with pedestrians on the way.

CONFLICTS OF INTEREST

The authors declare they have no conflicts of interest.

ACKNOWLEDGMENT

This work was supported by JSPS KAKENHI Grant Number 25350477.

AUTHORS INTRODUCTION

Prof. Joo Kooi Tan

She is presently with Department of Mechanical and Control Engineering in Kyushu Institute of Technology as a Professor. Her current research interests include three-dimensional shape/motion recovery, human detection and its motion analysis from videos. She was awarded SICE Kyushu Branch Young Author’s Award in 1999, the AROB Young Author’s Award in 2004, Young Author’s Award from IPSJ of Kyushu Branch in 2004 and BMFSA Best Paper Award in 2008, 2010, 2013 and 2015. She is a member of IEEE, The Information Processing Society, The Institute of Image Electronics Engineers, and The Biomedical Fuzzy Systems Association of Japan.

Mr. Kenta Hori

He obtained B.E. and M.E. from Kyushu Institute of Technology. His research includes computer vision, pedestrian detection and machine learning.

Prof. Seiji Ishikawa

He graduated Tokyo University and was awarded BE, ME and PhD there. He is now Professor Emeritus, Kyushu Institute of Technology. He was Visiting Researcher of The University of Sheffield, UK, and Visiting Professor of Utrecht University, NL. Professor Ishikawa is an active researcher in the field of visual sensing. His research interests include human motion recovery, description and recognition. He was awarded The Best Paper Award in 2008, 2013 and 2015 from Biomedical & Fuzzy System Association, Japan. He is a member of IEEE.