Head rotation classification via video analysis

The problem of automatically measuring the engagement level of people is an area of great interest across a wide variety of fields. Engagement level measuring can be used to monitor drivers to guarantee safety of driving. Another application can be measurement of involvement levels of computer game players.In this thesis we focus on classification of head rotations to automatically measure the engagement level of viewers to the displayed video content. In the literature a number of methods are proposed for measurement of engagement level. Some of them use expensive sensors such as eye trackers while some others require complicated installments.In this thesis we use an on the shelf Web camera with a simple setting.. We define the problem as a two class classification problem, i.e., "involved in the screened content" or "not involved." Engagement classification has been performed by integrating head location and head movement features. The head movement based classification is activated where the location detection fails. 2-D feature vectors that comprise amplitude and angle of the estimated optical flow vectors extracted by the SIFT flow algorithm are used for motion classification. Head location is represented by 3-D feature vectors that consist of the initial head coordinates detected by Viola-Jones face detector and area of the detected region.. SIFT flow is a probabilistic optical flow estimation method and it is preferred for motion estimation because the probabilistic methods are more powerful than gradient based methods. This mainly because probabilistic methods produce useful extensions of the standard quadratic gradient techniques for computing optical flow, including an automatic gain control mechanism, and the incorporation of a prior bias on the flow. It provides (two-dimensional) flow vector confidence information, allowing later stages of processing to weight their use of the vectors accordingly, and it provides a framework for properly combining flow information with probabilistic information from other sources. We use images which have different head pose angle in different directions to report performance of engagement level classification of TV viewers. There are 8 angles, ±15, ±30, ±45, ±60 degrees and 6 angles, ±15, ±30, ±60 degrees of pan movement (x direction) and tilt movement (y direction), respectively. In addition to all these, there are different head pose combinations of pan and tilt angles for diagonal directions. Each angle is represented with one class for each direction. This means that there are 8 and 6 classes for pan and tilt movement. We classify head pose according to the angle classes. After classification we separate result classes in two groups ( involved in the screened content or not ), in order to report performance of engagement level classification. In this thesis, classification and performance reports are made in each direction separately. SIFT flow algorithm is used to get motion features to classificiation of head movement. Before applying SIFT flow algorithm to the frames, we get SIFT image from the RGB image. SIFT is a local descriptor to characterize local gradient information. SIFT descriptor is a sparse feature representation that consists of both feature extraction and detection. But in the SIFT flow algorithm, only use the feature extraction component. Per pixel in SIFT image, is represented with 128 dimensional vector. SIFT flow algorithm has an energy function similar to that of optical flow to estimate SIFT flow from two SIFT images. This energy function is designed to make flow vectors smooth, with discontinuities agreeing with object boundaries. Based on these two criteria, the energy function of SIFT flow is formulates. SIFT flow algorithm is a probabilistic based method as mentioned before. Therefore, minimum value of energy function (flow vector) is not calculated directly. On the contrary, probabilistic based optical flow, SIFT flow uses graph based methods which decrease complexity of the algorithm. To find the optimum solution of energy functions, tree-reweighted belief propagation algorithm is used. The optimum solution of that energy function, which is called "belief", is flow vectors in x and y direction. In classification problem we do not use flow vectors directly. We evaluate angle and magnitude of vectors and these are 2-D features to classify head movement. In this thesis, we discuss the problem as a binary classification. So in motion classification, we evaluated 28 binary datasets from 8 classes of the pan movement (x direction) and 15 binary datasets from 6 classes of tilt movement (y direction). To classify, each video frame classifies with each binary dataset. After classification, video frames are labeled in terms of angle class with majority voting. According to angle class labels, we decided involvement level as "involve" or "not involve". Class ±15 or ±30 degree means "involve", whereas class ±45 or ±60 degree means "not involve". Head location features are calculated through the instrument of Viola Jones face detector. Fast feature computation and efficient feature selection properties of the Viola - Jones algorithm is important advantages. We get 3-D feature which is x, y and A. x and y are the coordinates of beginning location and A is the area of the head location which is found with Viola-Jones face detector. Head location estimation problem is considered as a binary classification and location is classified "middle" or "side". In Viola-Jones algorithm, according to value of area A, if beginning pixels of location x, y is located respectively in 5%, 12%, 25% of middle of image pixels, classify as a "middle", otherwise "side". In this thesis main goal is involvement level classification, so class "middle" represent "probable involve" and class "side" represent "probable not involve". In our thesis, we try to increase performance of classification of involvement level, involve or not involve. This is achieved by integrating head location and head movement features. Firstly, head location features are calculated. Video frames are classified according to these features. After that, motion estimation and classification is applied to video frames of which head locations can't be found and video frames are in class "middle". With motion classification, if head movement is ±15 degree or ±30 degree, these frames are labeled as class "involve". If head movement is ±45 degree or ±60 degree, these frames are labeled as class "not involve". Motion estimation and classification is not applied to video frames, which are classified as "side" with head location classification. These frames are labeled as "not involve" directly. Pointing'04 database is used as a training and test dataset for motion classification. This database consists of 15 sets of images. In each set, there are 2 series (one of them for training and one of them for test) of 63 images of the same person with varying pan and tilt movement. The reason why we used database for training set is the difficulty of specifying the class label of video frames visually. But it is not true for head location classification, so the recorded real videos are used for training set in head location classification. In addition to the Pointing04 test database, recorded real videos are also used for the test set. As a classification method, Random Forest is preferred. Random forest is a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution of all trees in the forest. To report classification performance, accuracy has been the most common metric. But in real datasets where class distributions are not specific, a classifier that always predicts the most common class may deceptively yield high accuracy. So we used recall, precision and F1 score, which is the harmonic mean between precision and recall. Precision is the fraction of retrieved instances that are relevant while recall is the fraction of relevant instances that are retrieved. First of all, motion classification test was made with Pointing04 train and test set. The results were calculated for each direction separately in terms of F1 score. For pan movement (x direction) and tilt movement (y direction) binary classification results were 72% and 62% on average, respectively. For diagonal movement (both x and y direction) binary classification result was 62% on average. In addition to these tests, for pan movement which gives best results in motion classification test was also made with recorded real test video. When only motion features were used for classification, performance was 62% in terms of F1 score. When motion estimation and face detector are applied and obtained features are integrated, performance increased to 71%. When only head location features are used, classification performance was 71%. But classification, which is made with only head location features, gives that whether a class label is "probable involved" or "probable not involved". These classes are not enough to classify video frames involvement level. As a future work, to show the success of integration of head location and head movement features and SIFT flow algorithm, we will make some test with different videos which have different resolution, different direction and different people. Besides, to increase the performance of classification we have the option of adding some different features in addition to 5-D features which we used now.