Comparison and usage of local feature based methods for 3d face recognition

Facial recognition is one of the most routine and comfortable processes performed by people in their daily life. Could this also apply to computers? This question has led to the emergence of automatic face recognition, one of the most popular areas of computer vision.With the development of laser scanning technology, 3D point clouds have become easy to obtain. Thus, facial recognition has become a popular field of study by using a three-dimensional point cloud against the constraints of automatic face recognition using two-dimensional images. Face recognition is a popular research area, since it has numerous applications, including law enforcement, surveillance systems, border security, access control, and entertainment systems. The accuracy of 2D face recognition using facial images is influenced by many factors such as pose exchange, lighting condition, expression difference, and occlusion (data loss due to glasses, hair-beard, hand or other object). In order to overcome the these problems, 3D face recognition has been proposed as an alternative or complementary method instead of the conventional 2D face recognition. The aim of the thesis is to approach 3D face recognition processes from a different dimension. In this context, the facilities of using automatic 3D local keypoint detector algorithms in face recognition are being investigated. The thesis is thought to be both a guideline for the future work of this study and a guide for the different studies on this subject. In the scope of the thesis, face recognition algorithm was developed using 3D keypoint based methods. As an application data, face data belonging to 10 people were modeled in 3D by using a laser scanner. The faces were scanned at a density of 1100 dots/$inc^{2}$. The scans are 2 natural facial expressions and 1 laughing facial expression. A total of 30 point clouds were collected from 10 people. Thus, a 3D face database has been created for querying. Point cloud data only contains X, Y, Z coordinates. The algorithm consists of 3 steps. In the first step, 3D points are defined on the point clouds using ISS and LSP methods. The LSP determines whether there is a feature point depending on the local neighborhood of a point in the point cloud. The method determines the feature of 3D point based on 'Shape Index'. Shape Index is a quantitative measure at that point, defined by the maximum and minimum principal curvatures at a point. In the ISS method, the neighborhood is determined for each point by first using a radius sphere. The weight of each point is calculated according to the number of neighboring points. In the second step, key points are defined using Point Feature Histograms (PFH) and Fast Point Feature Histograms (FPFH) histogram methods. Thus, the feature histogram of each is obtained. Point Feature Histograms (PFH) are multidimensional histograms created using the different properties of each pair of points within the neighborhood of a keypoint. 3D coordinate (X, Y, Z) values and surface normals are used as input data. Each point is defined by p position vector and n surface normal. Fast Point Feature Histograms (FPFH) is the accelerated form of the PFH method. The method consists of two steps. The method consists of two steps. In the first step; The support of a keypoint are determined using a sphere. The geometric relationship between the neigbouring points in the sphere and the keypoint is calculated. The histogram, which consists only of the geomatric features between the keypoint and the neighboring points, is called Simplified Point Feature Histogram (SPFH). In the second step; the SPFH is calculated for each adjacent point of the keypoint. The generated histogram is weighted by dividing it to the euclidean distance between the key point and the neighboring point. Finally, the average of the histogram of the keypoint and the weighted histogram of the neighboring points are summed. Thus the Fast Point Feature Histogram of the keypoint is obtained. The method of histogram generation is the same as PFH. In the third step, the keypoints in different point clouds are matched using the feature histograms obtained. Statistical methods are used to compare generated histograms. Thus, the two closest similar points between the different point clouds are matched. In previous comparative studies, the most successful statistical matching method was found to be Kullback-Leiber Divergence. For this reason, the Kullback-Leiber divergence method is used in this thesis. Algorithms for 3D face recognition system were developed in MATLAB environment. One of the most important steps of the algorithms is to determine the support of the points. In this thesis study, the neighborhood radius is determined as 4 mm. Thus, it is aimed that the calculation load is not too much while the geometric properties are determined adequately. The 3D face recognition system was developed based on the identification application. Accordingly, a gallery was created with 10 3D human face data. It is expected that the system will match the different model 3D face data given from the outside with the face of the right person in the gallery. The system is arranged as a combination of a keypoint detector and a keypoint descriptor algorithm. The matching process is applied separately for ISS-PFH, ISS-FPFH, LSP-PFH and LSP-FPFH algorithms. In the first case, a database was created from the scans with natural face expressions of each person. As input to the system, a second scan of the natural face of a person in the database is given. Once the keypoints have been detected and described, the individual similarity of each scan in the gallery and the model scan has been determined. The corresponding points counts of the four algorithm pairs are shown in the charts. In the latter case, scanning with a laughing facial expression of a person in the database system is given as input. Thus, face expression differences has also been examined. As a results, in the natural face expression, ISS-PFH algorithm, 9 out of 10 people; 7 out of 10 people with ISS-FPFH algorithm; with the LSP-PFH algorithm, 8 out of 10 people; with the LSP-FPFH algorithm, 8 out of 10 people are correctly defined. Both the PFH and FPFH algorithms specify the precise geometric properties of the face in a small area. However, PFH is better in this regard as expected. When PFH is used, more key points are matching between the two point clouds. It has been determined that the person with the wrong matching moves during the scan. The small movements cause the geometry of the face to change, so the recognition process takes place in the wrong way. Considering the number of faces recognized, it is seen that ISS-PFH is the best method to detect the right person. When the cases where different face expressions are given to the system are examined, the ISS-PFH algorithm has 5 out of 10 persons; The ISS-FPFH algorithm has 3 out of 10 people; The LSP-PFH algorithm correctly identified 4 out of 10 and the LSP-FPFH algorithm correctly identified 4 out of 10 people. The ISS-PFH algorithm also performed best in situations with different facial expressions The positional accuracy of the matched points has been examined. ICP was applied to the matching point clouds for this purpose. This process has only been performed in recognition applications using natural facial expressions. These operations were performed in the MATLAB environment. Euclidean distance between corresponding keypoints in the two point cloud is calculated. It has been accepted that the points are shorter than 10 mm. When root mean square errors of correct point matches are examined, there is no significant difference between the methods. In all methods a root mean square error of about 3 mm was determined with an accuracy of 10 mm. The ISS algorithm finds about 1/4 less points than the LSP algorithm. However, the number of correctly matched points with a slight difference is greater in ISS-based algorithms. The difference between keypoint descriptor algorithms has been determined. The correct matching rate for PFH is up to 60% with 10 mm error, while FPFH histograms are around 25% - 30%. When all scans are examined, the algorithm pair having the most accurate matching ratio is ISS-PFH. Second is the LSP-PFH algorithm pair. There was no significant difference between ISS-PFH and LSP-PFH. However, when the ISS-FPFH and LSP-FPFH algorithms are examined, only a small number of correct matches are obtained. At the same time geometric differences affect the performance of the FPFH algorithm significantly. However, the methods used are only effective in recognizing faces with natural facial expressions. If the face expression is different, the algorithms used are not successful. For this reason, the use of algorithms alone for different facial expressions is not sufficient, and the use of other algorithms should be investigated.