Polygon based feature extraction for human action recognition in videos

A new silhouette-based feature extraction method for the recognition and classification of human actions in videos is proposed. For this purpose, new image forms called "Motion History Image (HGG)" and "Poly-Silhouette (PoS)", a new "Polygonization Algorithm (PoG)" and a new "Polygon Encoding Algorithm (PoC)" have been developed. The proposed method is based on obtaining HGGs from "Silhouette Videos (SiVi)", converting them to polygon form with fewer corners and encoding them. Since traditional silhouette creation methods cannot meet the body integrity, to overcome this problem, the codes of Yolact++ were changed and human masks produced by the model is saved as SiVis. HGGs that better describe motion have been obtained by summing the silhouettes in non-consecutive SiVi frames according to bounding box centroids. The polygonization process has been applied to the HGG images. PoG quickly reduces the contour coordinates of the HGG to the desired size. The resulting polygonized HGG images are called PoS. The PoC algorithm takes the PoS image as a parameter and creates a vector with each edge, starting from the upper left corner coordinate, while tracing in counterclockwise direction. It calculates the related vector code by projecting the vector on to the area that are presented in the study. When the codes of all edge vectors of a PoS are recorded, the polygon code for the PoS is formed. An approach similar to the k-mer grouping technique in genetic algorithms has been developed in order to be able to process with long polygon codes and to create feature vectors of different lengths. This idea not only provided for the normalization of the code, but also for the adaptation of the code according to an adaptive kernel. Different lengths were chosen as k and groups were normalized by dividing them into the largest codes according to the k length. The proposed feature extraction method guarantees the generation of feature vectors of equal length on PoS, eliminating the need for the use of extra dimension reduction algorithms to overcome the size problem. Different k-mer lengths were tested on HMDB51 and UCF101 datasets and comparisons of "processing speed" and "classification accuracy" were made. The results of best k-mer and classification accuracy were compared with studies and it was found that more successful results were obtained. In addition, extra deep network applications were performed out on the yoga dataset created specifically for the study, and remarkable encouraging results were obtained.