Compositional modelling of first-person actions as verb-noun streams using LSTM based late fusion strategies

Analysis of first-person videos involving human actions could help in the solutions of many problems. These videos include a large number of fine-grained action categories with hand-object interactions. In this thesis, compositional modeling of verb and object streams with various fusion strategies is proposed to recognize human actions in first-person videos. We utilize 3D Convolutional Neural Network model, C3D, for verb stream to model video-based features in multiple scales, and we utilize object detection model, YOLO, for object stream to model objects interacting with hand. Two fusion strategies are proposed to combine these two streams. In the first one, human actions are obtained by simple multiplication without learning. In the second, LSTM based models are proposed. Experimental results on EGTEA Gaze+ dataset obtained from these two different fusion strategies show that our composite models present promising results compared to the baseline action models.