Recognizing students' faces in the classroom and estimating their engagement via computer vision techniques in real time

Aim of this study is to track students in a classroom via webcam; detect, track and recognize their faces; and detect their engagement based on head pose estimation. For this purpose, we developed a real time C++ application by utilizing OpenCV and Dlib computer vision and machine learning libraries. Face recognition module of our application allows us to create a face recognition training set for each student. Our application captures images of the student in front of the webcam in every 5 seconds while he/she is changing his/her head pose and facial expressions. Then, face recognition model is created via Local Binary Patterns Histograms method by using the face recognition training set. In student engagement detection module, we solve Perspective-n-Point problem for each student to estimate their head rotation angles (pitch, yaw and roll). Assuming camera and the teacher are in the same direction, our application detects student engagement if students' head rotation points out the camera or not. Success of face recognition and head pose estimation parts are tested with UPNA Head Pose Database. Local Binary Patterns Histograms method is preferred in our application, which gave the best accuracies. Head pose estimation tests showed that the angles returned from our computer vision based method are nearly the same as the angles measured by sensors. Student engagement detection part is also tested with UPNA Head Pose Database. We extracted random 1000 images from the database. 5 human labelers annotated each image as "Engaged" or "NotEngaged". Half of the annotated images are used for training and others are used for testing. Engagement classifiers were trained with the labelled dataset. Experiments were made with Random Forest, Decision Tree, SVM and KNN machine learning algorithms to classify student engagement. SVM classifier achieved 72.4% accuracy on the labelled dataset.