Interpretation of procedural hand gestures using computer vision

World Health Organization (WHO) recommends a hand washing procedure for hand hygiene against infections consisting of various hand gestures performed for certain periods of time. Especially in the health sector, hand washing according to this procedure is inspected by authorized personnel periodically. In this work, a computer vision-based system was developed to inspect performing of 8 hand washing gestures derived from WHO's recommendations and their washing times autonomously. Two new datasets were created using color, depth data from camera and synthetic data from rendered images. Color (RGB) images, projection images from depth data, point cloud and merged point cloud data models were obtained from datasets. With these frame-based data models, hand washing gestures were classified using 16 different neural network models, including models which are spatio-temporal. Also, on synthetic data, classification with 4 and 32 classes according to soap amount was performed. The proposed system uses multi thread-based pre-processing for camera images in real time for efficiency. In classifications, %100 accuracy was achieved with camera dataset, %56,5 and %48,5 accuracies were achieved with render dataset.