Action quality assessment with multivariate time series

Action quality assessment using computerized methods is considered to be a promising direction in objective evaluation of actions in several domains including health, sport and education. In a typical architecture for quality assessment, a classification or regression system is asked to assign a query action to a predefined category or a continuous label that determines its quality level. Such systems are still trained manually, and they may have inconsistent annotations. Hence, an attempt to categorize or quantify the quality level can be biased due to potentially scarce or skewed training data. In this thesis, we approach the quality assessment problem as a pairwise ranking task where we relatively assess two input actions to identify better performance instead of assessing their absolute levels. To this end, we propose a novel computational model that takes two action data in the form of multi-variate time-series acquired from motion sensors and reports the probability of a query sample having a better quality than a reference one. The ranking model is built upon an attention-enhanced Siamese Long Short-Term Memory (LSTM) Network fed by piecewise aggregate approximation of time-series data. A probabilistic ranking layer is proposed to make the final relative assessment. The pairwise model is further extended to create an empirical feature representation in a regression setup. The model is adopted in three different applications, namely, gait assessment in Parkinson's Disease (PD) patients using foot sensors, surgery skill assessment using kinematics sensors and diving quality assessment using estimated pose from video recordings. According to experimental results, the proposed model achieves higher assessment accuracy than the existing models for pairwise ranking in all common datasets. The new regression model with new ranking-based empirical feature representation also outperforms the existing models when applied in their experimental setup. The proposed model is further shown to be accurate in individual progress monitoring. The model that is developed in this thesis can be considered as a generic model for several pairwise ranking tasks provided that the inputs are in the form of multi-variate time-series signals. While LSTM layer makes the model applicable for all sequential signals, attention enhancement extends its ability to adopt novel signals obtained from different measurement modalities. Proposed rank layer with probabilistic loss function allows the Siamese model to handle relative comparison of inputs instead of their direct evaluation for similarity. This relative assessment approach may overcome the limitations of having consistent annotations to define quality levels and provide a more interpretable means for objective skill assessment. Moreover, the model allows monitoring the skill development of individuals by comparing two activities at different time points. We expect that this model will find a wide range of applications in several domains, but more particularly in sports and healthcare.