Investigation of the levels of process and result data to prediction complex problem solving with regression trees

The present study examines the predictive capacity of process and result data in complex problem-solving skills, employing different machine learning algorithms. The data set comprises 915 participants from the Programme for International Student Assessment (PISA) 2012. The process data was obtained from the log file of the first question of the Climate Control unit in the problem-solving assessment in PISA 2012. Various cognitive and affective characteristics were used as result data. The following machine learning algorithms were employed: single regression tree, bagging, random forest, gradient boosting, conditional inference tree, bagging based on conditional inference trees, random forest based on conditional inference trees, and boosting algorithms based on conditional inference trees. The main results indicated that process data exhibited moderate prediction performance, result data exhibited moderate-good prediction performance, and process+result data exhibited good prediction performance. (2) Gradient boosting based on conditional inference trees performed better on process data, both gradient boosting and gradient boosting based on conditional inference trees performed better on result data, and gradient boosting performed better on process+result data. However, there were no important differences between the metric values of all methods. (3) In general, the most important variables in the process data were the VOTAT strategy score and total time, in the outcome/result data the mathematics literacy score and reading literacy score, and in the process+result data the mathematics literacy score and VOTAT strategy score. It is notable that the mathematics literacy score was particularly effective.