Biostatistical approaches to the assessment of contribution by A new factor into risk prediction models

Introduction: Risk prediction model is a regression method that is used for predicting clinical outcomes, such as disease or death, and including risk factors. Along with the developments in application fields and technological progress, discovery of new risk factors has become inevitable. Receiver Operating Characteristic (ROC) curve, that has a major role in including these new risk factors into the model and evaluating the performance of the model, has some limitations. The most important limitation of ROC is that the difference between predictive regression coefficients is highly conservative. To be used as a solution to this limitation, Pencina et al. (2008) proposed two methods which are based on reclassification tables and explaining the new factor's contribution to the risk model. These two methods are called the Net Reclassification Improvement (NRI) and Integrated Discrimination Improvement (IDI). In this study, the aim is to investigate the performance and compare the advantages and disadvantages of ΔAUC, NRI and IDI techniques that are nested models. Material and Method: In order to predict the death probabilities of 864 babies hospitalized in the Newborn Intensive Care Unit in Çukurova University Medical Faculty Balcalı Hospital between the years 2013 and 2015, SNAP-II and SNAPPE-II risk prediction models are used. Surfactant treatment for the baby and corticosteroid (celestone) treatment for the mother are added into the model as a new variable. In the simulation study, the insignificant variable is added to the fitted model and then several datasets are generated using different sample sizes (100, 250, 500 and 1000) and different prevalence values (0.10, 0.25 and 0.50) with 1000 replications. These datasets are used to calculate False Positive Ratios (FPR) of the ΔAUC, NRI and IDI methods. Results: Improvement in the performance of the model obtained by including both celestone and surfactant variables into the SNAP-II risk prediction model is decided to be significant. Nevertheless, Improvement in the performance of the model obtained by including celestone into the SNAPPE-II risk prediction model is founded significant only by continuous NRI. The simulation study showed that the FPR of the continuous NRI is the highest while FPR of the ΔAUC and IDI is the lowest. Moreover, it is concluded that categorical NRI has high FPR values at low prevalence levels and depending on the increase in prevalence values, FPR values tend to decrease. Conclusion: Whereas ΔAUC statistic is decided to be the approach that has the most tendency to accept H0 by yielding very conservative results, the continuous NRI behaves quite the opposite. IDI statistic is decided to be the most appropriate method since its FPR values are lower than the other two methods and it concludes that the performance increase is significant in the real dataset.