System design for predicting the quality of the output of image segmentation models

The accuracy of the extracted diagnostic information largely depends on the quality of the segmentation masks obtained from medical images. There is limited work on segmentation quality control, which generally involves training large and complex models. In this study, the quality of the output of deep segmentation models is predicted using automatically computed thresholds. The relationship between the calculated threshold values and segmentation qualities was analyzed. Experiments were conducted using three different methods: a 4-decoder segmentation model, which is an ensemble method that can produce well-calibrated outputs for image segmentation, a method with test time adaptation to the single-decoder version of this model, and a method with a dropout layer added to the single-decoder version. Using the thresholding methods of Otsu, Li and Yen, thresholds are automatically computed to threshold the probability maps of the masks predicted by the segmentation model. A linear regression model was used to predict the quality scores, which are the Dice similarity scores, with the thresholds calculated for each segmentation mask. The proposed methods are validated using two cardiac MRI image datasets, STACOM 2013 and CINE MRI, focusing on segmenting the left atrium, and DRIVE fundus image dataset used to segment retinal vessels. STACOM 2018, IOSTAR and STARE datasets, which are datasets out of the distribution of the data on which the model was trained, were also added to the test set to evaluate the generalization ability of the model. The predicted and real quality scores obtained using the proposed methods generally show a strong correlation for the Otsu threshold, an average correlation for the Li threshold, and a weaker correlation for the Yen threshold. Among the threshold values obtained with automatic thresholding methods, the Otsu threshold was found to be the most successful among the threshold values that can be used to estimate segmentation quality at low cost for well-calibrated models.