Tooth segmentation in panoramic radiographs using preprocessing and deep learning

Dental segmentation is the process of defining the boundaries of teeth and the anatomical structures that surround them in dental radiographic images. This process separates dental structures into distinct boundaries on digital images, making them suitable for analysis. In particular, dental segmentation forms the basis of numerous clinical applications in artificial intelligence (AI)-assisted systems, such as tooth detection, caries analysis, identification of periodontal diseases, and orthodontic planning. Segmentation supports clinical decision-making processes by providing accurate, rapid, and repeatable analyses for diagnosis and treatment planning. In this context, developing automated dental segmentation systems has become essential, particularly in regions with a shortage of specialists. This study examines how data augmentation, preprocessing techniques, and attention blocks affect the performance of the U-Net deep learning model for segmentation in dental radiographs. The study aims to determine the most effective combination of image processing filters, data augmentation strategies, and deep learning architectures for automated tooth segmentation. The study uses a dataset of 1,000 dental radiographs from Tufts University. The dataset is divided into 85% for training and 15% for testing. Preprocessing techniques such as contrast-limited adaptive histogram equalization (CLAHE) and Otsu thresholding were applied to enhance image contrast and reduce noise. Grad-CAM-based heatmaps were employed to interpret the segmentation focus regions. Additionally, data augmentation methods, including mirroring and rotations of 5° and 10°, as well as translations, were employed to enhance the model's generalization capability. The proposed U-Net model was trained for 100 epochs using the Adam optimization algorithm. Model performance was evaluated using the Dice score, Intersection over Union (IoU), and Pixel Accuracy (PA) metrics. Without data augmentation, the Attention U-Net model incorporating CLAHE preprocessing and attention blocks achieved the highest performance with Dice, IoU, and PA scores of 91.01%, 83.51%, and 97.86%, respectively. When data augmentation with CLAHE preprocessing was applied, the classical U-Net model trained with augmented data using the mirroring method achieved the highest performance, with 91.02% Dice, 83.51% IoU, and 97.86% PA values. The findings demonstrate that applying appropriate data augmentation and preprocessing techniques, as well as attention blocks, significantly improves model reliability and accuracy, especially when the dataset is limited. This study aims to support healthcare professionals by automating tooth segmentation and diagnosis using deep learning models enhanced with preprocessing methods and attention blocks. This approach is valuable for healthcare facilities in remote areas with limited access to specialist radiologists.