Segmentation of papillary thyroid carcinoma from fine needle aspiration cytology samples using deep learning algorithms

Papillary thyroid carcinoma (PTC) is the most common malignancy among endocrine system tumors and accounts for a large proportion of thyroid cancers worldwide. Early diagnosis and accurate diagnosis are among the most critical factors directly affecting the success of treatment. In particular, the importance of rapid and reliable decision support systems is increasing, as much as expert interpretation, in the interpretation of images obtained from cytological samples. In recent years, with advances in digital pathology and the development of artificial intelligence-based algorithms, automatic analysis applications in medical images have come to the fore. In this thesis study, the segmentation of PTC samples obtained from FNA (fine needle aspiration) cytology images was targeted. The dataset used in the study was created by the study's principal investigator and labeled in detail with the support of a pathologist and a biomedical engineer. The images have a resolution of 2448x1920 and are classified as benign or malignant. The segmentation process was performed only on malignant cases, with the aim of clearly delineating the boundaries of the target region. For deep learning-based segmentation, the YOLOv8-seg and new-generation YOLOv11-seg models based on the YOLO architecture were preferred, and both models were trained separately and evaluated using performance metrics such as mAP50 and mAP50-95. According to the results obtained, the YOLOv11-seg model demonstrated more successful segmentation performance with values of 97.8% mAP50 and 70.1% mAP50-95. This study presents an original and applicable method that will contribute to the use of AI-supported diagnostic systems in the field of digital pathology.