Mitosis detection from breast histopathology images with transformer architectures

Breast cancer is the most common type of cancer among women. In histopathological image analysis, the detection and count of mitotic cells serve as a crucial biomarker for grading cancer and predicting its aggressiveness. Manual identification of mitotic cells by pathologists is a time-consuming and challenging process. With advancements in deep learning architectures, numerous automated mitosis detection methods have been proposed. However, most mitosis detection methods have poor generalization ability across image fields. This thesis aims to develop a novel method that addresses the challenges of mitosis detection in breast histopathology images, providing high accuracy, objectivity, and rapid results. For this purpose, three different automated mitosis detection methods based on the transformer architecture, which has recently demonstrated remarkable success in medical applications, have been proposed. The first proposed model is an enhanced version of DETR for mitosis detection, referred to as Mi-DETR. The Mi-DETR model is optimized with a CSPResNeXt backbone, a layer reduction strategy, and the CIoU loss function. In the second proposed model, the SegFormer architecture is utilized for mitosis segmentation. The ICPR14 and TUPAC16 datasets used in the experiments were converted into segmentation datasets with the SAM model for mitosis segmentation. Various encoders, ranging from MiT-B0 to MiT-B5, were tested and fine-tuned for precise mitosis segmentation. The third proposed model, a hybrid method called Mi-SegDeTr, combines the strengths of DETR and SegFormer for both mitosis detection and segmentation. All experiments were conducted using the ICPR14 and TUPAC16 breast histopathology datasets. As a result, the proposed Mi-DETR model achieved F1-scores of 0.921 on the ICPR14 dataset and 0.950 on the TUPAC16 dataset. The SegFormer model yielded F1-scores of 0.8962 on the ICPR14 dataset and 0.8272 on the TUPAC16 dataset. Finally, the Mi-SegDeTr model achieved F1-scores of 0.9044 for segmentation and 0.9658 for detection on the ICPR14 dataset, and 0.9189 for segmentation and 0.9684 for detection on the TUPAC16 dataset. The results obtained on both datasets demonstrate that the proposed Mi-SegDeTr model performs competitively with state-of-the-art mitosis detection methods.