Development and implementation of machine learning based diagnosis approaches in serous effusion cytopathology

Serous effusions are a frequently encountered specimen type in cytopathological evaluation. Cytopathological evaluation is time-consuming, exhausting, and causes intra-pathologist and inter-pathologists variability. In the thesis work, machine learning-based automated diagnosis approaches in serous effusion cytopathology are proposed. First, a new residual learning-based convolutional neural network model was proposed for stain normalization in cytopathological images. It was observed that the proposed model significantly increases the success of the nuclei segmentation methods. Second, a new network architecture based on the ensemble of fully convolutional neural networks was proposed for nuclei segmentation. It was seen that the success of segmentation achieved with the proposed ensemble network architecture exceeded the success achieved by the models alone. Third, modern convolutional object detectors were proposed for nuclei detection. As a result of improvements in the YOLOv3 architecture, it was observed that the proposed object detectors provide faster detection compared to other object detectors with a robust detection success. Finally, popular convolutional neural network models in the literature were analyzed for serous cell classification, and an optimum convolutional neural network model was proposed. The proposed model has the least learnable parameters, thus significantly reduces the test time. In this thesis work, a novel data set consisting of pleural effusion cytopathology images was also proposed for each of the preprocessing, detection, segmentation, and classification steps.