Detection of prostate cancer with CNN and transfer learning methodsenhanced with fine tuning

Cancer is one of the most high-risk diseases for humans. Prostate cancer is the second most common cancer in men after lung cancer and early diagnosis is vital. Artificial intelligence technologies have begun to be utilized in the diagnosis of prostate cancer, obtaining more effective and accurate results and avoiding potential errors in human-oriented methods. In this study, in order to improve the classification performance in prostate cancer diagnosis, transfer learning method and fine-tuning processes are applied, which have higher success and learning capability with less training data, unlike machine learning methods. Accuracy results were obtained with various CNN architectures with a feature extraction approach to a two-class dataset consisting of prostate cancer MRI images with 'significant' and 'not-significant' classes. In order to increase these rates, pre-trained transfer learning models were used and the Densenet201 model achieved the highest accuracy result with 98.63% with the combination of cross-validation method and RMSProp optimization method. The proposed transfer learning model achieved an improvement of about 26% compared to the feature extraction method. As part of the study on a different prostate dataset, we first developed a basic Sequential model, designed a pyramid-like architecture by changing the number of neurons in the layers of the basic model we developed, and observed performance improvement by fine-tuning and transfer learning in this model. In addition, in order to compare the results, the recently popular Vision Transformer (ViT) and the MaxViT v2 model, which is a hybrid combination of CNN and Vision Transformer, were also evaluated. As a result of these comparisons, the highest classification accuracy of 96.77% was obtained from the Fine-tuned Enhanced model. The results of the study show that the ability of the deep learning model to recognize different diseases increases significantly with transfer learning and fine-tuning.