Utilizing explainable artificial intelligence approaches to transformer based models in medical image analysis

In recent years, deep learning–based methods have gained significant prominence in the field of medical image analysis, particularly in disease diagnosis and clinical decision-support systems. The examination of complex tissue structures in high-resolution medical images has increased the need for explainable and reliable machine learning approaches beyond traditional methods. The overall objective of this thesis is to develop explainable and generalizable classification models by integrating convolution and transformer-based architectures with Explainable Artificial Intelligence (XAI) techniques across different medical imaging domains (histopathology, dermatology, and otoscopy). In the first phase, Convolutional Neural Network, Vision Transformer (ViT), and EfficientNet-B1 architectures were compared for the automatic classification of lung cancer histopathology images, and Local Interpretable Model-agnostic Explainable (LIME) method was applied for explainability. Additionally, a second study involving FourierNet (FNet) combined with LIME was conducted on lung, colon, and breast cancer datasets to evaluate the medical relevance of model outputs, which was confirmed through pathologist feedback. In the second phase, ViT and Data-efficient Image Transformer (DeiT) models were trained on skin lesion and tympanic membrane (TM) image datasets to investigate the generalizability of transformer-based models across different clinical domains. For skin lesion classification, the two best-performing XAI methods—Gradient-weighted Class Activation Map (Grad-CAM) and Layer-wise Relevance Propagation (LRP)—were fused to propose a novel hybrid explainability approach based on Principal Component Analysis. For otoscopic images, another hybrid strategy was introduced by integrating Attention Rollout (AR) and LRP using Canonical Correlation Analysis. Explainability performance was quantitatively evaluated using deletion and insertion causal metrics. The results demonstrate that the ViT model achieved the highest accuracy, precision, and area under the curve (AUC) across all datasets. On the skin lesion dataset, ViT achieved AUC values of 0.9192 (binary classification) and 0.9784 (multi-class classification), while on the TM dataset the model reached the highest AUC of 0.9976. The proposed hybrid explainability approaches improved the reliability of model decisions by yielding lower deletion and higher insertion scores compared to individual XAI methods. In conclusion, the models developed within current thesis provide both high classification performance and reliable explainability across different medical imaging domains, thereby enhancing clinician confidence and increasing the transparency of clinical decision-support systems. Future studies will focus on improving clinical applicability through the integration of multi-center datasets, multimodal data fusion, and advanced causal analysis strategies.