Evaluation of local explainability methods in Turkish text classification tasks

The primary goal is to evaluate the explainability of transformer models for Turkish text classification tasks, focusing on local explainability techniques. With the advancement of complex encoder-based neural networks, which often act as "black boxes", there is a growing need for quantitative evaluation of existing explainability techniques. This study addresses this need by exploring local explainability methods that clarify how models reach their decisions. We use BERT-based models, specifically BERTurk and TurkishBERTweet, and apply local explainability techniques such as SHAP, LIME, and Integrated Gradients (IG). We evaluate these techniques based on their ability to preserve crucial information and provide insight into the model's decision-making through aspects like Mean of Probabilities and Incremental Deletion by comparing with a baseline approach. These aspects aim to calculate the faithfulness of the explanations. Our results demonstrate that the gradient-based technique, IG, effectively identifies salient tokens that correlate with the class of the full text. The research also explores the impact of the thresholding level on explainability evaluation aspects and compares the performance across two explainability techniques, IG and LIME. The study suggests that the IG method is effective in generating salient explanations when we use an encoder-based model in Turkish text classification tasks as it adeptly captures and highlights the nuanced contributions of context-dependent words and phrases, which are important in understanding Turkish morphological complexity.