Resolution of Turkish scalar implicatures by large language models

As for a scalar pair consisting of a weaker term and a stronger term, scalar implicatures are the pragmatic inferences drawn when the use of the weaker term denotes the negation of the stronger term. This study investigates the pragmatic capabilities of Large Language Models (LLM) in the resolution of scalar implicatures. As part of the study, a Natural Language Inference (NLI) dataset in Turkish, ImplicaTR, was developed, which is a four-class NLI dataset that enriches the conventional entailment-neutral-contradiction classification with a new class, implicature. With a total of 2,340 sets of sentences spanning five different linguistics categories, ImplicaTR has a size of 20,340 rows. Two experiments were conducted on ImplicaTR. In Experiment 1, various BERT models and generative models such as Gemma, Llama-2, and Mistral were inspected on pragmatic reasoning, and we found that LLMs can reason about scalar inferences with an accuracy over 98% on test dataset. As a result, we obtained an NLI model that does more fine-grained analysis. In Experiment 2, we carried out a linguistic inquiry within an ablation study to reveal the factors influencing the entailment and implicature resolution of the models. Our findings showed that the frequency of individual scalar items is positively correlated with the model's ability to resolve the pragmatic inferences. From the linguistic perspective, our study demonstrated that scalar reasoning is not solely a pragmatic process nor a lexical one; both mechanisms seem to play a role in implicature inferencing.