Domain adaptation of transformer-based language models in low-resource languages: A study on Turkish legal texts

Despite advancements in natural language processing, further research is needed for low-resource languages. In particular, for languages with a complex language structure and limited resources like Turkish, the capabilities of Transformer-based language models in terminology-rich domains such as law have emerged as a significant research area. In this study, the domain adaptation process of Transformer-based language models with Turkish legal texts was examined and the effects of this process on the performance of the models are evaluated. Legal texts collected from the Turkish Council of State decision query page were used for the study, and these texts were structured for masked language modeling and named entity recognition tasks to create unique datasets. For domain adaptation, the mBERT model, which has a multilingual structure including Turkish, and the BERTurk model customized for Turkish were used. During the domain adaptation process, the vocabulary of the models was expanded with legal words and a masked language modeling approach was applied in which legal words and random words were masked. The models were fine-tuned for the named entity recognition task and the results were evaluated. The results show that the domain-adapted models achieved higher performance for the named entity recognition task in legal texts compared to the baseline models. The domain adaptation resulted in an F1 Score increase of 3.147% for the mBERT model and 1.130% for the BERTurk model. This study demonstrates that domain adaptation in low-resource languages is an important process in improving the performance of Transformer-based language models.