Zero-shot and few-shot named entity recognition in environmental sciences domain

Novel architectures in natural language processing enable to transfer knowledge of the model for specific tasks. For many downstream tasks, training the model from scratch has become unnecessary since transfer learning can be leveraged for such cases. This can be achieved by finetuning a pretrained Large Language Models (LLM). In this study, a lightweight version of BERT, DistilBERT which is pretrained to predict next sentence was fine-tuned to handle Named Entity Recognition, as one of the most important information extraction task in context of textual data. Transfer learning also enable to transfer knowledge of the model to unseen domains. In this context, we created a domain-specific dataset in the environmental sciences domain. Also, to recognize specific entities, custom NER labels for entities in environmental sciences domain have been defined. To evaluate transfer learning ability of the model, zero-shot, one-shot and ten-shots learning procedures have been conducted on created dataset. To improve transfer learning, we have pre-trained the model a generic Turkish dataset. Finally, artificially generated data that specific to environmental sciences domain have been combined with our created dataset to improve the prediction performance of the model in zero-shot and few-shot setups. In the study, pretraining the model with generic dataset and introducing artificially generated dataset evaluated individually and together. In addition, presence of semantically related entities in the dataset have been investigated and improvements in prediction performance regardless of shot number are seen. The evaluation of tests demonstrates promising results and enlightens improvements in terms of transfer learning.