TR-MTEB: A comprehensive benchmark and embedding model suite for Turkish sentence representations

We introduce TR-MTEB, the first large-scale, task-diverse benchmark designed to evaluate sentence embedding models for Turkish. Covering six core tasks as classification, clustering, pair classification, retrieval, bitext mining, and semantic textual similarity, TR-MTEB integrates 26 high-quality datasets, including both native Turkish corpora and carefully curated machine-translated resources. To support embedding model development, we construct a 34.2 million pair Turkish sentence corpus derived from over 50 public datasets, applying multi-stage filtering to ensure semantic alignment. Using this corpus, we train two monolingual Turkish sentence embedding models from a \texttt{BERTurk} backbone: one with contrastive pretraining using Cached-MNRL and one further fine-tuned on TR-MTEB tasks. Despite being trained with significantly less data and computation resources than their multilingual counterparts, our models outperform the Turkish BERT baseline across all tasks and achieve competitive performance in classification and similarity, while highlighting limitations in cross-lingual alignment. All datasets, models, and code are publicly released to promote reproducibility and further research in Turkish natural language processing and low-resource sentence representation.