ModernBERT-TR

We are releasing ModernBERT-TR. It has 150M parameters, was trained with masked-language-model pretraining, and supports 8,192-token inputs. Throughout this article, ModernBERT-TR refers to our new model release; we earlier released a 1,024-token model called ModernBERT-TR 1k. In addition to the base model, we are releasing a family of downstream task models for aspect-polarity classification, aspect-term extraction, personally identifiable information recognition, MASSIVE intent classification and slot filling, dense retrieval, cross-encoder reranking, and guardrail. For the guardrail task we also release Guardrail-TR, a ~405K-row Turkish prompt-safety dataset. Evaluation metrics are reported on a 0 to 100 scale.

The base encoder

Our model is based on the ModernBERT architecture. It has 22 transformer layers, width 768, 12 attention heads, and GLU feed-forward layers of width 1,152. Our cased WordPiece tokenizer has 50,008 tokens, similar to ModernBERT-TR 1k.

Attention alternates by layer. Layers 0, 3, 6, and every third layer thereafter use global attention; the other layers use a 128-token local window. Rotary position embeddings use theta 160,000 in global layers and 10,000 in local layers.

Training chronology

All pretraining runs used packed sequences, dynamic masking, BF16, seed 42, and StableAdamW. A step contained about 1.05M tokens: 1,024 sequences at length 1,024 during pretraining, then 128 sequences at length 8,192 during context extension and terminal decay.

In the chronology below, 1k pretraining refers to the 200B-token run used in this release. It is different from the earlier public model ModernBERT-TR 1k.

We ran several ablations on the 8k context extension and decay phases which scored slightly below the selected 8k terminal decay on internal transfer evaluations. These are not part of the released weights.

The released ModernBERT-TR is the result of the two averaging steps. First, we averaged the 1k short-decay and 1k low-LR continuation checkpoints in equal proportions. We then averaged that result equally with the 8k terminal-decay checkpoint. In the final model, the two 1k checkpoints contribute 25% each and the 8k checkpoint contributes 50%.

The merged model uses the 8k terminal-decay checkpoint’s configuration, preserving its RoPE theta settings.

In our Hugging Face repository, we share four named revisions:

We share these intermediate artifacts for controlled analysis and to allow for more stable continued pretraining or fine-tuning from a higher learning rate.

ModernBERT-TR draws 25% of its weights from the 1k low-LR continuation, 25% from the 1k short decay, and 50% from the 8k terminal decay.

Of the merge methods tested, only simple parameter averaging improved at least one of the checkpoints being merged. TIES, SFTM, and STAR task-vector variants performed worse. Large continued-pretraining changes and differing rotary-position settings may contribute, but our experiments do not establish a cause. TIES also failed within the 1k group, where the rotary-position settings did not change.

Base model evaluation

We evaluate ModernBERT-TR in three settings: TabiBench for broad task-specific fine-tuning across 28 tasks, TrGLUE for a fixed five-seed Turkish GLUE evaluation, and encoder-fast-eval for frozen-encoder linear probing. These benchmarks use different tasks, metrics, and scales.

ModernBERT-TR compared with BERTurk, TabiBERT, and mmBERT in three separate evaluation settings. mmBERT leads TabiBench, BERTurk leads TrGLUE, and ModernBERT-TR leads the frozen-encoder evaluation.

TabiBench category scores for four encoders. ModernBERT-TR leads text classification, natural-language inference, and information retrieval among these models; mmBERT leads the other five categories.

To compare results across metrics, we rank the models within each task and average those ranks by capability.

Mean rank across the 47 tasks, grouped by capability; lower is better. ModernBERT-TR has the best mean rank for classification, pairwise inference, and general retrieval, and ties TabiBERT on acceptability.

We share below a dotplot of all task scores for ModernBERT-TR and the three comparison encoders.

Task-level scores across the same 47 tasks. Pink marks ModernBERT-TR, gray dots mark the three comparison models. Bold labels indicate tasks where ModernBERT-TR leads.

ModernBERT-TR is 0.13 points above our prior release, ModernBERT-TR 1k, on TabiBench overall. mmBERT’s lead is largely due to their ~6 point lead in code retrieval tasks. Note that none of the benchmarks directly and reliably test long-context task performance which is a gap we find worth addressing in future work.

The downstream family

Every downstream model starts from the ModernBERT-TR base model. We release these models to showcase the capabilities of ModernBERT-TR and to provide end-users with useful ready-to-use artifacts or starting points for further fine-tuning with less data. We train all model parameters and use mean pooling for sequence-level tasks. We also release new datasets for some tasks.

Aspect-polarity classification

Our aspect-polarity model receives a (sentence, aspect) pair and predicts whether the sentence’s sentiment towards the aspect is negative, neutral, or positive.

Example of input and output for aspect-polarity classification.

Training combines about 10.5k M-ABSA Turkish aspect pairs, 11.5k rows from a Turkish clothing ABSA dataset, and 17.2k aspect pairs from DeepSeek-V4-Pro annotations of 12k real Turkish review sentences. The sentences come equally from e-commerce reviews, vitamins and supplements reviews, and movie reviews. We release this dataset as ABSA-TR.

Our recipe uses learning rate 3e-5, batch size 32, 15 epochs, 10% linear warmup followed by linear decay, weight decay 0.01, and label smoothing 0.1. Within each training run, the best M-ABSA validation macro-F1 checkpoint is retained. ABSA-TR’s development set is used to select the recipe and release checkpoint.

On ABSA-TR test split, the five-seed mean macro-F1 is 72.74±2.10; the released checkpoint scores 72.87. However, neutral F1 is ~36.

Aspect-term extraction

Our aspect-term extraction model predicts aspect spans using O, B-ASP, and I-ASP tags. We train it on 8.7k translated M-ABSA sentences and the aforementioned ABSA-TR dataset. Unlike polarity classification, we do not use the clothing dataset.

We align whitespace words to character spans, tokenize them with the release tokenizer, and label on their first WordPiece. We train our model for 15 epochs at learning rate 4e-5, batch size 32, 10% linear warmup and linear decay. We use validation span F1 for hyperparameter tuning and release-seed selection.

Inference needs to mirror training. Split the text into whitespace words, tokenize that word list with is_split_into_words=True, take the first WordPiece prediction for each word_id, then BIO-decode adjacent words into spans. This keeps an aspect such as Ankara'ya together even when its WordPieces include Ankara, ', and ya, while B-ASP followed by I-ASP becomes one multiword aspect span.

Example of first-WordPiece BIO labeling for aspect-term extraction.

The five-seed mean ABSA-TR exact entity-span F1 is 64.09±0.30; the released checkpoint scores 64.48 on ABSA-TR test and 69.46 on M-ABSA test.

Personally identifiable information recognition

Our PII recognition model is a text token classifier with 25 KVKK-oriented entity types and 51 BIO tags. Training mixes real documents with two kinds of synthetic documents. In the first, an LLM writes a Turkish document and marks entities inline, so the spans are known when the text is written; checksummed IDs such as TCKN and IBAN are filled in by a generator rather than invented by the model. In the second, those same labeled documents are rendered to images, distorted, run through Turkish Tesseract, and the known spans are realigned onto the OCR text. If a labeled character span cannot be mapped onto whitespace words, that document is dropped from word-level BIO training. The training split contains 7.8k documents and the validation split contains 63.

An example showing WordPieces grouped into word-level BIO decisions for the PII recognition model.

Real-document labels come from a three-model panel, an additional judge for spans proposed by only one model, and deterministic validation for identifiers such as Turkish identity numbers, tax numbers, IBANs, and card numbers. The test dataset uses a five-model panel and deterministic rules. It contains 300 documents and 5.7k spans, but covers only 17 of the 25 labels.

The model trains for 3 epochs at learning rate 2e-5, batch size 16, 10% linear warmup followed by cosine decay, and weight decay 0.01. We use weighted cross-entropy with normalized (inverse frequency)^0.3 class weights. The main score is character-span overlap micro-F2 on the test set. F2 weights recall twice as heavily as precision, because a missed identifier is worse than an over-tag. The three-seed mean F2 is 80.85±0.19; our released checkpoint scores 81.01. On the same test set, mean precision is 72.1 and recall is 83.4.

We plan to release the synthetic training set. We will publish the test set if we find a safe path to do so; otherwise, we will create a new test set and report the model’s performance on it.

MASSIVE intent classification and slot filling

We train our model on the Turkish MASSIVE dataset, which contains the human-localized MASSIVE 1.1 Turkish split with about 11.5k training, 2k validation, and 3k test utterances.

The model has a 60-class intent head and a 111-class token head representing O and BIO tags for 55 slot types. Training combines intent cross-entropy with label smoothing 0.1 and slot cross-entropy. We train for 15 epochs at learning rate 5e-5, head learning rate 1e-4, batch size 64, 10% linear warmup and linear decay, and weight decay 0.01. We retain the best validation exact-match checkpoint and share its two heads as an intent classification model and a slot filling model.

Intent classification maps the input to one of 60 labels.

Slot filling selects one BIO label per word, then joins adjacent B-X and I-X words into spans.

Five-seed test means are 88.21±0.16 intent accuracy, 85.43±0.14 intent macro-F1, 75.27±0.31 slot entity-F1, and 67.21±0.34 frame exact-match. Frame exact-match means a datapoint is only counted as correct if both the intent label and every slot span are correct.

Distillation

The model family uses three forms of distillation. As discussed above, aspect-based sentiment models learn from labels generated for real reviews. The embedding model matches teacher vectors, pairwise geometry, and task-specific similarity distributions. The reranker matches a teacher’s relative scores over candidate lists.

Distillation methods used to train the models include teacher annotation, vector matching, and listwise score matching.

Dense retrieval

Our embedding model returns one normalized 768-dimensional vector using mean pooling. Retrieval queries use the fixed instruction Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:{text}; documents and symmetric tasks use raw text.

Prompted queries and unprompted passages become normalized vectors; cosine similarity is used to measure alignment.

To train our model, we use Qwen3-Embedding-8B to generate normalized 4,096-dimensional targets for 7.9M generic Turkish text rows. During embedding transfer, a 4,096-to-768 projector and the student learn through cosine matching, batch-similarity matching, and a teacher-ordered relative-similarity margin. This phase uses global batch size 128, 4.9k projector-only updates, then 8.5k joint updates; peak projector and backbone learning rates are 5e-4 and 5e-5. The projector is trained on 1.7M instances; the unfrozen backbone is trained on 1.1M. The projector is then discarded.

We then fine-tune on several tasks at once: Turkish retrieval examples with teacher-chosen hard negatives, Turkish question answering, natural-language inference, semantic textual similarity, supervised contrastive classification, and continued matching to teacher vectors. Each retrieval example has one relevant passage and seven hard negatives; training pulls the query toward the relevant passage, pushes it away from the negatives, and also matches the teacher’s scores over those candidates using InfoNCE and KL divergence. We train with learning rate 2e-5, weight decay 0.01, maximum length 512 for retrieval, and 128 for the shorter-text tasks because the training data does not contain long documents. We retain three checkpoints: a retrieval run with 12k steps, a multi-task run at 24k steps, and a multi-task run with added English data at 8k steps. We average their weights equally to produce the released embedding model.

We evaluate our model on the MTEB(Turkish) benchmark, which uses 26 tasks, maximum length 2,048, and an unweighted mean of task scores. The release scores 68.14, compared with 67.17 for Turkish E5 Large and 68.59 for its 8B teacher. It leads Turkish E5 Large in three of six categories and is 3.764 times smaller by exact local parameter counts. We do not evaluate long-context embedding quality beyond 2,048 tokens.

We share detailed evaluation results in the following tables. Qwen3-Embedding-8B is shown for reference only.

Cross-encoder reranking

Our reranker scores (query, document) pairs. The embedding model first retrieves a shortlist; the reranker scores and sorts only those documents, not the full corpus.

The query and document are encoded together to produce one relevance score.

To form each retrieval training example, Qwen3-Embedding-8B compares a query with every document in that dataset’s corpus. We pair a known relevant document with seven hard negatives: documents not relevant that the embedding model ranks highly. Candidates scoring at least 95% as high as the relevant document are discarded because they may also be relevant. Qwen3-Reranker-8B then scores only the eight selected pairs, and the student learns the teacher’s relative scores. Training draws 4k examples each from FiQA, MS MARCO, NFCorpus, and Quora, in addition to all 894 SciFact examples, for a total of 16.9k examples. Test queries are excluded.

Counter-argument training adds 3.6k machine-translated ArguAna examples, repeated three times for balance. Each contains the annotated best counter-argument, the four same-stance passages scored highest by the reranker trained using the earlier recipe, and three randomly sampled opposing-stance passages. The loss ranks the best counter-argument first and other opposing-stance passages above same-stance passages to prevent the model from treating support as a counter-argument.

We train the model for 2 epochs with maximum length 512, learning rate 1.5e-5, batch size 16, weight decay 0.05, and 10% linear warmup followed by linear decay. We use 400 examples for validation and retain the checkpoint with the highest NDCG@8.

The cross-encoder reorders a dense retriever’s top k documents improving on its first-stage recall.

We rerank the embedding model’s top 100 documents. Across the six reported tasks, mean NDCG@10 improves by 8.38 points.

Both models use a maximum length of 2,048 in this evaluation.

Guardrail

Our guardrail model scores Turkish user prompts only. It does not classify assistant responses. A single classification head produces 11 sigmoid probabilities: one overall unsafe score and one score for each of 10 hazard categories. Categories are multi-label, so a prompt may violate more than one. We do not prescribe decision thresholds; downstream systems choose their own cutoffs.

Example of input and multi-label safety scores for the guardrail model.

Our taxonomy is inspired by the MLCommons hazards taxonomy. It covers 10 unsafe categories, including violent and non-violent crimes, identity-based hate, harassment, adult sexual content, CSAE, self-harm, injection and jailbreak attempts, political misinformation, and privacy violations. Training also includes about 207K safe prompts. Many of these are hard negatives that look risky but are not harmful, so the model does not over-refuse benign requests.

We release the training data as Guardrail-TR. It contains 405K Turkish user prompts with binary safe/unsafe labels and multi-label categories, stratified 80/10/10 into train, validation, and test.

Curation pipeline for Guardrail-TR: sources, preprocessing, Turkish-localized unredaction, translation, two-phase cultural adaptation, LLM-as-a-judge labeling, and rare-class synthesis.

To build Guardrail-TR, we draw on 14 open-source datasets spanning both Turkish and English resources, combining them through a multi-stage curation workflow. Both streams are deduplicated and quality-filtered: Turkish sources with heuristic checks, English sources through an analogous preprocessing and filtering step. About 28K rows contain redacted spans, which we unredact by substituting names, cities, and related entities drawn from a Turkish localization pool. We then resample underrepresented classes from single-turn, moderation-flagged LMSYS and WildChat prompts. Next we translate the English prompts with TranslateGemma-27B. This is followed by a cultural adaptation phase in two steps, with Gemma-4-31B, in a CultureGuard-style process: we first detect culturally grounded content, then rewrite it for a Turkish context (~26K rows).

After merging the Turkish and English streams, which resulted in a 401K row dataset, a three-model jury labels each prompt for the 10 categories: Gemma-4-26B-A4B, Qwen3.6-35B-A3B, and gpt-oss-120b. A category is kept if at least two judges agree on it, or if at least one judge agrees with the original source label. Finally we add about 4K synthetic rows for underrepresented classes with Gemma-4-31B, bringing the release to 405K.

We fine-tune from ModernBERT-TR for 5 epochs with backbone learning rate 2e-5, head learning rate 1e-4, layer-wise LR decay 0.9, adamw_torch_fused, cosine scheduling, and class-weight β 1.0. On the Guardrail-TR test split, the released checkpoint scores 93.0 F1 on the binary safe/unsafe decision, 91.7 macro AUPRC, and 88.6 weighted F1 across all hazard categories jointly.

Packaging and deployment

We share our models in three formats to allow deployment in different scenarios.

PyTorch weights can be used with Transformers, or with Sentence Transformers for the embedding model and the reranker.

Each repository also contains ONNX exports in multiple precisions under onnx/, allowing the model to be used with Transformers.js and on phones through ONNX Runtime Mobile.

Additionally, we share our models in encoderfile format, allowing them to be used with no additional dependencies.

We showcase two demos to demonstrate the deployment of our models. The first is a web application that runs the PII recognition model entirely in the browser. The second is a native Android application that runs the intent and slot models fully on the phone, using quantized ONNX exports.

The web demo of our PII recognition model.

Intent and slot predictions using our demo Android application.

Artifacts

The models and datasets described in this article are published under the ModernBERT-TR Collection. Each downstream task model also has a self-contained encoderfile repository:

Alongside the model family, we release two datasets, ABSA-TR for aspect-based sentiment analysis and Guardrail-TR for Turkish prompt-safety classification.

We hope this work improves Turkish natural language processing and helps the community build better models.