TabiBERT: A Large-Scale ModernBERT Foundation Model for Turkish

We introduce TabiBERT (https://huggingface.co/boun-tabilab/TabiBERT), a state-of-the-art monolingual Turkish encoder based on the ModernBERT architecture. Previously, the Turkish NLP landscape lacked a monolingual encoder trained from scratch that systematically incorporates modern architectural advances, such as rotary positional embeddings and FlashAttention-based attention implementations. To bridge this gap, we trained TabiBERT on one trillion tokens sampled from an 84.88B-token multi-domain corpus containing web text (73%), scientific publications (20%), source code (6%), and mathematical content (0.3%). TabiBERT supports a context length of 8,192 tokens, which is 16x that of the original BERT models, and delivers up to a 2.65x inference speedup with significantly reduced GPU memory overhead. To support rigorous evaluation, we concurrently developed TabiBench (https://huggingface.co/collections/boun-tabilab/tabibench), a comprehensive benchmark with 28 datasets across eight task categories. TabiBERT achieves a GLUE-style macro-average score of 77.58, outperforming the previous de facto standard, BERTurk, by 1.62 absolute points and establishing state-of-the-art results on five out of eight task categories, including major gains in question answering (+9.55 points).

Our development process revealed several critical bottlenecks spanning computational infrastructure, data quality, and evaluation methodology.

In terms of computational and replicability challenges, our initial pre-training trials on 8×H100 SXM GPUs experienced massive throughput degradation, achieving only 30% of the speed documented in the ModernBERT paper. Hardware tests on 8×A100 GPUs yielded nearly identical performance, confirming that the bottleneck lay entirely in the data loader rather than in the accelerator hardware. Consultation with the ModernBERT authors revealed that high-throughput training requires a comprehensive pre-tokenization pipeline. This hidden prerequisite highlighted a widespread problem regarding replicability in large-scale modeling. Driven by this experience, we placed emphasis on maintaining high-quality documentation for our own pre-training and fine-tuning procedures to facilitate future research and adoption.

Regarding data quality and benchmarking challenges, we discovered during downstream evaluation that several commonly used fine-tuning datasets in Turkish NLP research were completely unstandardized. Most critically, the field suffered from the lack of standardized data splits, making a true "apples-to-apples" comparison across different models entirely impossible. Furthermore, several existing fine-tuning datasets exhibited quality issues, including inconsistent annotations, labeling ambiguities, and insufficient sample sizes. This systemic lack of reliable benchmarks led us to develop TabiBench, in which we enforced a strict manual filtering process to isolate high-fidelity evaluation targets and established concrete, standardized train-test-validation splits to ensure reproducible performance tracking.

Finally, we faced a distinct methodological challenge during token classification experiments (NER and POS tagging), in which initial subword-level evaluation yielded highly distorted results. Because different tokenizers segment the same text into varying numbers of subwords, this discrepancy made subword-level scores incomparable across models; architectures with higher token fertility could artificially inflate their scores by trivially predicting more fragmented tokens per word. To address this, we modified our methodology to use word-level evaluation, aggregating predictions at word boundaries before computing metrics. This approach ensures a fair comparison across models regardless of tokenization strategy. Ultimately, our work demonstrates that building a regional language foundation model requires equal dedication to advanced architectures, explicit preprocessing transparency, and robust evaluation methodologies.