BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//LatentShift//latentshift.ai schedule//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:LatentShift · TabiBERT: A Large-Scale ModernBERT Foundation M
 odel for Turkish
X-WR-TIMEZONE:Europe/Istanbul
REFRESH-INTERVAL;VALUE=DURATION:PT12H
X-PUBLISHED-TTL:PT12H
BEGIN:VEVENT
UID:ed01-tabibert@latentshift.ai
DTSTAMP:20261007T125617Z
DTSTART:20261017T111500Z
DTEND:20261017T113500Z
SUMMARY:LatentShift · TabiBERT: A Large-Scale ModernBERT Foundation Model 
 for Turkish · Ebrar Kızıloğlu
DESCRIPTION:Short talk · 20 min · Turkish\n\nEbrar Kızıloğlu · Resear
 ch Assistant\, TABILAB\, Boğaziçi University\n\nTurkish Sign Language (T
 İD) interpretation.\n\nWe introduce TabiBERT (https://huggingface.co/boun
 -tabilab/TabiBERT)\, a state-of-the-art monolingual Turkish encoder based 
 on the ModernBERT architecture. Previously\, the Turkish NLP landscape lac
 ked a monolingual encoder trained from scratch that systematically incorpo
 rates modern architectural advances\, such as rotary positional embeddings
  and FlashAttention-based attention implementations. To bridge this gap\, 
 we trained TabiBERT on one trillion tokens sampled from an 84.88B-token mu
 lti-domain corpus containing web text (73%)\, scientific publications (20%
 )\, source code (6%)\, and mathematical content (0.3%). TabiBERT supports 
 a context length of 8\,192 tokens\, which is 16x that of the original BERT
  models\, and delivers up to a 2.65x inference speedup with significantly 
 reduced GPU memory overhead. To support rigorous evaluation\, we concurren
 tly developed TabiBench (https://huggingface.co/collections/boun-tabilab/t
 abibench)\, a comprehensive benchmark with 28 datasets across eight task c
 ategories. TabiBERT achieves a GLUE-style macro-average score of 77.58\, o
 utperforming the previous de facto standard\, BERTurk\, by 1.62 absolute p
 oints and establishing state-of-the-art results on five out of eight task 
 categories\, including major gains in question answering (+9.55 points).\n
 \nOur development process revealed several critical bottlenecks spanning c
 omputational infrastructure\, data quality\, and evaluation methodology.\n
 \nIn terms of computational and replicability challenges\, our initial pre
 -training trials on 8×H100 SXM GPUs experienced massive throughput degrad
 ation\, achieving only 30% of the speed documented in the ModernBERT paper
 . Hardware tests on 8×A100 GPUs yielded nearly identical performance\, co
 nfirming that the bottleneck lay entirely in the data loader rather than i
 n the accelerator hardware. Consultation with the ModernBERT authors revea
 led that high-throughput training requires a comprehensive pre-tokenizatio
 n pipeline. This hidden prerequisite highlighted a widespread problem rega
 rding replicability in large-scale modeling. Driven by this experience\, w
 e placed emphasis on maintaining high-quality documentation for our own pr
 e-training and fine-tuning procedures to facilitate future research and ad
 option.\n\nRegarding data quality and benchmarking challenges\, we discove
 red during downstream evaluation that several commonly used fine-tuning da
 tasets in Turkish NLP research were completely unstandardized. Most critic
 ally\, the field suffered from the lack of standardized data splits\, maki
 ng a true "apples-to-apples" comparison across different models entirely i
 mpossible. Furthermore\, several existing fine-tuning datasets exhibited q
 uality issues\, including inconsistent annotations\, labeling ambiguities\
 , and insufficient sample sizes. This systemic lack of reliable benchmarks
  led us to develop TabiBench\, in which we enforced a strict manual filter
 ing process to isolate high-fidelity evaluation targets and established co
 ncrete\, standardized train-test-validation splits to ensure reproducible 
 performance tracking.\n\nFinally\, we faced a distinct methodological chal
 lenge during token classification experiments (NER and POS tagging)\, in w
 hich initial subword-level evaluation yielded highly distorted results. Be
 cause different tokenizers segment the same text into varying numbers of s
 ubwords\, this discrepancy made subword-level scores incomparable across m
 odels\; architectures with higher token fertility could artificially infla
 te their scores by trivially predicting more fragmented tokens per word. T
 o address this\, we modified our methodology to use word-level evaluation\
 , aggregating predictions at word boundaries before computing metrics. Thi
 s approach ensures a fair comparison across models regardless of tokenizat
 ion strategy. Ultimately\, our work demonstrates that building a regional 
 language foundation model requires equal dedication to advanced architectu
 res\, explicit preprocessing transparency\, and robust evaluation methodol
 ogies.\n\nhttps://latentshift.ai/schedule/tabibert/
LOCATION:Bağlarbaşı Kongre ve Kültür Merkezi\, Boğaziçi Salonu\, Sel
 ami Ali Mah.\, Gazi Cad. No:22\, 34664 Üsküdar\, İstanbul
GEO:41.024366;29.033571
URL:https://latentshift.ai/schedule/tabibert/
CATEGORIES:LatentShift,Short talk
STATUS:CONFIRMED
TRANSP:OPAQUE
BEGIN:VALARM
ACTION:DISPLAY
TRIGGER:-PT10M
DESCRIPTION:Starts in 10 minutes · TabiBERT: A Large-Scale ModernBERT Foun
 dation Model for Turkish
END:VALARM
END:VEVENT
END:VCALENDAR