MameLoshnLM: Yiddish Language Model and Evaluation Benchmark

A Purpose-Built Language Model for Yiddish

Despite Yiddish's centuries-old literary heritage, the language has been largely absent from modern AI systems. This paper introduces MameLoshnLM, an 8-billion-parameter language model specifically trained for Yiddish—the first open-source model of its kind. The key finding: a model trained on carefully curated Yiddish data significantly outperforms general multilingual models, even those several times larger, and better captures the language's defining grammatical and vocabulary patterns.

The problem the authors tackle is stark. Yiddish, spoken by hundreds of thousands of people globally, has rich printed and literary traditions but minimal presence in digital form. When Yiddish does appear in large web-scale datasets used to train multilingual models, it's often mixed with errors: machine translations, misidentified text, or outright noise. This corrupted signal leads multilingual models to learn distorted versions of Yiddish. General-purpose models therefore perform poorly on Yiddish-specific tasks, and there's been no benchmark to measure progress systematically.

Method: Clean Data Plus Targeted Evaluation

The authors built two critical resources from scratch. Oytser is a pretraining corpus combining contemporary Yiddish web sources with classical literature—selected and cleaned to minimize noise and misclassification. Kashes is a multi-task benchmark covering four domains: translation (to/from English), linguistic analysis (part-of-speech tagging, named entity recognition), information extraction (relation extraction, question answering), and general language understanding.

Using Oytser, the authors continued pretraining Llama 3.1 8B—a general-purpose open-source model—on Yiddish-specific data. This approach (called continued pretraining or adapter training) is more practical than training from scratch, since it leverages the broad knowledge already in Llama while specializing the model for Yiddish.

Results and Broader Implications

MameLoshnLM outperforms open-source multilingual baselines of similar scale across Kashes tasks. More tellingly, the model captures Yiddish's distinctive morphological and lexical patterns better than general models. For instance, the authors show that MameLoshnLM learns proper Yiddish inflections and word relationships, whereas multilingual models trained on noisy web data conflate or misrepresent them.

This finding reveals a systemic issue: scaling to many languages via web crawling degrades performance on low-resource and historically underrepresented languages. Clean, curated data for a single language—even in smaller quantity—beats large multilingual datasets polluted with noise.

The paper's significance extends beyond Yiddish. The authors present a practical template: for any language with rich offline traditions but limited digital footprint (Old English, Sanskrit, Ladino, and others), similar approaches could unlock progress. The work demonstrates that addressing language diversity in NLP isn't merely about ethical inclusivity—it's also a technical problem requiring language-specific data curation and evaluation, not just bigger multilingual models.