---
base_model: EleutherAI/pythia-410m
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- kalavai
- specialist
- mixture-of-experts
- decentralized-training
- yoruba
---

# KALAVAI — Yoruba Specialist (pythia-410m, seed 137)

Fine-tuned [EleutherAI/pythia-410m](https://huggingface.co/EleutherAI/pythia-410m) on **Yoruba** data as part of the [KALAVAI](https://arxiv.org/abs/2603.22755) decentralized cooperative training protocol.

- **Paper:** [KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training](https://arxiv.org/abs/2603.22755)
- **GitHub:** [mechramc/Kalavai](https://github.com/mechramc/Kalavai)

## Paper results

Yoruba PPL 41.9→7.7 (5.4×), Welsh 102.7→22.1 (4.6×), Tamil 4.2→3.0. MoE fusion of 4 specialists: +21.76% over best specialist (seeds 137+2026).

## How to use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("mechramc/kalavai-cross-lingual-yoruba-specialist-seed137")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-410m")
```

This model is one specialist in a KALAVAI cooperative.
To reproduce the MoE fusion results from the paper, load multiple domain specialists
and combine them with a trained MoE router (see the [paper](https://arxiv.org/abs/2603.22755)
and [GitHub](https://github.com/mechramc/Kalavai) for details).

## Citation

```bibtex
@article{kumaresan2026kalavai,
  title     = {{KALAVAI}: Predicting When Independent Specialist Fusion Works
               --- A Quantitative Model for Post-Hoc Cooperative {LLM} Training},
  author    = {Kumaresan, Ramchand},
  journal   = {arXiv preprint arXiv:2603.22755},
  year      = {2026},
  url       = {https://arxiv.org/abs/2603.22755}
}
```