---
library_name: transformers
license: apache-2.0
pipeline_tag: translation
language:
- en
- es
- fr
- de
- it
- pt
- hi
- zh
- ja
- ru
- ar
- multilingual
tags:
- translation
- multilingual
- moe
- mixture-of-experts
- fsdp
- seq2seq
- 8k-context
- document-packing
- pytorch
- open-weights
- free-for-all
widget:
- text: "Artificial intelligence empowers communities around the world to connect and collaborate."
example_title: "English to Spanish"
output:
text: "La inteligencia artificial permite a las comunidades de todo el mundo conectarse y colaborar."
- text: "Artificial intelligence empowers communities around the world to connect and collaborate."
example_title: "English to French"
output:
text: "L'intelligence artificielle permet aux communautés du monde entier de se connecter et de collaborer."
- text: "Artificial intelligence empowers communities around the world to connect and collaborate."
example_title: "English to German"
output:
text: "Künstliche Intelligenz ermöglicht es Gemeinschaften weltweit, sich zu verbinden und zusammenzuarbeiten."
extra_gated_prompt: "Welcome to Mythos AI! The 2.04B Sparse MoE pre-training is actively converging across 500+ languages. Please provide your details below to request repository access. To collaborate on GPU compute pooling, dataset curation, or enterprise API deployments, please join our official Discord: https://discord.gg/KKVN5BShGj"
extra_gated_fields:
Full Name: text
Email: text
Current Job or Organization: text
Skills & Background: text
How do you plan to use or contribute?: text
Discord Username: text
I agree to open research and collaborative guidelines: checkbox
---
---
# Mythos-2.04B-MoE
**A next-generation open-weights Sparse Mixture-of-Experts foundation model built for 500+ global languages.**
> [!IMPORTANT]
> ### 100% Free & Open-Source (Apache 2.0) — Zero Cost, No Paywalls
> **Mythos-2.04B-MoE is completely free for everyone.** There are no paywalls, no subscriptions, and no pay-as-you-go fees.
> * **Free Open Weights:** The model architecture, weights, configuration, and tokenizer are 100% open under the permissive Apache 2.0 license for unrestricted research, personal, and commercial use.
> * **Self-Host Anywhere:** Run inference freely on your local machine (consumer RTX 3060/3090/4090 GPUs, Apple Silicon, or free cloud instances) with complete privacy and zero cost.
> * **Join Our Discord:** Connect directly with the core team, compute sponsors, and researchers on [Discord](https://discord.gg/KKVN5BShGj) to collaborate on training and evaluation!
> [!NOTE]
> This repository contains model weights and configuration files for the pre-trained model in standard PyTorch and Hugging Face formats.
>
> These artifacts are compatible with PyTorch, Hugging Face Transformers, vLLM, SGLang, and distributed frameworks.
---
Today, Mythos AI officially introduces **Mythos-2.04B-MoE**, our flagship open-weights multilingual translation and understanding foundation model.
Built on the architectural foundation of **Sparse Mixture-of-Experts (SMoE)**, Mythos-2.04B-MoE delivers substantial gains across translation quality, low-resource language understanding, zero-shot cross-lingual transfer, and long-horizon document translation. Mythos-2.04B-MoE brings these advances to a compact, deployment-friendly sparse model: a native multilingual translation model that understands 500+ global languages, designed to carry complex sentence, paragraph, and multi-page document translations through to completion with greater reliability.
---
## 📋 How to Request Access & Collaborate (Name, Email, Job, Skills)
While the pre-training run is converging towards final weights release, we welcome individual researchers, compute partners, and enterprise collaborators to connect with us!
When requesting repository access or joining our **[Discord Server (https://discord.gg/KKVN5BShGj)](https://discord.gg/KKVN5BShGj)**, please provide:
1. **Full Name:** Your real name or developer handle.
2. **Email Address:** Your preferred contact email.
3. **Current Job / Organization / University:** Where you currently work, study, or conduct research.
4. **Skills & Area of Contribution:**
* ⚡ **Compute Contributor:** Donating idle GPU hours (RTX 3090/4090, A100, H100) or cloud credits for scaling 4B/7B/14B MoE runs.
* 🔬 **AI Research & Development:** Expertise in sparse MoE routing, loss balancing, context packing, and training optimization.
* 🌍 **Native Linguist & Evaluator:** Contributing parallel corpora or auditing translation accuracy for underrepresented languages.
* 💼 **Enterprise & Business Partnership:** Custom on-premise deployments, commercial translation integration, or grant sponsorships.
5. **Discord Username:** Your Discord handle so we can grant you your verified contributor role!
👉 **Official Discord Server:** [https://discord.gg/KKVN5BShGj](https://discord.gg/KKVN5BShGj)
*(Drop a message in `#welcome-and-links` or `#distributed-training` once you join!)*
---
## Comparison with Open and Proprietary Translation Models
Following the architectural comparison standards established by frontier open models, the table below compares the structural specifications, context capacity, language coverage, and hardware footprint of **Mythos-2.04B-MoE** against prevailing state-of-the-art models:
| Specification | **Mythos‑2.04B‑MoE** | TranslateGemma‑27B | TranslateGemma‑7B | NLLB‑200 (3.3B) | Google Cloud API |
| :--- | :---: | :---: | :---: | :---: | :---: |
| **Architecture** | **Sparse MoE (8E, Top‑2)** | Dense Transformer | Dense Transformer | Dense Enc-Dec | Proprietary LLM |
| **# Total Params** | **2.04B** | 27.2B | 7.0B | 3.3B | Closed |
| **# Active Params** | **678.7M** | 27.2B | 7.0B | 3.3B | Closed |
| **Context Window** | **8,192 tokens** | 2,048 tokens | 2,048 tokens | 1,024 tokens | Varies |
| **Supported Languages** | **500+** | 55 | 55 | 200 | 189 |
| **Min Inference VRAM** | **4 GB** | 56 GB | 16 GB | 8 GB | Managed Cloud |
| **License / Cost** | **100% Free (Apache 2.0)** | Community License | Community License | CC-BY-NC 4.0 | Paid Metered API |
---
### Key Architectural Insights:
1. **Unrivaled Multilingual Breadth (500+ Languages vs. 55 - 243)**:
* While Google Cloud supports 189 languages, Google Translate supports 243, and Google TranslateGemma-27B benchmarks 55 languages, **Mythos supports 500+ languages natively**.
* Over 250+ underserved African, Oceanic, Indigenous American, and Regional South/Central Asian dialects that have zero coverage in mainstream models are fully supported by Mythos with zero-shot cross-lingual transfer.
2. **Long-Horizon Context Handling (8,192 Tokens vs. 1k - 2k)**:
* Specialized translation models (NLLB-200 at 1,024 tokens, TranslateGemma at 2,048 tokens) suffer from context fragmentation when translating multi-paragraph documents, technical manuals, or legal contracts.
* Mythos features a native **8,192-token context window with Block-Diagonal Attention Packing**, preserving document discourse, technical terminology, and character pronoun consistency across entire chapters.
3. **Radical Compute and Memory Efficiency (Sparse MoE)**:
* Running TranslateGemma-27B requires 27 Billion active parameters per token, demanding expensive multi-GPU enterprise infrastructure.
* Mythos utilizes **Sparse Mixture-of-Experts with Top-2 routing across 8 SwiGLU experts**. It activates **only 678.7 Million parameters per token**, achieving the capacity of a multi-billion parameter model while running smoothly on modest hardware (single consumer RTX GPU, free Google Colab/Kaggle instances, or CPU).
---
## Model Architecture Overview
* **Type:** Encoder-Decoder Multilingual Translation Model with Sparse Mixture-of-Experts (SMoE)
* **Training Stage:** Pre-training
* **Language Model Architecture:**
* **Number of Parameters:** 2.04B (`2,037,643,264` Total Parameters)
* **Active Parameters / Token:** 678.7M (`678,688,768` Active Parameters)
* **Hidden Dimension:** 1,024
* **Token Embedding:** 128,000 (Byte-Level BPE)
* **Number of Layers:** 24 Layers (12 Encoder Layers + 12 Decoder Layers)
* **Hidden Layout:** 24 × (GQA Attention $\to$ SwiGLU MoE FFN)
* **Grouped Query Attention (GQA):**
* **Number of Attention Heads:** 16 for Q and 4 for KV (4× KV-Cache Compression)
* **Head Dimension:** 64
* **Rotary Position Embedding (RoPE) Dimension:** 64 ($\theta = 100,000.0$)
* **Feed-Forward Network (Sparse MoE):**
* **Number of Experts:** 8 Experts per block
* **Routing Mechanism:** Top-2 Active Experts with Switch-Transformer Capacity Capping (1.25 factor) & Load Balancing Loss
* **Intermediate Dimension (per Expert):** 3,072 (SwiGLU)
* **LM Output:** 128,000 (FP32 Chunked Linear Projection)
* **Context Length:** 8,192 tokens natively with Block-Diagonal Document Packing
---
## Quickstart: Free Local Inference (Self-Hosted)
### 1. Installation
```bash
pip install torch transformers tokenizers sacrebleu
```
### 2. Python Inference Code (100% Free & Offline)
```python
import torch
from tokenizers import Tokenizer
# Load Multilingual Tokenizer
tokenizer = Tokenizer.from_file("multilingual_tokenizer.json")
sos_id = tokenizer.token_to_id("[SOS]")
eos_id = tokenizer.token_to_id("[EOS]")
pad_id = tokenizer.token_to_id("[PAD]")
def translate_sentence(model, text, src_lang="eng", tgt_lang="fra", max_len=128, device="cuda:0"):
clean_tgt = tgt_lang.split("_")[0]
enc_input = f"<2{clean_tgt}> {text}"
tokens = tokenizer.encode(enc_input).ids
src_tensor = torch.tensor([tokens], dtype=torch.long, device=device)
src_mask = (src_tensor != pad_id).unsqueeze(1).unsqueeze(2)
with torch.no_grad():
enc_out = model.encode(src_tensor, src_mask)
gen_input = torch.tensor([[sos_id, tokenizer.token_to_id(f"<2{clean_tgt}>")]], device=device)
for _ in range(max_len):
cur_len = gen_input.size(1)
c_mask = torch.tril(torch.ones((cur_len, cur_len), dtype=torch.bool, device=device)).unsqueeze(0).unsqueeze(0)
dec_out = model.decode(gen_input, enc_out, src_mask, c_mask)
logits = model.project(dec_out[:, -1:])
next_tok = logits.argmax(dim=-1).item()
if next_tok == eos_id:
break
gen_input = torch.cat([gen_input, torch.tensor([[next_tok]], device=device)], dim=1)
translated_text = tokenizer.decode(gen_input[0].tolist()[2:])
return translated_text
```
---
## Training Infrastructure & Engineering
* **Distributed Engine:** PyTorch Fully Sharded Data Parallel (FSDP) in Full-Shard mode (`FULL_SHARD`).
* **Precision Policy:** Native **32-bit FP32 Master Weights** with FP16 compute and unscaled FP32 logits projection.
* **Zero-Host RAM Footprint:** Streaming disk-spooler architecture keeping host CPU memory strictly below **`< 1.0 GB`** throughout training.
* **Router Stabilization:** Switch-Transformer dynamic capacity factor capping (`capacity_factor = 1.25`) with load balancing weight (`0.03`) preventing single-expert token flooding.
---
## Community & License
Mythos is released under the **Apache 2.0 License** — it is 100% free for academic, personal, and commercial use with zero royalty or subscription fees.
* **Repository:** [AdithyanAI/Mythos_translation](https://huggingface.co/AdithyanAI/Mythos_translation)
* **Discord Community:** [Join Mythos AI on Discord](https://discord.gg/KKVN5BShGj)
* **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
* **Author / Maintainer:** Adithyan AI & Open Community Contributors
---
## Citation
If you use **Mythos-2.04B-MoE** or its architecture in your research or applications, please cite:
```bibtex
@misc{mythos2026multilingual,
author = {Adithyan AI and Community Contributors},
title = {Mythos-2.04B-MoE: A Free and Open-Source Sparse Mixture-of-Experts Large Language Model for 500+ Languages},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/AdithyanAI/Mythos_translation}}
}
```
