You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Welcome to Mythos AI! The 2.04B Sparse MoE pre-training is actively converging across 500+ languages. Please provide your details below to request repository access. To collaborate on GPU compute pooling, dataset curation, or enterprise API deployments, please join our official Discord: https://discord.gg/KKVN5BShGj

💻 GitHub · 🤗 Hugging Face · 💬 Discord · 🤝 Discussions

⚡ Quickstart Code · 🏛️ Architecture · 💼 Enterprise & Collaboration

Mythos-2.04B-MoE

A next-generation open-weights Sparse Mixture-of-Experts foundation model built for 500+ global languages.

100% Free & Open-Source (Apache 2.0) — Zero Cost, No Paywalls

Mythos-2.04B-MoE is completely free for everyone. There are no paywalls, no subscriptions, and no pay-as-you-go fees.

Free Open Weights: The model architecture, weights, configuration, and tokenizer are 100% open under the permissive Apache 2.0 license for unrestricted research, personal, and commercial use.

Self-Host Anywhere: Run inference freely on your local machine (consumer RTX 3060/3090/4090 GPUs, Apple Silicon, or free cloud instances) with complete privacy and zero cost.

Join Our Discord: Connect directly with the core team, compute sponsors, and researchers on Discord to collaborate on training and evaluation!

This repository contains model weights and configuration files for the pre-trained model in standard PyTorch and Hugging Face formats.

These artifacts are compatible with PyTorch, Hugging Face Transformers, vLLM, SGLang, and distributed frameworks.

Today, Mythos AI officially introduces Mythos-2.04B-MoE, our flagship open-weights multilingual translation and understanding foundation model.

Built on the architectural foundation of Sparse Mixture-of-Experts (SMoE), Mythos-2.04B-MoE delivers substantial gains across translation quality, low-resource language understanding, zero-shot cross-lingual transfer, and long-horizon document translation. Mythos-2.04B-MoE brings these advances to a compact, deployment-friendly sparse model: a native multilingual translation model that understands 500+ global languages, designed to carry complex sentence, paragraph, and multi-page document translations through to completion with greater reliability.

📋 How to Request Access & Collaborate (Name, Email, Job, Skills)

While the pre-training run is converging towards final weights release, we welcome individual researchers, compute partners, and enterprise collaborators to connect with us!

When requesting repository access or joining our Discord Server (https://discord.gg/KKVN5BShGj), please provide:

Full Name: Your real name or developer handle.
Email Address: Your preferred contact email.
Current Job / Organization / University: Where you currently work, study, or conduct research.
Skills & Area of Contribution:
- ⚡ Compute Contributor: Donating idle GPU hours (RTX 3090/4090, A100, H100) or cloud credits for scaling 4B/7B/14B MoE runs.
- 🔬 AI Research & Development: Expertise in sparse MoE routing, loss balancing, context packing, and training optimization.
- 🌍 Native Linguist & Evaluator: Contributing parallel corpora or auditing translation accuracy for underrepresented languages.
- 💼 Enterprise & Business Partnership: Custom on-premise deployments, commercial translation integration, or grant sponsorships.
Discord Username: Your Discord handle so we can grant you your verified contributor role!

👉 Official Discord Server: https://discord.gg/KKVN5BShGj
(Drop a message in #welcome-and-links or #distributed-training once you join!)

Comparison with Open and Proprietary Translation Models

Following the architectural comparison standards established by frontier open models, the table below compares the structural specifications, context capacity, language coverage, and hardware footprint of Mythos-2.04B-MoE against prevailing state-of-the-art models:

Specification	Mythos‑2.04B‑MoE	TranslateGemma‑27B	TranslateGemma‑7B	NLLB‑200 (3.3B)	Google Cloud API
Architecture	Sparse MoE (8E, Top‑2)	Dense Transformer	Dense Transformer	Dense Enc-Dec	Proprietary LLM
# Total Params	2.04B	27.2B	7.0B	3.3B	Closed
# Active Params	678.7M	27.2B	7.0B	3.3B	Closed
Context Window	8,192 tokens	2,048 tokens	2,048 tokens	1,024 tokens	Varies
Supported Languages	500+	55	55	200	189
Min Inference VRAM	4 GB	56 GB	16 GB	8 GB	Managed Cloud
License / Cost	100% Free (Apache 2.0)	Community License	Community License	CC-BY-NC 4.0	Paid Metered API

Key Architectural Insights:

Unrivaled Multilingual Breadth (500+ Languages vs. 55 - 243):
- While Google Cloud supports 189 languages, Google Translate supports 243, and Google TranslateGemma-27B benchmarks 55 languages, Mythos supports 500+ languages natively.
- Over 250+ underserved African, Oceanic, Indigenous American, and Regional South/Central Asian dialects that have zero coverage in mainstream models are fully supported by Mythos with zero-shot cross-lingual transfer.
Long-Horizon Context Handling (8,192 Tokens vs. 1k - 2k):
- Specialized translation models (NLLB-200 at 1,024 tokens, TranslateGemma at 2,048 tokens) suffer from context fragmentation when translating multi-paragraph documents, technical manuals, or legal contracts.
- Mythos features a native 8,192-token context window with Block-Diagonal Attention Packing, preserving document discourse, technical terminology, and character pronoun consistency across entire chapters.
Radical Compute and Memory Efficiency (Sparse MoE):
- Running TranslateGemma-27B requires 27 Billion active parameters per token, demanding expensive multi-GPU enterprise infrastructure.
- Mythos utilizes Sparse Mixture-of-Experts with Top-2 routing across 8 SwiGLU experts. It activates only 678.7 Million parameters per token, achieving the capacity of a multi-billion parameter model while running smoothly on modest hardware (single consumer RTX GPU, free Google Colab/Kaggle instances, or CPU).

Model Architecture Overview

Type: Encoder-Decoder Multilingual Translation Model with Sparse Mixture-of-Experts (SMoE)
Training Stage: Pre-training
Language Model Architecture:
- Number of Parameters: 2.04B (2,037,643,264 Total Parameters)
- Active Parameters / Token: 678.7M (678,688,768 Active Parameters)
- Hidden Dimension: 1,024
- Token Embedding: 128,000 (Byte-Level BPE)
- Number of Layers: 24 Layers (12 Encoder Layers + 12 Decoder Layers)
- Hidden Layout: 24 × (GQA Attention $\to$ SwiGLU MoE FFN)
- Grouped Query Attention (GQA):
  - Number of Attention Heads: 16 for Q and 4 for KV (4× KV-Cache Compression)
  - Head Dimension: 64
  - Rotary Position Embedding (RoPE) Dimension: 64 ($\theta = 100,000.0$)
- Feed-Forward Network (Sparse MoE):
  - Number of Experts: 8 Experts per block
  - Routing Mechanism: Top-2 Active Experts with Switch-Transformer Capacity Capping (1.25 factor) & Load Balancing Loss
  - Intermediate Dimension (per Expert): 3,072 (SwiGLU)
- LM Output: 128,000 (FP32 Chunked Linear Projection)
- Context Length: 8,192 tokens natively with Block-Diagonal Document Packing

Quickstart: Free Local Inference (Self-Hosted)

1. Installation

pip install torch transformers tokenizers sacrebleu

2. Python Inference Code (100% Free & Offline)

import torch
from tokenizers import Tokenizer

# Load Multilingual Tokenizer
tokenizer = Tokenizer.from_file("multilingual_tokenizer.json")
sos_id = tokenizer.token_to_id("[SOS]")
eos_id = tokenizer.token_to_id("[EOS]")
pad_id = tokenizer.token_to_id("[PAD]")

def translate_sentence(model, text, src_lang="eng", tgt_lang="fra", max_len=128, device="cuda:0"):
    clean_tgt = tgt_lang.split("_")[0]
    enc_input = f"<2{clean_tgt}> {text}"
    tokens = tokenizer.encode(enc_input).ids
    src_tensor = torch.tensor([tokens], dtype=torch.long, device=device)
    src_mask = (src_tensor != pad_id).unsqueeze(1).unsqueeze(2)

    with torch.no_grad():
        enc_out = model.encode(src_tensor, src_mask)
        gen_input = torch.tensor([[sos_id, tokenizer.token_to_id(f"<2{clean_tgt}>")]], device=device)
        
        for _ in range(max_len):
            cur_len = gen_input.size(1)
            c_mask = torch.tril(torch.ones((cur_len, cur_len), dtype=torch.bool, device=device)).unsqueeze(0).unsqueeze(0)
            dec_out = model.decode(gen_input, enc_out, src_mask, c_mask)
            logits = model.project(dec_out[:, -1:])
            next_tok = logits.argmax(dim=-1).item()
            if next_tok == eos_id:
                break
            gen_input = torch.cat([gen_input, torch.tensor([[next_tok]], device=device)], dim=1)

    translated_text = tokenizer.decode(gen_input[0].tolist()[2:])
    return translated_text

Training Infrastructure & Engineering

Distributed Engine: PyTorch Fully Sharded Data Parallel (FSDP) in Full-Shard mode (FULL_SHARD).
Precision Policy: Native 32-bit FP32 Master Weights with FP16 compute and unscaled FP32 logits projection.
Zero-Host RAM Footprint: Streaming disk-spooler architecture keeping host CPU memory strictly below < 1.0 GB throughout training.
Router Stabilization: Switch-Transformer dynamic capacity factor capping (capacity_factor = 1.25) with load balancing weight (0.03) preventing single-expert token flooding.

Community & License

Mythos is released under the Apache 2.0 License — it is 100% free for academic, personal, and commercial use with zero royalty or subscription fees.

Repository: AdithyanAI/Mythos_translation
Discord Community: Join Mythos AI on Discord
License: Apache 2.0
Author / Maintainer: Adithyan AI & Open Community Contributors

Citation

If you use Mythos-2.04B-MoE or its architecture in your research or applications, please cite:

@misc{mythos2026multilingual,
  author       = {Adithyan AI and Community Contributors},
  title        = {Mythos-2.04B-MoE: A Free and Open-Source Sparse Mixture-of-Experts Large Language Model for 500+ Languages},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/AdithyanAI/Mythos_translation}}
}

Downloads last month: -; Downloads are not tracked for this model. How to track