Klovis-442M — French Language Model
A 442M-parameter French language model using a Recurrent-Depth Transformer architecture, fully designed and trained from scratch by Eric Houzelle.
Every component — architecture, training pipeline, and inference engine — was written in PyTorch without relying on any pre-trained weights or third-party model code. Trained end-to-end on a single NVIDIA L40S GPU.
Key Facts
| Parameters | 442M |
| Architecture | Recurrent-Depth Transformer (RDT) |
| Language | French |
| Tokenizer | CamemBERT (camembert-base, 32k vocab) |
| Context window | 512 tokens |
| Chat format | ChatML |
| Training hardware | 1x NVIDIA L40S |
| SFT val_loss | 1.56 |
| License | Apache 2.0 |
Quick Start
pip install transformers torch safetensors sentencepiece
Conversational Mode (ChatML)
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Klovis-ai/Klovis-442M-french-130426"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
prompt = (
"<|system|>\n"
"Tu es un assistant intelligent. Réponds directement et précisément.<|end|>\n"
"<|user|>\n"
"Que peut-on visiter à Paris ?<|end|>\n"
"<|assistant|>\n"
)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
inputs["input_ids"],
max_new_tokens=200,
temperature=0.5,
top_p=0.85,
top_k=50,
do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Special Tokens
| Token | Role |
|---|---|
<|system|> |
Start of system message |
<|user|> |
Start of user message |
<|assistant|> |
Start of assistant response |
<|end|> |
End of turn |
Architecture — Recurrent-Depth Transformer
Klovis uses a Recurrent-Depth Transformer (inspired by OpenMythos/Parcae, Prairie et al. 2026), where a single Transformer block is applied iteratively:
Input → [Prelude: 4 layers] → [Shared Block x 8 steps] → [Coda: 4 layers] → Output
| Component | Detail |
|---|---|
| Embedding dim | 1536 |
| Prelude layers | 4 (unique) |
| Coda layers | 4 (unique) |
| Recurrent steps | 8 |
| Query heads | 24 |
| KV heads (GQA) | 6 |
| FFN hidden dim | 8192 |
| FFN activation | SwiGLU |
| Normalization | RMSNorm (pre-norm) |
| Position encoding | RoPE |
| LTI Injection | Linear Time-Invariant state coupling |
| ACT Halting | Adaptive Computation Time (per-position early exit) |
| Depth LoRA | Rank-32 adapters per recurrent step |
| Weight tying | Input embeddings <-> output projection |
Why Recurrent-Depth?
The RDT architecture achieves 442M effective parameters with significantly fewer unique parameters by reusing a single Transformer block across 8 depth steps. This enables:
- Deeper reasoning through iterative refinement
- Dynamic compute via ACT (positions that are "easy" halt early)
- Step specialization via Depth LoRA adapters
Training
Phase 1 — Pre-training
| Data | French texts (Wikipedia FR, FineWeb-2, CulturaX, C4) |
| Optimizer | AdamW (beta1=0.9, beta2=0.95, wd=0.1) |
| Scheduler | Linear warmup -> cosine decay |
| Precision | Mixed precision (bfloat16) |
| Block size | 512 tokens |
Phase 2 — Supervised Fine-Tuning (SFT)
Fine-tuned on curated French conversational datasets across 25 epochs, with prompt masking (only assistant tokens contribute to the loss).
| Epochs | 25 |
| Learning rate | 2e-5 |
| Effective batch size | 128 (16 x 8 gradient accumulation) |
| Final val_loss | 1.56 |
| Chat format | ChatML |
SFT Datasets
| Dataset | Samples |
|---|---|
| kurakurai/luth-sft (multiple subsets) | ~300k |
| angeluriot/french_instruct | 20k |
| Houzeric/everyday-conversations | Custom |
| Houzeric/physics-FR-qa-dataset | 5k |
| le-leadboard/MATH_LVL5_fr | 1.4k |
Example Outputs
Here are real outputs from the model (temperature=0.2, top_p=0.7, top_k=30):
User: Bonjour
Klovis: J'espère que vous allez bien ! Je suis ravi de partager mon expérience avec vous aujourd'hui. Si vous avez des questions spécifiques ou si vous souhaitez approfondir mes connaissances dans ce domaine, n'hésitez pas à me le faire savoir !
User: Que puis-je visiter à Paris ?
Klovis: La Tour Eiffel est une structure emblématique située dans le cœur historique de Paris. Construite pour l'Exposition universelle de 1889, elle offre une vue imprenable sur la ville. Cette avenue pavée offre une expérience unique de shopping et d'histoire, avec ses nombreux cafés, restaurants et boutiques de luxe qui témoignent de son riche passé. Les visiteurs peuvent profiter d'une promenade matinale ou d'un dîner gastronomique dans un restaurant étoilé Michelin.
User: Explique-moi ce qu'est la gravité.
Klovis: La gravité est une force qui agit sur tout objet ou chose, comme l'air ou l'eau dans son ensemble. Les effets de cette accélération peuvent varier considérablement selon les conditions atmosphériques. En résumé, la gravité est un concept fondamental en physique moderne, représentant la gravité comme un phénomène dynamique influencé par des facteurs tels que les mouvements tectoniques, les forces gravitationnelles et les champs électromagnétiques.
The model produces fluent, well-structured French with rich vocabulary. Factual accuracy is limited (as expected for a 442M model trained from scratch) — responses may contain hallucinations or approximations.
What to Expect
With 442M parameters, Klovis produces:
- Grammatically correct, fluent French
- Well-structured responses (numbered lists, bold headers, paragraphs)
- Relevant vocabulary across science, culture, history
- Coherent conversational interactions
Known limitations:
- Factual responses may be incorrect or fabricated (hallucinations)
- Mathematical reasoning is limited
- Context limited to 512 tokens
- French only
This model demonstrates what a single developer can achieve with a novel architecture at small scale. It is not intended to replace larger models for production use.
License
Apache 2.0 — https://www.apache.org/licenses/LICENSE-2.0
Designed and trained by Eric Houzelle.
- Downloads last month
- 7