---
license: apache-2.0
language:
- it
library_name: echo-1.58
tags:
- masked-diffusion
- ternary
- 1.58-bit
- quantization-aware-training
- bitnet
- italian
pipeline_tag: fill-mask
---

These are research artifacts accompanying the paper *Native Ternary
Quantization-Aware Training for Masked Diffusion Language Models*. They are 341M-parameter
masked-diffusion language models trained on Italian FineWeb-2 with a 32k SentencePiece
tokenizer. **This is not a production model.** At roughly 12 tokens per parameter neither the
ternary nor the full-precision model composes fluent text; free generation degenerates
identically at both precisions. Use these checkpoints for reproduction, infilling analysis,
and as paired baselines, not as downstream generators.

- **Architecture:** bidirectional masked-diffusion transformer, `d_model` 1024, 24 layers, 16
  heads, `d_ff` 2816, tied embeddings, 32001 vocabulary (mask token id 32000).
- **Format:** bfloat16, verified to reproduce the full-precision evaluation within 0.0003
  masked-CE of the fp32 master.
- **Code and reproduction:** https://github.com/lupodevelop/echo-1.58
- **Tokenizer:** `spm_it.model` (included).

## Evaluation (common protocol, sequence length 1024, identical seeded masks)

| 341M model | masked-CE | perplexity | vs FP16 twin |
|---|---|---|---|
| FP16 twin | 4.8100 | 122.7 | ceiling |
| Ternary baseline | 4.9852 | 146.2 | +19.2% |
| + continued distillation | 4.9125 | 136.0 | +10.8% |
| + from-scratch recipe | 4.9878 | 146.6 | +19.5% |

# Echo-1.58 341M, Ternary Baseline

The bare ternary masked-diffusion model, trained natively at 1.58 bits with BitNet-style QAT
and no recovery recipe. It is the starting point for the recovery experiments and the +19.2%
perplexity penalty against the FP16 twin. Post-training quantization of a full-precision model
to this precision collapses (26x perplexity for round-to-nearest); this model is what native
training buys instead.