| --- |
| license: apache-2.0 |
| language: |
| - it |
| library_name: echo-1.58 |
| tags: |
| - masked-diffusion |
| - ternary |
| - 1.58-bit |
| - quantization-aware-training |
| - bitnet |
| - italian |
| pipeline_tag: fill-mask |
| --- |
| |
| These are research artifacts accompanying the paper *Native Ternary |
| Quantization-Aware Training for Masked Diffusion Language Models*. They are 341M-parameter |
| masked-diffusion language models trained on Italian FineWeb-2 with a 32k SentencePiece |
| tokenizer. **This is not a production model.** At roughly 12 tokens per parameter neither the |
| ternary nor the full-precision model composes fluent text; free generation degenerates |
| identically at both precisions. Use these checkpoints for reproduction, infilling analysis, |
| and as paired baselines, not as downstream generators. |
|
|
| - **Architecture:** bidirectional masked-diffusion transformer, `d_model` 1024, 24 layers, 16 |
| heads, `d_ff` 2816, tied embeddings, 32001 vocabulary (mask token id 32000). |
| - **Format:** bfloat16, verified to reproduce the full-precision evaluation within 0.0003 |
| masked-CE of the fp32 master. |
| - **Code and reproduction:** https://github.com/lupodevelop/echo-1.58 |
| - **Tokenizer:** `spm_it.model` (included). |
|
|
| ## Evaluation (common protocol, sequence length 1024, identical seeded masks) |
|
|
| | 341M model | masked-CE | perplexity | vs FP16 twin | |
| |---|---|---|---| |
| | FP16 twin | 4.8100 | 122.7 | ceiling | |
| | Ternary baseline | 4.9852 | 146.2 | +19.2% | |
| | + continued distillation | 4.9125 | 136.0 | +10.8% | |
| | + from-scratch recipe | 4.9878 | 146.6 | +19.5% | |
|
|
| # Echo-1.58 341M, Ternary Baseline |
|
|
| The bare ternary masked-diffusion model, trained natively at 1.58 bits with BitNet-style QAT |
| and no recovery recipe. It is the starting point for the recovery experiments and the +19.2% |
| perplexity penalty against the FP16 twin. Post-training quantization of a full-precision model |
| to this precision collapses (26x perplexity for round-to-nearest); this model is what native |
| training buys instead. |
|
|
