README.md

---
license: apache-2.0
language:
- it
library_name: echo-1.58
tags:
- masked-diffusion
- ternary
- 1.58-bit
- quantization-aware-training
- bitnet
- italian
pipeline_tag: fill-mask
---

# Echo-1.58 · 341M · Ternary, Continued Distillation

The recommended ternary checkpoint from *Native Ternary Quantization-Aware Training for Masked
Diffusion Language Models*. A 341M-parameter masked-diffusion language model trained natively
at 1.58 bits on Italian FineWeb-2, then recovered by distilling from its full-precision twin.

> **Research artifact, not a production model.** At roughly 12 tokens per parameter neither the
> ternary nor the full-precision model composes fluent text; free generation degenerates
> identically at both precisions. Use it for reproduction, infilling analysis, and as a paired
> baseline.

![Recovery across scales](https://huggingface.co/lupodevelop/echo-mdlm-341m-ternary-continued/resolve/main/recovery-scale.png)

## What this checkpoint is

Starting from the bare ternary baseline, this model was distilled from the FP16 twin for a
further 1B tokens with a learnable per-channel scale (0.07% extra memory). It recovers **41% of
the ternary-to-FP16 gap** in cross-entropy (48-49% on held-out test and out-of-domain Italian
Wikipedia) and retains about **92% of FP16 infilling accuracy** at every mask ratio, in a model
5.4x smaller than its twin. The paper's central practical finding is that this post-hoc route
scales while applying the same recipe from the first step does not.

## Evaluation (common protocol, sequence length 1024, identical seeded masks)

| 341M model | masked-CE | perplexity | vs FP16 twin |
|---|---|---|---|
| FP16 twin | 4.8100 | 122.7 | ceiling |
| Ternary baseline | 4.9852 | 146.2 | +19.2% |
| **This model (continued distillation)** | **4.9125** | **136.0** | **+10.8%** |
| From-scratch recipe (null control) | 4.9878 | 146.6 | +19.5% |

## Model details

- **Architecture:** bidirectional masked-diffusion transformer; `d_model` 1024, 24 layers, 16
  heads, `d_ff` 2816, tied embeddings, 32001 vocabulary (mask token id 32000).
- **Format:** `model_bf16.pt`, verified within 0.0003 masked-CE of the fp32 master.
- **Tokenizer:** `spm_it.model` (SentencePiece unigram, included).

## Loading

The checkpoint uses the architecture in the [`echo-1.58`](https://github.com/lupodevelop/echo-1.58)
repository:

```python
import torch
from huggingface_hub import hf_hub_download
from model.arch import TernaryDiffusionLM, TernaryDiffusionConfig
from model.mitigations import MitigatedDiffusionLM
from model.quant import MitigationConfig, QATConfig

path = hf_hub_download("lupodevelop/echo-mdlm-341m-ternary-continued", "model_bf16.pt")
ck = torch.load(path, map_location="cpu")
cfg = TernaryDiffusionConfig(**ck["cfg"])
model = MitigatedDiffusionLM(cfg, MitigationConfig(), QATConfig(chan_scale=True))
model.load_state_dict({k: v.float() for k, v in ck["model"].items()})
model.eval()
```

See the repository for infilling (`eval/cloze.py`) and generation (`eval/sample.py`).

## Related models

Part of the [Echo-1.58 collection](https://huggingface.co/collections/lupodevelop/echo-158-ternary-masked-diffusion-lms-6a5dd179a5f7ad08cc277404):
the FP16 twin, the bare ternary baseline, the from-scratch null control, and the 27M research
ladder.

## Citation

```bibtex
@article{scaratti2026ternary,
  author  = {Daniele Scaratti},
  title   = {Recovering the Ternary Penalty in Masked Diffusion Language Models: What Adds Capacity, What Only Reorganizes, and What Repetition Costs},
  year    = {2026},
  doi     = {10.5281/zenodo.21455055}
}
```