lupodevelop/echo-mdlm-341m-ternary-continued

Echo-1.58 · 341M · Ternary, Continued Distillation

The recommended ternary checkpoint from Native Ternary Quantization-Aware Training for Masked Diffusion Language Models. A 341M-parameter masked-diffusion language model trained natively at 1.58 bits on Italian FineWeb-2, then recovered by distilling from its full-precision twin.

Research artifact, not a production model. At roughly 12 tokens per parameter neither the ternary nor the full-precision model composes fluent text; free generation degenerates identically at both precisions. Use it for reproduction, infilling analysis, and as a paired baseline.

What this checkpoint is

Starting from the bare ternary baseline, this model was distilled from the FP16 twin for a further 1B tokens with a learnable per-channel scale (0.07% extra memory). It recovers 41% of the ternary-to-FP16 gap in cross-entropy (48-49% on held-out test and out-of-domain Italian Wikipedia) and retains about 92% of FP16 infilling accuracy at every mask ratio, in a model 5.4x smaller than its twin. The paper's central practical finding is that this post-hoc route scales while applying the same recipe from the first step does not.

Evaluation (common protocol, sequence length 1024, identical seeded masks)

341M model	masked-CE	perplexity	vs FP16 twin
FP16 twin	4.8100	122.7	ceiling
Ternary baseline	4.9852	146.2	+19.2%
This model (continued distillation)	4.9125	136.0	+10.8%
From-scratch recipe (null control)	4.9878	146.6	+19.5%

Model details

• Architecture: bidirectional masked-diffusion transformer; d_model 1024, 24 layers, 16 heads, d_ff 2816, tied embeddings, 32001 vocabulary (mask token id 32000).
• Format: model_bf16.pt, verified within 0.0003 masked-CE of the fp32 master.
• Tokenizer: spm_it.model (SentencePiece unigram, included).

Loading

The checkpoint uses the architecture in the echo-1.58 repository:

import torch
from huggingface_hub import hf_hub_download
from model.arch import TernaryDiffusionLM, TernaryDiffusionConfig
from model.mitigations import MitigatedDiffusionLM
from model.quant import MitigationConfig, QATConfig

path = hf_hub_download("lupodevelop/echo-mdlm-341m-ternary-continued", "model_bf16.pt")
ck = torch.load(path, map_location="cpu")
cfg = TernaryDiffusionConfig(**ck["cfg"])
model = MitigatedDiffusionLM(cfg, MitigationConfig(), QATConfig(chan_scale=True))
model.load_state_dict({k: v.float() for k, v in ck["model"].items()})
model.eval()

See the repository for infilling (eval/cloze.py) and generation (eval/sample.py).

Related models

Part of the Echo-1.58 collection: the FP16 twin, the bare ternary baseline, the from-scratch null control, and the 27M research ladder.

Citation

@article{scaratti2026ternary,
  author  = {Daniele Scaratti},
  title   = {Recovering the Ternary Penalty in Masked Diffusion Language Models: What Adds Capacity, What Only Reorganizes, and What Repetition Costs},
  year    = {2026},
  doi     = {10.5281/zenodo.21455055}
}