README.md

metadata

license: apache-2.0
language:
  - it
library_name: echo-1.58
tags:
  - masked-diffusion
  - ternary
  - 1.58-bit
  - quantization-aware-training
  - bitnet
  - italian
pipeline_tag: fill-mask

These are research artifacts accompanying the paper Native Ternary Quantization-Aware Training for Masked Diffusion Language Models. They are 341M-parameter masked-diffusion language models trained on Italian FineWeb-2 with a 32k SentencePiece tokenizer. This is not a production model. At roughly 12 tokens per parameter neither the ternary nor the full-precision model composes fluent text; free generation degenerates identically at both precisions. Use these checkpoints for reproduction, infilling analysis, and as paired baselines, not as downstream generators.

• Architecture: bidirectional masked-diffusion transformer, d_model 1024, 24 layers, 16 heads, d_ff 2816, tied embeddings, 32001 vocabulary (mask token id 32000).
• Format: bfloat16, verified to reproduce the full-precision evaluation within 0.0003 masked-CE of the fp32 master.
• Code and reproduction: https://github.com/lupodevelop/echo-1.58
• Tokenizer: spm_it.model (included).

Evaluation (common protocol, sequence length 1024, identical seeded masks)

341M model	masked-CE	perplexity	vs FP16 twin
FP16 twin	4.8100	122.7	ceiling
Ternary baseline	4.9852	146.2	+19.2%
+ continued distillation	4.9125	136.0	+10.8%
+ from-scratch recipe	4.9878	146.6	+19.5%

Echo-1.58 341M, Ternary Baseline

The bare ternary masked-diffusion model, trained natively at 1.58 bits with BitNet-style QAT and no recovery recipe. It is the starting point for the recovery experiments and the +19.2% perplexity penalty against the FP16 twin. Post-training quantization of a full-precision model to this precision collapses (26x perplexity for round-to-nearest); this model is what native training buys instead.