Initial release

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,46 @@
+---
+license: apache-2.0
+language:
+- it
+library_name: echo-1.58
+tags:
+- masked-diffusion
+- ternary
+- 1.58-bit
+- quantization-aware-training
+- bitnet
+- italian
+pipeline_tag: fill-mask
+---
+
+These are research artifacts accompanying the paper *Native Ternary
+Quantization-Aware Training for Masked Diffusion Language Models*. They are 341M-parameter
+masked-diffusion language models trained on Italian FineWeb-2 with a 32k SentencePiece
+tokenizer. **This is not a production model.** At roughly 12 tokens per parameter neither the
+ternary nor the full-precision model composes fluent text; free generation degenerates
+identically at both precisions. Use these checkpoints for reproduction, infilling analysis,
+and as paired baselines, not as downstream generators.
+
+- **Architecture:** bidirectional masked-diffusion transformer, `d_model` 1024, 24 layers, 16
+  heads, `d_ff` 2816, tied embeddings, 32001 vocabulary (mask token id 32000).
+- **Format:** bfloat16, verified to reproduce the full-precision evaluation within 0.0003
+  masked-CE of the fp32 master.
+- **Code and reproduction:** https://github.com/lupodevelop/echo-1.58
+- **Tokenizer:** `spm_it.model` (included).
+
+## Evaluation (common protocol, sequence length 1024, identical seeded masks)
+
+| 341M model | masked-CE | perplexity | vs FP16 twin |
+|---|---|---|---|
+| FP16 twin | 4.8100 | 122.7 | ceiling |
+| Ternary baseline | 4.9852 | 146.2 | +19.2% |
+| + continued distillation | 4.9125 | 136.0 | +10.8% |
+| + from-scratch recipe | 4.9878 | 146.6 | +19.5% |
+
+# Echo-1.58 341M, Ternary Baseline
+
+The bare ternary masked-diffusion model, trained natively at 1.58 bits with BitNet-style QAT
+and no recovery recipe. It is the starting point for the recovery experiments and the +19.2%
+perplexity penalty against the FP16 twin. Post-training quantization of a full-precision model
+to this precision collapses (26x perplexity for round-to-nearest); this model is what native
+training buys instead.

config.json

@@ -0,0 +1,14 @@
+{
+  "model_type": "echo-mdlm",
+  "objective": "masked-diffusion",
+  "d_model": 1024,
+  "n_layers": 24,
+  "n_heads": 16,
+  "d_ff": 2816,
+  "vocab_size": 32001,
+  "mask_token_id": 32000,
+  "max_seq_len": 2048,
+  "tie_embeddings": true,
+  "quantization": "ternary-1.58bit-qat",
+  "precision_stored": "bfloat16"
+}

model_bf16.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:986e574b6f5fba71a142ec8496c52d419954899a71367b7c0744af4a131627cf
+size 748200412

spm_it.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:139bc5399e44c8b604196b1d45b96244904254e627b101d4600920dd031df091
+size 795469