Add Confucius3-Math EAGLE3 draft head (epoch_2/step_12000, ~38% trained)

README.md

@@ -0,0 +1,59 @@
+---
+license: apache-2.0
+base_model: netease-youdao/Confucius3-Math
+tags:
+- eagle3
+- speculative-decoding
+- draft-model
+- vllm
+---
+
+# Confucius3-Math EAGLE3 Draft Head
+
+An [EAGLE3](https://arxiv.org/abs/2503.01840) speculative-decoding draft head for
+[**netease-youdao/Confucius3-Math**](https://huggingface.co/netease-youdao/Confucius3-Math)
+(Qwen2.5-14B, R1-style inline-think). Trained with [SpecForge](https://github.com/sgl-project/SpecForge)
+on the same rollout data as a matched DFlash / D-PACE drafter, for a method comparison.
+
+> ⚠️ **Intermediate checkpoint.** This is `epoch_2 / step_12000` of a planned 6-epoch
+> (31,734-step) run — roughly **38%** through training. Acceptance length will still
+> improve with further training. A final checkpoint will replace this when the run completes.
+
+## Architecture
+
+| | |
+|---|---|
+| architecture | `LlamaForCausalLMEagle3` (1 layer) |
+| hidden size | 5120 |
+| heads (q/kv) | 40 / 8 |
+| intermediate | 13824 |
+| draft vocab | 32000 (with `d2t`/`t2d` mapping to the 152064 target vocab) |
+| aux hidden layers | `[1, 23, 44]` (SpecForge default) |
+| dtype | bfloat16 |
+
+## Training
+
+- SpecForge EAGLE3 defaults: auto-generated draft config, lr 1e-4, warmup 0.015, grad-norm 0.5, TTT length 7.
+- Copied from the matched DFlash/D-PACE run: same preformatted Confucius3-Math rollout data (148,112 rows), `confucius-r1` chat template, max-length 4096, batch-size 4, 6 epochs.
+
+## Usage (vLLM)
+
+```bash
+vllm serve netease-youdao/Confucius3-Math \
+  --speculative-config '{"method":"eagle3","model":"noctuashap/Confucius3-Math-Eagle3","num_speculative_tokens":7}' \
+  --attention-backend flash_attn --trust-remote-code
+```
+
+`num_speculative_tokens` is the draft length — lower it (e.g. 2–3) for high-concurrency /
+compute-bound serving, raise it (up to ~7) for low-batch / latency-oriented serving.
+
+## Measured acceptance (gsm8k / math500, thinking mode, vLLM)
+
+At an earlier checkpoint (step_8000) with `num_speculative_tokens=7`:
+
+| dataset | accept length | tok/s (single stream) |
+|---|---|---|
+| gsm8k | ~4.3 | ~365 |
+| math500 | ~4.5 | ~386 |
+
+(Confucius3-Math is R1-style always-thinking, so thinking and non-thinking modes are equivalent.)

config.json

@@ -0,0 +1,31 @@
+{
+  "architectures": [
+    "LlamaForCausalLMEagle3"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151646,
+  "draft_vocab_size": 32000,
+  "dtype": "bfloat16",
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 13824,
+  "max_position_embeddings": 131072,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 1,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 152064
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53c98895cdc119505590c84a07c07ac7e5e67a620df5aa684abc9e7089a71ffd
+size 1109319664