Update to 50k-page chunked-rolling head: 1.79x single-stream (from 1.36x)

README.md

@@ -16,20 +16,22 @@ An **EAGLE‑3.1 speculative‑decoding draft head** for [`datalab-to/chandra-oc
 
 ## Results
 
-Measured in vLLM on olmOCR‑bench pages (single‑stream, greedy, CUDA graphs, RTX 3090 Ti @ 450 W):
+Single‑stream, greedy, vLLM + CUDA graphs on an RTX 3090 Ti, over olmOCR‑bench pages:
 
 | | tok/s | speedup |
 |---|---|---|
-| baseline (no spec) | 84.1 | 1.0× |
-| **+ this head (num_spec=5)** | **114.3** | **1.36×** |
+| baseline (no spec) | 58.0 | 1.0× |
+| **+ this head (num_spec=5)** | **104.0** | **1.79×** |
 
-Mean accepted length ≈ 2.0–2.5; per‑position acceptance ≈ [0.58, 0.37, 0.25, 0.20, 0.12]. Quality: median char‑similarity vs baseline 0.9991 (differences confined to bbox‑pixel jitter / equivalent figure captions — i.e. bf16 greedy non‑determinism, not degradation).
+Per‑position acceptance ≈ [0.92, …] at depth (train acc0 ≈ 0.92, acc6 ≈ 0.9). Quality: median char‑similarity vs baseline 0.999 — differences are confined to bbox‑pixel jitter / equivalent figure captions (bf16 greedy non‑determinism, not degradation; 36/40 pages identical once bbox coordinates are normalized).
+
+**Note on batching:** spec‑decode wins at **low concurrency** (interactive). At full batch the GPU saturates and the draft becomes overhead (~0.75× baseline) — disable spec or drop `num_speculative_tokens` to 1–2 for bulk throughput.
 
 ## Architecture
 
 EAGLE‑3.1 (single decoder layer) over Chandra‑2's Qwen3.5 text backbone:
 - `hidden_size` 2560, `head_dim` 256, GQA 16/4 — matched to the target.
-- **EAGLE 3.1**: `fc_norm` (per‑aux‑layer RMSNorm) + `norm_output` (post‑norm recurrence) — fixes attention drift, holds acceptance at depth.
+- **EAGLE 3.1**: `fc_norm` (per‑aux‑layer RMSNorm) + `norm_output` (post‑norm recurrence) — fixes attention drift, holds acceptance at depth (narrow acc0→acc6 gap).
 - **Vocab pruning**: `draft_vocab_size` 32768 (from 248320; 99.99% token coverage) → ~7.6× smaller lm_head, cheaper draft forward. `d2t` buffer maps pruned ids back to the full vocab.
 - Aux hidden states captured from the target's **full‑attention** layers `[3, 15, 27]` (Chandra‑2 is a hybrid 3:1 linear/full‑attention model).
 
@@ -48,11 +50,11 @@ llm = LLM(
 )
 ```
 
-Use the Chandra `ocr_layout` prompt in the user turn (image + instruction). Spec‑decode helps most at **low concurrency**; at high batch the GPU saturates and a smaller `num_speculative_tokens` (1–2) is usually better.
+Use the Chandra `ocr_layout` prompt in the user turn (image + instruction).
 
 ## Training
 
-- **On‑policy data**: Chandra‑2's own HTML‑with‑layout OCR over ~11k document pages.
-- **Features**: hidden states extracted offline from the frozen 5B target (decoupled from draft training so it fits 24 GB and trains with true DDP gradient sharing).
+- **On‑policy data**: Chandra‑2's own HTML‑with‑layout OCR over **~36.6k document pages**.
+- **Features**: hidden states extracted offline from the frozen 5B target (decoupled from draft training so it fits a 24 GB GPU and trains with true DDP gradient sharing).
 - **Objective**: forward‑KL distillation with TTT (length 7), trained in vLLM's EAGLE input‑shift convention so it serves correctly.
-- 4× RTX 3090 Ti, DDP, ~8k steps.
+- **Schedule**: 2 epochs via chunked‑rolling DDP (extract a chunk → train → continue to the next), since the full rep set exceeds disk. 4× RTX 3090 Ti.

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6beeccb1b3359cc516f491a1e556f528201ba5bad92ce220e3cb5193ec425883
+oid sha256:5a41428408acfda95015a633a41dedd72b2a85287903238538333cc8fa2cda09
 size 1704235808