---
license: apache-2.0
language:
- ko
- en
base_model:
- google/gemma-4-12B-it
- BCCard/gemma-4-12B-it-FP8-Dynamic
datasets:
- sh2orc/bccard-maywell-jojo0217-markai-lcw99-kendamarron-microsoft
- Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered
tags:
- eagle3
- speculative-decoding
- speculators
- vllm
library_name: speculators
---

# gemma-4-12B-it EAGLE-3 Draft

An EAGLE-3 draft (speculator) for accelerating `gemma-4-12B-it` via speculative decoding.
Trained from scratch on **on-policy responses regenerated by the verifier itself**, over a
general-purpose QA prompt set (~450k prompts: English Magpie + Korean/English sh2orc).

## Overview

- **Method**: EAGLE-3 (`vllm-project/speculators`)

- **Verifier (target)**: `BCCard/gemma-4-12B-it-FP8-Dynamic` (FP8; used for serving and hidden-state extraction)

- **Reused weights**: BF16 embed/lm_head from `google/gemma-4-12B-it` (standard EAGLE-3)

- **Warm start**: none — trained from scratch (no pretrained speculator initialization)

- **Sequence length**: 4096

## Training data

~450k prompts total, from two sources:

- ~150k from [sh2orc/bccard-maywell-jojo0217-markai-lcw99-kendamarron-microsoft](https://huggingface.co/datasets/sh2orc/bccard-maywell-jojo0217-markai-lcw99-kendamarron-microsoft) — Korean / English QA

- ~300k from [Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered) — English

Only the prompt (`instruction`) is used; original answers are discarded and regenerated
on-policy by the verifier.

## Serving (vLLM)

```bash
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve BCCard/gemma-4-12B-it-FP8-Dynamic -tp 1 \
  --max-model-len 8192 \
  --speculative-config '{
    "model": "BCCard/MoAI-gemma-4-12B-it-speculator.eagle3",
    "num_speculative_tokens": 4,
    "method": "eagle3",
    "draft_tensor_parallel_size": 1
  }'
```

Tune `num_speculative_tokens` in the 4–8 range based on measured acceptance / TPS.
The draft uses the verifier's tokenizer.

## Performance

Per-position acceptance at training time (validation split).

| position | full_acc | cond_acc |
| --- | --- | --- |
| 0 | 0.665 | 0.665 |
| 1 | 0.412 | 0.620 |
| 2 | 0.264 | 0.640 |

Mean accepted length ≈ 2.3 tokens/step, roughly ~2.3x speedup (measure on your own traffic).

Train and validation metrics match closely, so there is no overfitting.

## Limitations

- Trained on general-purpose QA (mostly English Magpie + Korean/English sh2orc). Domain- or
  language-specific traffic (e.g. Korean finance) may benefit from another training cycle on
  matched data, raising acceptance.

- Acceptance is measured against the verifier `BCCard/gemma-4-12B-it-FP8-Dynamic`.
  Pairing the draft with a different target will change results.

## License

**Apache 2.0**. The base Gemma 4 (Apache 2.0 since 2026-04, the first Gemma family to adopt
it) and the verifier `BCCard/gemma-4-12B-it-FP8-Dynamic` (Apache 2.0) are both Apache 2.0,
and this draft was trained from scratch (no third-party speculator weights), so it is
released under Apache 2.0 as well.

Apache 2.0 requires only attribution of the original copyright and disclosure of
modifications, with no restrictions on commercial use, modification, or redistribution.
(This is informational, not legal advice.)