---
library_name: speculators
base_model:
- RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic
license: apache-2.0
tags:
- speculative-decoding
- dflash
- speculators
---
 
# NVIDIA-Nemotron-3-Ultra-550B-A55B-speculator.dflash
 
This is a DFlash speculator model for [RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic](https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic).
 
## Training Details
 
This model was trained using the [Speculators](https://github.com/vllm-project/speculators) library on a subset of [Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered) and the `train_sft` split of [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k). Responses were regenerated by NVIDIA-Nemotron-3-Ultra-550B-A55B.

 
 Commands 

 Using the [Speculators](https://github.com/vllm-project/speculators) library and the helper scripts provided in the repo.

 ### Prepare data
  ```bash
  # In virtual environment with speculators installed
  python scripts/prepare_data.py \
    --model RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic \
    --data ./regenerated_data.jsonl \
    --output ./output \
    --seq-length 8192
  ```

 ### Launch vLLM
  ```bash
  # In (separate) virtual environment with vllm installed
  CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 vllm_venv/bin/python scripts/launch_vllm.py \
    RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic \
    --target-layer-ids 7 47 88 \
    -- --port 8000 \
    --gpu-memory-utilization 0.9 \
    --disable-uvicorn-access-log \
    --tensor-parallel-size 8
  ```

 ### Launch training
 Must be run once vLLM has finished launching and is running in the background.
  ```bash
  # In virtual environment with speculators installed
  CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun \
    --standalone \
    --nproc_per_node 8 \
    scripts/train.py \
    --verifier-name-or-path RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic \
    --speculator-type dflash \
    --num-layers 5 \
    --data-path ./output \
    --vllm-endpoint http://localhost:8000/v1 \
    --save-path ./output/checkpoints \
    --epochs 5 \
    --lr 0.0006 \
    --total-seq-len 8192 \
    --on-missing generate \
    --on-generate delete \
    --seed 42 \
    --log-freq 10 \
    --draft-vocab-size 32000 \
    --draft-arch llama \
    --target-layer-ids 7 47 88 \
    --draft-hidden-act silu \
    --scheduler-type cosine \
    --max-anchors 3072 \
    --prefetch-factor 2 \
    --num-workers 16 \
    --sliding-window-indices 0 1 2 3 4 \
    --optimizer muon
  ```
 
 

 
## Model Specifications

| | |
|---|---|
| **Base Model** | RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic |
| **Chat Template** | RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic (use `/chat/completions` endpoint) |
| **Format** | Safetensors |
| **License** | Apache 2.0 |
| **Validation Hardware** | Nvidia B200 |
 
## Deployment
 
  ```bash
  # Deploy with speculative decoding
  vllm serve RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-dynamic \
      --tensor-parallel-size 8 \
      --max-model-len 16384 \
      --speculative-config '{
          "model": "RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-speculator.dflash",
          "num_speculative_tokens": 7,
          "method": "dflash"
      }'
  ```
 
 

 

## Acceptance Rate

Per-position token acceptance rates across datasets:

| Dataset | Pos 0 | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Pos 7 | Avg. Length |
|---------|-------|-------|-------|-------|-------|-------|-------|-------|------------|
| HumanEval | 76.3% | 53.9% | 37.3% | 25.8% | 18.8% | 14.0% | 10.5% | 7.5% | 3.44 |
| math_reasoning | 88.4% | 74.8% | 61.3% | 50.7% | 41.0% | 32.9% | 25.4% | 19.3% | 4.94 |
| qa | 64.9% | 37.9% | 21.5% | 12.8% | 7.7% | 4.5% | 2.7% | 1.5% | 2.53 |
| question | 68.4% | 41.3% | 24.1% | 15.3% | 10.4% | 7.1% | 4.8% | 3.0% | 2.74 |
| rag | 75.3% | 52.0% | 34.6% | 23.7% | 16.2% | 11.1% | 7.4% | 4.3% | 3.25 |
| summarization | 71.6% | 44.5% | 25.6% | 14.9% | 8.3% | 4.7% | 2.3% | 1.1% | 2.73 |
| tool_call | 73.3% | 48.4% | 29.2% | 18.5% | 12.3% | 8.0% | 5.2% | 3.5% | 2.98 |
| translation | 64.8% | 41.0% | 24.9% | 14.6% | 8.3% | 4.7% | 2.7% | 1.4% | 2.62 |
| writing | 68.2% | 41.0% | 24.1% | 15.1% | 10.2% | 7.0% | 4.7% | 3.0% | 2.73 |

 
## References

**Paper**: [DFlash: Block Diffusion for Flash Speculative Decoding](https://arxiv.org/abs/2602.06036)