NVFP4 Quantized Qwen3.6-35B-REAP-Pruned-ratio-0.5-NVFP4

This is an NVFP4 quantized version of RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.5 with llm-compressor. The model has both weights and activations quantized to NVFP4 format in compressed-tensors.

It was created with llm-compressor version 0.10.1.dev131+g22ebb057. The compression run was performed on a machine with 64 GB system RAM and an NVIDIA GeForce RTX 5070 Ti 16 GiB GPU.

Serving

Tested on an NVIDIA GeForce RTX 5070 Ti 16 GiB with vLLM's NVFP4 linear path and FlashInfer CUTLASS MoE backend. The vLLM environment needs FlashInfer installed, including the JIT cache package:

uv pip install flashinfer-python flashinfer-cubin flashinfer-jit-cache

flashinfer-jit-cache is important because it avoids running the FlashInfer JIT compilation phase at serve startup, which can run out of memory even on a machine with 64 GiB system RAM.

export CUDA_VISIBLE_DEVICES=0
export OMP_NUM_THREADS=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

vllm serve sroecker/Qwen3.6-35B-REAP-Pruned-ratio-0.5-NVFP4 \
  --host 0.0.0.0 \
  --port 8000 \
  --language-model-only \
  --reasoning-parser qwen3 \
  --moe_backend flashinfer_cutlass \
  --max-model-len auto \
  --max-num-seqs 1 \
  --max-num-batched-tokens 1024 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.94 \
  --mm-processor-cache-gb 0

This configuration reported 35,623 FP8 KV-cache tokens.

Benchmark

Benchmarked through the OpenAI-compatible vLLM endpoint with:

uvx llama-benchy --base-url "http://0.0.0.0:8000/v1" \
  --depth 0 2048 4096 8096 \
  --tg 128 \
  --latency-mode generation

model	test	t/s	peak t/s	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
sroecker/Qwen3.6-35B-REAP-Pruned-ratio-0.5-NVFP4	pp2048	28408.75 +/- 37.69		132.43 +/- 0.10	72.13 +/- 0.10	132.43 +/- 0.10
sroecker/Qwen3.6-35B-REAP-Pruned-ratio-0.5-NVFP4	tg128	133.27 +/- 0.02	134.33 +/- 0.02
sroecker/Qwen3.6-35B-REAP-Pruned-ratio-0.5-NVFP4	pp2048 @ d2048	20669.58 +/- 430.28		258.59 +/- 4.14	198.28 +/- 4.14	258.59 +/- 4.14
sroecker/Qwen3.6-35B-REAP-Pruned-ratio-0.5-NVFP4	tg128 @ d2048	132.89 +/- 0.03	133.94 +/- 0.03
sroecker/Qwen3.6-35B-REAP-Pruned-ratio-0.5-NVFP4	pp2048 @ d4096	18389.86 +/- 185.67		394.47 +/- 3.38	334.17 +/- 3.38	394.47 +/- 3.38
sroecker/Qwen3.6-35B-REAP-Pruned-ratio-0.5-NVFP4	tg128 @ d4096	131.84 +/- 0.02	132.88 +/- 0.03
sroecker/Qwen3.6-35B-REAP-Pruned-ratio-0.5-NVFP4	pp2048 @ d8096	16152.89 +/- 1.75		688.37 +/- 0.02	628.06 +/- 0.02	688.37 +/- 0.02
sroecker/Qwen3.6-35B-REAP-Pruned-ratio-0.5-NVFP4	tg128 @ d8096	131.05 +/- 0.01	132.08 +/- 0.01

Creation Script

Run with:

uv run examples/quantization_w4a4_fp4/rangerx_qwen3_6_reap_pruned_nvfp4_optimized_bucketed_batch8.py

import torch
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from datasets import load_dataset
from torch.nn.utils.rnn import pad_sequence
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from transformers.models.qwen3_5_moe.modeling_qwen3_5_moe import (
    Qwen3_5MoeDecoderLayer,
)

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

# NOTE: This example requires transformers >= v5

MODEL_ID = "RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.5"
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-NVFP4"


def qwen3_5_moe_decoder_forward_for_calibration(
    self,
    hidden_states,
    position_embeddings,
    attention_mask=None,
    position_ids=None,
    past_key_values=None,
    **kwargs,
):
    residual = hidden_states
    hidden_states = self.input_layernorm(hidden_states)

    if self.layer_type == "linear_attention":
        hidden_states = self.linear_attn(
            hidden_states=hidden_states,
            cache_params=past_key_values,
            attention_mask=attention_mask,
        )
    elif self.layer_type == "full_attention":
        hidden_states, _ = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_values=past_key_values,
            position_embeddings=position_embeddings,
            **kwargs,
        )

    hidden_states = residual + hidden_states

    residual = hidden_states
    hidden_states = self.post_attention_layernorm(hidden_states)
    hidden_states = self.mlp(hidden_states)
    hidden_states = residual + hidden_states

    return hidden_states


# The upstream decoder contains a defensive tuple-unpack branch after the MoE MLP.
# During llm-compressor sequential tracing, that branch is autowrapped into a helper
# that references "_" before assignment. The Qwen3.5 MoE MLP returns a tensor for this
# model, so removing the branch keeps the calibration forward equivalent and traceable.
Qwen3_5MoeDecoderLayer.forward = qwen3_5_moe_decoder_forward_for_calibration

# Load model.
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)
tokenizer = getattr(processor, "tokenizer", processor)
pad_token_id = tokenizer.pad_token_id
if pad_token_id is None or pad_token_id < 0:
    pad_token_id = tokenizer.eos_token_id

# No need to include mtp layers as they are not loaded
# through Qwen3_5MoeForConditionalGeneration
recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=[
        "re:.*lm_head",
        "re:visual.*",
        "re:model.visual.*",
        "re:.*mlp.gate$",
        "re:.*embed_tokens$",
        "re:.*shared_expert_gate$",
        "re:.*linear_attn.*",
    ],
)

NUM_CALIBRATION_SAMPLES = 256
MAX_SEQUENCE_LENGTH = 4096
BATCH_SIZE = 8
SEQUENTIAL_TARGETS_PER_SUBGRAPH = 2
PADDING_STATS_INTERVAL = 25
TRACING_IGNORE = [
    "_update_causal_mask",
    "create_causal_mask",
    "_update_mamba_mask",
    "make_causal_mask",
    "get_causal_mask",
    "mask_interface",
    "mask_function",
    "_prepare_4d_causal_attention_mask",
    "_prepare_fsmt_decoder_inputs",
    "_prepare_4d_causal_attention_mask_with_cache_position",
    "_update_linear_attn_mask",
    "project_per_layer_inputs",
    "apply_mask_to_padding_states",
]

ds = load_dataset(
    "HuggingFaceH4/ultrachat_200k",
    split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]",
)
ds = ds.select_columns(["messages"])
ds = ds.shuffle(seed=42)


def _to_sequence_tensor(value):
    tensor = torch.as_tensor(value)
    if tensor.ndim == 2 and tensor.shape[0] == 1:
        tensor = tensor.squeeze(0)
    return tensor


def _percentile(sorted_values, percentile):
    if not sorted_values:
        return 0

    index = round((len(sorted_values) - 1) * percentile)
    return sorted_values[index]


def _padding_waste(lengths, batch_size):
    real_tokens = sum(lengths)
    padded_tokens = 0

    for start in range(0, len(lengths), batch_size):
        batch_lengths = lengths[start : start + batch_size]
        padded_tokens += max(batch_lengths) * len(batch_lengths)

    waste = 1 - (real_tokens / padded_tokens) if padded_tokens else 0
    return real_tokens, padded_tokens, waste


def _print_length_report(label, lengths):
    sorted_lengths = sorted(lengths)
    real_tokens, padded_tokens, waste = _padding_waste(lengths, BATCH_SIZE)
    print(
        f"[lengths] {label}: "
        f"samples={len(lengths)} "
        f"min={sorted_lengths[0]} "
        f"p50={_percentile(sorted_lengths, 0.50)} "
        f"p90={_percentile(sorted_lengths, 0.90)} "
        f"max={sorted_lengths[-1]} "
        f"padding_waste={waste:.1%} "
        f"tokens={real_tokens}/{padded_tokens}"
    )


def preprocess_function(example):
    messages = [
        {"role": m["role"], "content": [{"type": "text", "text": m["content"]}]}
        for m in example["messages"]
    ]
    encoded = processor.apply_chat_template(
        messages,
        tokenize=True,
        return_dict=True,
        add_generation_prompt=False,
        processor_kwargs={
            "return_tensors": "pt",
            "padding": False,
            "truncation": True,
            "max_length": MAX_SEQUENCE_LENGTH,
            "add_special_tokens": False,
        },
    )
    return {key: _to_sequence_tensor(value).tolist() for key, value in encoded.items()}


def add_length(example):
    return {"length": len(example["input_ids"])}


ds = ds.map(preprocess_function, batched=False, remove_columns=ds.column_names)
ds = ds.map(add_length, batched=False)

shuffled_lengths = ds["length"]
_print_length_report("shuffled", shuffled_lengths)

ds = ds.sort("length")
bucketed_lengths = ds["length"]
_print_length_report("length-bucketed", bucketed_lengths)
ds = ds.remove_columns(["length"])

padding_stats = {
    "batches": 0,
    "real_tokens": 0,
    "padded_tokens": 0,
}


def data_collator(batch):
    features = [
        {key: _to_sequence_tensor(value) for key, value in example.items()}
        for example in batch
    ]

    input_lengths = [feature["input_ids"].numel() for feature in features]
    batch_real_tokens = sum(input_lengths)
    batch_padded_tokens = max(input_lengths) * len(input_lengths)

    padding_stats["batches"] += 1
    padding_stats["real_tokens"] += batch_real_tokens
    padding_stats["padded_tokens"] += batch_padded_tokens

    batch_index = padding_stats["batches"]
    if batch_index <= 5 or batch_index % PADDING_STATS_INTERVAL == 0:
        batch_waste = 1 - (batch_real_tokens / batch_padded_tokens)
        total_waste = 1 - (
            padding_stats["real_tokens"] / padding_stats["padded_tokens"]
        )
        print(
            f"[padding] batch={batch_index} "
            f"size={len(features)} "
            f"max_len={max(input_lengths)} "
            f"waste={batch_waste:.1%} "
            f"running_waste={total_waste:.1%} "
            f"tokens={batch_real_tokens}/{batch_padded_tokens}"
        )

    collated = {}
    for key in features[0]:
        padding_value = pad_token_id if key == "input_ids" else 0
        collated[key] = pad_sequence(
            [feature[key] for feature in features],
            batch_first=True,
            padding_value=padding_value,
        )

    return collated


# Apply quantization.
oneshot(
    model=model,
    recipe=recipe,
    dataset=ds,
    batch_size=BATCH_SIZE,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    shuffle_calibration_samples=False,
    moe_calibrate_all_experts=True,
    data_collator=data_collator,
    # Optimized layerwise mode: batch calibration samples, keep the next cached
    # activation batch prefetched, and group two decoder layers into each subgraph.
    sequential_targets=["Qwen3_5MoeDecoderLayer"],
    sequential_targets_per_subgraph=SEQUENTIAL_TARGETS_PER_SUBGRAPH,
    sequential_prefetch=True,
    tracing_ignore=TRACING_IGNORE,
)

# Save to disk in compressed-tensors format.
model.save_pretrained(SAVE_DIR)
processor.save_pretrained(SAVE_DIR)

# MTP layers are excluded from the model through Qwen3_5MoeForConditionalGeneration
# Save them as-is from the original checkpoint into the quantized output.
try:
    save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir=SAVE_DIR)
except ValueError as exc:
    if "No tensors with prefix 'mtp'" not in str(exc):
        raise
    print(f"Skipping MTP tensor copy: {exc}")

Calibration Notes

Length bucketing reduced expected batch padding waste for the 256 calibration examples from 43.3% in shuffled order to 3.8% in bucketed order at batch size 8.

Batch sizes tested:

• 6: stable, lower VRAM, slower than batch 8
• 8: best observed balance
• 12: fit in VRAM but was slower and approached the 16 GiB limit