Instructions to use unsloth/Qwen3.6-27B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/Qwen3.6-27B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="unsloth/Qwen3.6-27B-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("unsloth/Qwen3.6-27B-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("unsloth/Qwen3.6-27B-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use unsloth/Qwen3.6-27B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/Qwen3.6-27B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3.6-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/Qwen3.6-27B-NVFP4

SGLang

How to use unsloth/Qwen3.6-27B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/Qwen3.6-27B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3.6-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/Qwen3.6-27B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3.6-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Unsloth Studio

How to use unsloth/Qwen3.6-27B-NVFP4 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3.6-27B-NVFP4 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3.6-27B-NVFP4 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/Qwen3.6-27B-NVFP4 to start chatting

Load model with FastModel

pip install unsloth
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
    model_name="unsloth/Qwen3.6-27B-NVFP4",
    max_seq_length=2048,
)

Docker Model Runner
How to use unsloth/Qwen3.6-27B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/unsloth/Qwen3.6-27B-NVFP4
```

DGX Spark Instructions!

#11

pinned

by danielhanchen - opened 11 days ago

Discussion

danielhanchen

Unsloth AI org 11 days ago

Hey guys, if you use a DGX Spark, ensure you use the correct settings. Use the flashinfer backend or you will get 2x slower inference!

Our blog has more details: https://unsloth.ai/docs/models/qwen3.6#dgx-spark-with-nvfp4-quants

danielhanchen pinned discussion 11 days ago

jeffutter

11 days ago

👏

Does anyone have benchmarks from the spark? If I’m reading correct all the benches in the documentation are on B200?

Curious what the spark looks like.

lvyou

11 days ago

sglang refuse to start:

(sglang-official) lvyou@gx10-4002:/sglang-official$ uv pip install -U --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang Resolved 195 packages in 13.83s
Prepared 1 package in 0.46ms
Uninstalled 1 package in 0.48ms
Installed 1 package in 2ms
~ nvidia-cusparselt-cu13==0.8.0
(sglang-official) lvyou@gx10-4002:/sglang-official$ python -m sglang.launch_server --model-path unsloth/Qwen3.6-27B-NVFP4 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/launch_server.py:55: UserWarning: 'python -m sglang.launch_server' is still supported, but 'sglang serve' is the recommended entrypoint.
Example: sglang serve --model-path [options]
warnings.warn(
[2026-07-11 20:35:52] Breakable CUDA graph is incompatible with multimodal model; disabling prefill CUDA graph.
[2026-07-11 20:35:52] Failed to get GPU memory capacity from nvidia-smi. Falling back to torch.cuda.mem_get_info(). Reported total GPU memory per device (MiB): [124546], using min: 124546 MiB.
[2026-07-11 20:35:52] Attention backend not specified. Use flashinfer backend by default.
[2026-07-11 20:35:52] Max running requests is reset to 48 for speculative decoding. You can override this by explicitly setting --max-running-requests.
[2026-07-11 20:35:52] server_args=ServerArgs(model_path='/home/lvyou/.cache/modelscope/hub/models/unsloth/Qwen3.6-27B-NVFP4', tokenizer_path='/home/lvyou/.cache/modelscope/hub/models/unsloth/Qwen3.6-27B-NVFP4', tokenizer_mode='auto', tokenizer_backend='huggingface', tokenizer_worker_num=1, detokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=False, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', model_config_parser='auto', json_model_override_args='{}', dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, enable_tf32_matmul=False, mem_fraction_static=0.8248909375, max_running_requests=48, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=8192, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, disable_priority_preemption=False, default_priority_value=None, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', prefill_only_disable_kv_cache=False, disable_radix_cache=False, enable_page_major_kv_layout=False, enable_unified_memory=False, disable_chunked_prefix_cache=False, disable_overlap_schedule=False, num_continuous_decode_steps=1, scheduler_recv_interval=1, enable_mixed_chunk=False, nccl_port=None, dist_timeout=None, dist_init_addr=None, nnodes=1, node_rank=0, tp_size=1, dcp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, dp_size=1, load_balance_method='round_robin', attn_cp_size=1, moe_dp_size=1, enable_prefill_cp=False, cp_strategy=None, enable_dsa_prefill_context_parallel=False, dsa_prefill_cp_mode='round-robin-split', enable_prefill_context_parallel=False, prefill_cp_mode='in-seq-split', enable_dp_attention=False, enable_dp_attention_local_control_broadcast=False, enable_dp_lm_head=False, enable_attn_tp_input_scattered=False, disable_attn_tp_gather=False, enable_p2p_check=False, device='cuda', base_gpu_id=0, gpu_id_step=1, random_seed=464693622, watchdog_timeout=300, soft_watchdog_timeout=None, sleep_on_idle=False, use_ray=False, custom_sigquit_handler=None, numa_node=None, gc_threshold=None, host='127.0.0.1', port=30000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, enable_http2=False, ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_keyfile_password=None, enable_ssl_refresh=False, api_key=None, admin_api_key=None, served_model_name='unsloth/Qwen3.6-27B-NVFP4', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, strip_thinking_cache=False, enable_strict_thinking=False, tool_call_parser=None, tool_server=None, sampling_defaults='model', asr_max_buffer_seconds=60, asr_max_concurrent_sessions=32, preferred_sampling_params=None, allow_auto_truncate=False, stream_interval=1, batch_notify_size=16, stream_response_default_include_usage=False, incremental_streaming_output=False, enable_streaming_session=False, enable_session_radix_cache=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, grpc_http_sidecar_port=None, enable_mfu_metrics=False, enable_metrics_for_all_schedulers=False, load_snapshot_publish_interval=15, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_forward_pass_metrics=False, forward_pass_metrics_worker_id='', forward_pass_metrics_ipc_name=None, enable_trace=False, trace_modules='request', otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, stat_loggers=None, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, attention_backend='flashinfer', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', radix_cache_backend=None, mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='auto', bf16_gemm_backend='auto', dsa_prefill_backend=None, dsa_decode_backend=None, dsa_paged_mqa_logits_backend='auto', dsa_topk_backend='sgl-kernel', disable_flashinfer_autotune=False, mamba_backend='triton', cuda_graph_config=CudaGraphConfig(decode=PhaseConfig(backend='full', max_bs=256, bs=[1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 40, 44, 48, 52, 56, 60, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], tc_compiler='eager', full_prefill_max_req=None), prefill=PhaseConfig(backend='disabled', max_bs=8192, bs=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192], tc_compiler='eager', full_prefill_max_req=None)), cuda_graph_backend_decode=None, cuda_graph_backend_prefill=None, cuda_graph_max_bs_decode=None, cuda_graph_max_bs_prefill=None, cuda_graph_bs_decode=None, cuda_graph_bs_prefill=None, cuda_graph_tc_compiler=None, disable_prefill_cuda_graph=False, disable_decode_cuda_graph=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, debug_cuda_graph=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, flashinfer_mla_disable_ragged=False, enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_fused_moe_sum_all_reduce=False, enable_deepseek_v4_fp4_indexer=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, pre_warm_nccl=False, enable_quant_communications=False, enable_flashinfer_allreduce_fusion=False, enforce_disable_flashinfer_allreduce_fusion=False, flashinfer_allreduce_fusion_backend=None, enable_aiter_allreduce_fusion=False, enable_torch_compile=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, torchao_config='', speculative_algorithm='EAGLE', speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=3, speculative_eagle_topk=1, speculative_num_draft_tokens=4, speculative_dflash_block_size=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_use_rejection_sampling=False, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_draft_window_size=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_skip_dp_mlp_sync=False, enable_multi_layer_eagle=False, speculative_adaptive=False, speculative_adaptive_config=None, decoupled_spec_bind_endpoint=None, decoupled_spec_connect_endpoints=None, decoupled_spec_rank=None, decoupled_spec_role='null', spec_trace_dir=None, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_max_trie_depth=18, speculative_ngram_capacity=10000000, speculative_ngram_external_corpus_path=None, speculative_ngram_external_sam_budget=0, speculative_ngram_external_corpus_max_tokens=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', deepep_mode='auto', deepep_dispatcher_output_dtype='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, enable_elastic_expert_backup=False, mooncake_ib_device=None, enable_deepep_waterfill=False, elastic_ep_rejoin=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, disable_shared_experts_fusion=False, enforce_shared_experts_fusion=False, max_mamba_cache_size=None, mamba_ssm_dtype=None, enable_mamba_cache_stochastic_rounding=False, mamba_cache_philox_rounds=0, mamba_full_memory_ratio=0.9, mamba_radix_cache_strategy='extra_buffer', uses_mamba_radix_cache=True, mamba_track_interval=256, enable_int8_mamba_checkpoint=False, int8_mamba_ckpt_size=None, linear_attn_backend='triton', linear_attn_decode_backend=None, linear_attn_prefill_backend=None, enable_linear_replayssm=False, linear_replayssm_cache_len=16, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='page_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='timeout', hicache_storage_backend_extra_config=None, enable_hisparse=False, hisparse_config=None, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, enable_mm_global_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, experts_shared_outer_loras=None, lora_use_virtual_experts=False, lora_strict_loading=False, lora_drain_wait_threshold=0.0, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', enable_lmcache=False, lmcache_config_file=None, kt_weight_path=None, kt_method='AMXINT4', kt_cpuinfer=None, kt_threadpool_count=2, kt_num_gpu_experts=None, kt_max_deferred_experts_per_token=None, dllm_algorithm=None, dllm_algorithm_config=None, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_ib_device=None, disaggregation_decode_enable_radix_cache=False, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_extra_slots=None, disaggregation_decode_polling_interval=1, optimistic_prefill_retries=0, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], encoder_bootstrap_port=8997, encoder_register_urls=[], enable_adaptive_dispatch_to_encoder=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, custom_weight_loader=[], weight_loader_disable_mmap=False, weight_loader_prefetch_checkpoints=False, weight_loader_prefetch_num_threads=4, weight_loader_drop_cache_after_load=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, engine_info_bootstrap_port=6789, modelexpress_config=None, download_dir=None, model_checksum=None, delete_ckpt_after_loading=False, decrypted_config_file=None, decrypted_draft_config_file=None, checkpoint_engine_wait_weights_before_ready=False, enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, prefill_delayer_queue_min_ratio=None, prefill_delayer_max_delay_ms=None, min_free_slots_delay=None, enable_deterministic_inference=False, rl_on_policy_target=None, kv_canary='none', kv_canary_real_data='none', kv_canary_sweep_interval=0, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, enable_custom_logit_processor=False, enable_return_hidden_states=False, enable_return_routed_experts=False, enable_return_indexer_topk=False, disable_outlines_disk_cache=False, enable_mis=False, forward_hooks=None, msprobe_dump_config=None)
[ERROR] loss is part of Qwen2_5_VLCausalLMOutputWithPast.init's signature, but not documented. Make sure to add it to the docstring of the function in /home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py.
[ERROR] logits is part of Qwen2_5_VLCausalLMOutputWithPast.init's signature, but not documented. Make sure to add it to the docstring of the function in /home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py.
[2026-07-11 20:35:53] torchcodec is not installed; audio inputs will fail at request time
[2026-07-11 20:35:53] Ignore import error when loading sglang.srt.multimodal.processors.mimo_v2: No module named 'torchcodec'
[2026-07-11 20:35:53] Ignore import error when loading sglang.srt.multimodal.processors.mimo_v2_asr: Can not import FA3 in sgl_kernel. Please check your installation.
[transformers] The use_fast parameter is deprecated and will be removed in a future version. Use backend="torchvision" instead of use_fast=True, or backend="pil" instead of use_fast=False.
[2026-07-11 20:35:54] Ignore import error when loading sglang.srt.models.mimo_audio: Can not import FA3 in sgl_kernel. Please check your installation.
[2026-07-11 20:35:54] Ignore import error when loading sglang.srt.models.mimo_v2: Can not import FA3 in sgl_kernel. Please check your installation.
[2026-07-11 20:35:54] Ignore import error when loading sglang.srt.models.mimo_v2_asr: Can not import FA3 in sgl_kernel. Please check your installation.
[2026-07-11 20:35:54] Ignore import error when loading sglang.srt.models.mimo_v2_nextn: Can not import FA3 in sgl_kernel. Please check your installation.
[2026-07-11 20:35:54] Using default HuggingFace chat template with detected content format: openai
[2026-07-11 20:35:54] Auto-detected template features: reasoning_config=ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None), reasoning_parser=qwen3, tool_call_parser=qwen3_coder
[transformers] The use_fast parameter is deprecated and will be removed in a future version. Use backend="torchvision" instead of use_fast=True, or backend="pil" instead of use_fast=False.
[ERROR] loss is part of Qwen2_5_VLCausalLMOutputWithPast.init's signature, but not documented. Make sure to add it to the docstring of the function in /home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py.
[ERROR] logits is part of Qwen2_5_VLCausalLMOutputWithPast.init's signature, but not documented. Make sure to add it to the docstring of the function in /home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py.
[2026-07-11 20:35:57] torchcodec is not installed; audio inputs will fail at request time
[2026-07-11 20:35:57] Ignore import error when loading sglang.srt.multimodal.processors.mimo_v2: No module named 'torchcodec'
[2026-07-11 20:35:57] Ignore import error when loading sglang.srt.multimodal.processors.mimo_v2_asr: Can not import FA3 in sgl_kernel. Please check your installation.
[2026-07-11 20:35:57] Init torch distributed begin.
[2026-07-11 20:35:58] Init torch distributed ends. elapsed=0.23 s, mem usage=0.03 GB
[2026-07-11 20:35:58] Ignore import error when loading sglang.srt.models.mimo_audio: Can not import FA3 in sgl_kernel. Please check your installation.
[2026-07-11 20:35:58] Ignore import error when loading sglang.srt.models.mimo_v2: Can not import FA3 in sgl_kernel. Please check your installation.
[2026-07-11 20:35:58] Ignore import error when loading sglang.srt.models.mimo_v2_asr: Can not import FA3 in sgl_kernel. Please check your installation.
[2026-07-11 20:35:58] Ignore import error when loading sglang.srt.models.mimo_v2_nextn: Can not import FA3 in sgl_kernel. Please check your installation.
[2026-07-11 20:35:58] Load weight begin. avail mem=114.84 GB
[2026-07-11 20:35:58] Multimodal attention backend not set. Use triton_attn.
[2026-07-11 20:35:58] Using triton_attn as multimodal attention backend.
[2026-07-11 20:35:58] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
[transformers] torch_dtype is deprecated! Use dtype instead!
[2026-07-11 20:35:58] Scheduler hit an exception: Traceback (most recent call last):
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/managers/scheduler.py", line 4313, in run_scheduler_process
scheduler = Scheduler(
server_args,
...<7 lines>...
dp_rank,
)
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/managers/scheduler.py", line 423, in init
self.init_model_worker()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/managers/scheduler.py", line 866, in init_model_worker
self.init_tp_model_worker()
~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/managers/scheduler.py", line 787, in init_tp_model_worker
self.tp_worker = TpModelWorker(**worker_kwargs)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/managers/tp_worker.py", line 273, in init
self._init_model_runner()
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/managers/tp_worker.py", line 385, in _init_model_runner
self._model_runner = ModelRunner(
~~~~~~~~~~~^
model_config=self.model_config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<15 lines>...
draft_model_idx=0 if self.is_multi_layer_eagle else None,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/model_executor/model_runner.py", line 581, in init
self.initialize()
~~~~~~~~~~~~~~~^^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/model_executor/model_runner.py", line 702, in initialize
self.load_model()
~~~~~~~~~~~~~~~^^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/model_executor/model_runner.py", line 1510, in load_model
self.model = self.loader.load_model(
~~~~~~~~~~~~~~~~~~~~~~^
model_config=self.model_config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
device_config=DeviceConfig(self.device, self.gpu_id),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/model_loader/loader.py", line 761, in load_model
model = _initialize_model(
model_config,
self.load_config,
quant_config,
)
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/model_loader/loader.py", line 320, in _initialize_model
return model_class(**kwargs)
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/models/qwen3_5.py", line 1646, in init
super().init(config, quant_config, prefix, language_model_cls)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/models/qwen3_vl.py", line 1264, in init
self.model = language_model_cls(
~~~~~~~~~~~~~~~~~~^
config=self.config,
^^^^^^^^^^^^^^^^^^^
quant_config=quant_config,
^^^^^^^^^^^^^^^^^^^^^^^^^^
prefix=add_prefix("model.language_model", prefix),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/models/qwen3_5.py", line 1239, in init
self.layers, self._start_layer, self._end_layer = make_layers(
~~~~~~~~~~~^
config.num_hidden_layers,
^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
prefix=f"{prefix}.layers",
^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/utils/common.py", line 1451, in make_layers
+ get_offloader().wrap_modules(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
(
^
...<3 lines>...
**(offloader_kwargs or {}),
^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/utils/offloader.py", line 36, in wrap_modules
return list(all_modules_generator)
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/utils/common.py", line 1453, in
layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/models/qwen3_5.py", line 1230, in get_layer
return layer_class(
config=config,
...<4 lines>...
is_nextn=is_nextn,
)
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/models/qwen3_5.py", line 622, in init
self.mlp = Qwen2MoeMLP(
~~~~~~~~~~~^
hidden_size=config.hidden_size,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
prefix=add_prefix("mlp", prefix.replace(".linear_attn", "")),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/models/qwen2_moe.py", line 186, in init
self.gate_up_proj = MergedColumnParallelLinear(
~~~~~~~~~~~~~~~~~~~~~~~~~~^
hidden_size,
^^^^^^^^^^^^
...<5 lines>...
tp_size=tp_size,
^^^^^^^^^^^^^^^^
)
^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/layers/linear.py", line 534, in init
super().init(
~~~~~~~~~~~~~~~~^
input_size=input_size,
^^^^^^^^^^^^^^^^^^^^^^
...<9 lines>...
use_presharded_weights=use_presharded_weights,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/layers/linear.py", line 331, in init
super().init(
~~~~~~~~~~~~~~~~^
input_size, output_size, skip_bias_add, params_dtype, quant_config, prefix
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/layers/linear.py", line 181, in init
self.quant_method = quant_config.get_quant_method(self, prefix=prefix)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 172, in get_quant_method
scheme = self.get_linear_scheme(layer=layer, layer_name=prefix)
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 798, in get_linear_scheme
scheme = self._get_scheme_from_parts( # type: ignore
weight_quant=weight_quant,
input_quant=input_quant,
)
File "/home/lvyou/sglang-official/.venv/lib/python3.13/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 634, in _get_scheme_from_parts
raise NotImplementedError("No compressed-tensors compatible scheme was found.")
NotImplementedError: No compressed-tensors compatible scheme was found.

[2026-07-11 20:35:58] Received sigquit from a child process. It usually means the child failed.
[2026-07-11 20:35:58] kill_process_tree called: parent_pid=112876, include_parent=True, pid=112876
Killed
(sglang-official) lvyou@gx10-4002:~/sglang-official$ nvidia-smi
Sat Jul 11 20:36:04 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GB10 On | 0000000F:01:00.0 Off | N/A |
| N/A 38C P8 3W / N/A | Not Supported | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 3946 G /usr/lib/xorg/Xorg 18MiB |
| 0 N/A N/A 4064 G /usr/bin/gnome-shell 6MiB |
+-----------------------------------------------------------------------------------------+
(sglang-official) lvyou@gx10-4002:~/sglang-official$

algorithmcookies

11 days ago

I have been following the guide to run it on DGX Spark
I installed a new uv venv, vllm version 0.24.0, and having issue below:
[core.py:1231] ValueError: --linear-backend=flashinfer_b12x was requested but no 'flashinfer_b12x' kernel exists for this layer type.

I did check the kernels, and it's shown below:
cap (12, 1) | b12x gemm True | b12x moe True

Anyone can help with this issue? Appreciated it!

originaltrini0

11 days ago

I am having the same errors as @algorithmcookies on vllm 0.25.0
Also tried sglang 0.5.15 and got the same error as @lvyou

Thanks

danielhanchen

Unsloth AI org 11 days ago

I have been following the guide to run it on DGX Spark
I installed a new uv venv, vllm version 0.24.0, and having issue below:
[core.py:1231] ValueError: --linear-backend=flashinfer_b12x was requested but no 'flashinfer_b12x' kernel exists for this layer type.

I did check the kernels, and it's shown below:
cap (12, 1) | b12x gemm True | b12x moe True

Anyone can help with this issue? Appreciated it!

I am having the same errors as @algorithmcookies on vllm 0.25.0
Also tried sglang 0.5.15 and got the same error as @lvyou

Thanks

We don't have a working DGX Spark at home atm, we may need to ask NVIDIA for help or someone to figure out what's wrong so we can update everyone! :(

bnenu

10 days ago

•

edited 10 days ago

@algorithmcookies , @originaltrini0
Hi,
First let me say thank you Unsloth for the great work :). I managed to run this model on a GB10 Blackwell (DGX Spark like ) following the main instructions.

Model: unsloth/Qwen3.6-27B-NVFP4 (unsloth's 2.5× faster NVFP4 dynamic quant)
Runtime: vLLM 0.24.0 + nvidia-cutlass-dsl 4.5.2 in a uv venv (baremetal, not Docker)
Spec decode: MTP (native multi-token prediction, num_speculative_tokens=2) — no draft
model, no patches
Launch command:

  export CUTE_DSL_ARCH=sm_121a
  ./qwen36-nvfp4/bin/vllm serve ~/models/Qwen3.6-27B-NVFP4 \
    --served-model-name qwen36-27b \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 262144 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 8 \
    --trust-remote-code \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --language-model-only \
    --default-chat-template-kwargs '{"enable_thinking": true}' \
    --seed 0

I dropped --moe-backend flashinfer_b12x --linear-backend flashinfer_b12x,because this 27b Dense uses mixed quantization and forcing --linear-backend flashinfer_b12x globally crashes with no 'flashinfer_b12x' kernel exists for this layer type because b12x has no FP8 kernel. Theoretically vllm should auto-select depending on the leyer and -moe-backend is not relevant for this model. I also had to update python3.12-dev (triton JIT for Gated DeltaNet kernels) and ninja-build (flashinfer JIT for cutlass SM120 FP4 GEMM).  Not sure what the target performance was intended but I got these figures on the first bench, if someone know what we shook have expected would be great to compare notes.

  ### Results (pp=128, tg=128, concurrency=1, 30 runs)

 ┌────────────────────┬─────────────────────┐
 │ Metric             │ Value               │
 ├────────────────────┼─────────────────────┤
 │ Decode throughput  │ 23.9 tok/s (±1.65)  │
 ├────────────────────┼─────────────────────┤
 │ Peak decode        │ 26.9 tok/s          │
 ├────────────────────┼─────────────────────┤
 │ Prefill throughput │ 793.8 tok/s (±42.9) │
 ├────────────────────┼─────────────────────┤
 │ TTFT               │ 298ms (±9.2)        │
 └────────────────────┴─────────────────────┘

 ### Comparison

 ┌──────────────────────────────────────────┬────────────────┐
 │ Setup                                    │ tok/s (decode) │
 ├──────────────────────────────────────────┼────────────────┤
 │ This setup (NVFP4 + MTP, vLLM)           │ 23.9           │
 ├──────────────────────────────────────────┼────────────────┤
 │ Qwen3.6-27B Q8_K_XL UD + MTP (llama.cpp) │ 12-15          │
 ├──────────────────────────────────────────┼────────────────┤
 │ Qwen3.6-27B Q4_K_M + DFlash (Lucebox)    │ 27-35          │
 ├──────────────────────────────────────────┼────────────────┤
 │ Qwen3.6-27B Q4_K_M stock (llama.cpp)     │ 7-11           │
 └──────────────────────────────────────────┴────────────────┘

NOTE on the Lucebox: the results could only be obtained while setting —temperature 0.0 so it wasn’t quite suitable for me, but I am still investigating.  

I don’t know about the output quality since I just started using smaller quants, I was pretty happy with Q8 as it did perform very well for my work with just minor exceptions on complex tasks, If anyone has some more experience with how 4bits quants produce outputs vs 8bits would like to know , the small improvement in speed is significant enough to be worth observing.

celikburak

10 days ago

I have been following the guide to run it on DGX Spark
I installed a new uv venv, vllm version 0.24.0, and having issue below:
[core.py:1231] ValueError: --linear-backend=flashinfer_b12x was requested but no 'flashinfer_b12x' kernel exists for this layer type.

I did check the kernels, and it's shown below:
cap (12, 1) | b12x gemm True | b12x moe True

Anyone can help with this issue? Appreciated it!

I am having the same errors as @algorithmcookies on vllm 0.25.0
Also tried sglang 0.5.15 and got the same error as @lvyou

Thanks

We don't have a working DGX Spark at home atm, we may need to ask NVIDIA for help or someone to figure out what's wrong so we can update everyone! :(

https://github.com/NNNtrance/Qwen3.6-35B-A3B-NVFP4-Fast-DGX-Spark

nanbogong

10 days ago

This workaround doesn't work at all; we can't get it running on DGX Spark.

ValueError: moe_backend='flashinfer_b12x' is not supported for unquantized MoE. Expected one of ['triton', 'flashinfer_trtllm', 'flashinfer_cutlass', 'aiter'].

fmarcano

9 days ago

•

edited 9 days ago

Thanks for the great work on these quants, and for the pinned DGX Spark instructions!
I'd like to contribute a case from a different runtime path than the baremetal vLLM 0.24/0.25 setups discussed above, in case it helps others.

Environment: DGX Spark (GB10 Blackwell), running the NVIDIA vLLM container nvcr.io/nvidia/vllm:26.04-py3 (vLLM 0.19.0) — not baremetal. Many Spark users run this container, so it's a common path.
The problem: The current main revision (ccdaab7) fails during weight loading with:

ValueError: There is no module or parameter named 'lm_head.weight_scale' in Qwen3_5ForCausalLM.
The available parameters belonging to lm_head are: {'lm_head.weight'}

Root cause I traced: Between the earlier revision 890bdef7a42feba6d83b6e17a03315c694112f2a and the current ccdaab7, the ignore list in config.json changed the lm_head entry from the literal string "lm_head" to the regex form "re:.*lm_head". vLLM 0.19's compressed-tensors loader doesn't resolve that regex for the output head, so it tries to load a quantized lm_head and aborts. (The broader "quantized lm_head not supported" issue is fixed in vLLM 0.22.1rc1.dev33+, but the NVIDIA container ships 0.19, so many Spark users hit this.)
Working fix on vLLM 0.19: pin the older revision and force offline mode:
--revision 890bdef7a42feba6d83b6e17a03315c694112f2a
plus the env var HF_HUB_OFFLINE=1 on the container.
With that, the server starts cleanly and both MTP and the vision/multimodal path work. Reverting the config ignore entry back to the literal "lm_head" (or documenting a vLLM 0.22.1+ requirement) would restore out-of-the-box 0.19 compatibility.

On backend flags: for this dense 27B (mixed quantization), --moe-backend flashinfer_b12x / --linear-backend flashinfer_b12x should be dropped — those are for the 35B MoE and cause the "no flashinfer_b12x kernel exists" error others hit above. I'm running with num_speculative_tokens=2, --tool-call-parser qwen3_xml, --reasoning-parser qwen3, kv-cache fp8, flashinfer attention, and the vision path enabled (no --language-model-only).

Perf (container vs baremetal): on the vLLM 0.19 container with n=2, once the session warms up I see ~18–22 tok/s decode with MTP acceptance around 87–98% (lower early on, ~16 tok/s / ~65% before the prefix cache fills). For comparison, @bnenu reported 23.9 tok/s on baremetal vLLM 0.24 — so the container is quite usable, with the newer baremetal stack somewhat ahead.

Happy to share more info about this. Thanks again!

danielhanchen

Unsloth AI org 8 days ago

Note for DGX Spark / RTX 6000 / Blackwell, first get the latest vLLM:

uv venv unsloth-nvfp4-env --python 3.13
source unsloth-nvfp4-env/bin/activate
uv pip install "vllm>=0.25.0" "flashinfer-python>=0.6.13" "nvidia-cutlass-dsl>=4.5.2" \
    --torch-backend=auto

Then non DGX Spark:

export CUTE_DSL_ARCH=sm_121a
vllm serve unsloth/Qwen3.6-27B-NVFP4

For DGX Spark:

export CUTE_DSL_ARCH=sm_121a
vllm serve unsloth/Qwen3.6-27B-NVFP4 --moe-backend flashinfer_b12x

SeriousMax

7 days ago

•

edited 7 days ago

I did everything according to the instructions, but I still couldn't run unsloth/Qwen3.6-27B-NVFP4 with vllm 0.25.1 on my DGX Spark:

(EngineCore pid=485152) DEBUG 07-15 12:37:01 [model_executor/.../compressed_tensors/compressed_tensors.py:861] Using scheme: CompressedTensorsW8A8Fp8 for language_model.model.layers.63.mlp.down_proj
(EngineCore pid=485152) DEBUG 07-15 12:37:01 [compilation/backends.py:107] Using InductorStandaloneAdaptor
(EngineCore pid=485152) DEBUG 07-15 12:37:01 [model_executor/.../compressed_tensors/compressed_tensors.py:861] Using scheme: CompressedTensorsW8A8Fp8 for language_model.lm_head
(EngineCore pid=485152) DEBUG 07-15 12:37:02 [config/compilation.py:1278] enabled custom ops: Counter({'mm_encoder_attn': 27, 'apply_rotary_emb': 27})
(EngineCore pid=485152) DEBUG 07-15 12:37:02 [config/compilation.py:1279] disabled custom ops: Counter({'gemma_rms_norm': 161, 'quant_fp8': 145, 'silu_and_mul': 64, 'rms_norm_gated': 48, 'chunk_gated_delta_rule': 48, 'rotary_embedding': 2, 'apply_rotary_emb': 2, 'conv3d': 1})
(EngineCore pid=485152) DEBUG 07-15 12:37:02 [model_executor/model_loader/base_loader.py:63] Loading weights on cuda ...
(EngineCore pid=485152) DEBUG 07-15 12:37:02 [model_executor/model_loader/weight_utils.py:506] Using model weights format [['model-00005-of-00005.safetensors', 'model-00003-of-00005.safetensors', 'model-00004-of-00005.safetensors', 'model-00002-of-00005.safetensors', 'model-00001-of-00005.safetensors']]
(APIServer pid=484948) DEBUG 07-15 12:37:04 [v1/engine/utils.py:1262] Waiting for 1 local, 0 remote core engine proc(s) to start.

I'm getting infinite "Waiting for 1 local, 0 remote core engine proc(s) to start". But the nvidia/Qwen3.6-27B-NVFP4 runs without any issues.

moranilt

7 days ago

•

edited 7 days ago

For 27b and 35b-a3b the same issue for DGX SPARK GB10 in docker.

Dockerfile:

FROM vllm/vllm-openai:v0.25.0

RUN pip install --no-cache-dir --upgrade pip setuptools wheel

RUN pip install --no-cache-dir "nvidia-cutlass-dsl>=4.5.2"

RUN pip install --no-cache-dir "flashinfer-python>=0.6.13"

docker-compose:

  qwen-moe:
    build: .
    container_name: qwen36-moe
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0'] 
              capabilities: [gpu]
    environment:
      - HF_TOKEN=token
      - CUTE_DSL_ARCH=sm_121a
      - CUDA_VISIBLE_DEVICES=0
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    ipc: host
    ports:
      - "8000:8000"
    command: >
      --model unsloth/Qwen3.6-35B-A3B-NVFP4
      --port 8000
      --reasoning-parser qwen3
      --dtype auto
      --kv-cache-dtype fp8
      --tensor-parallel-size 1
      --max-num-seqs 8
      --max-model-len 262000
      --gpu-memory-utilization 0.7
      --trust-remote-code
      --enable-prefix-caching
      --enable-prompt-tokens-details
      --max-num-batched-tokens 65536
      --enable-auto-tool-choice
      --tool-call-parser qwen3_coder
      --moe-backend flashinfer_b12x
      --enable-chunked-prefill
      --async-scheduling
      --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
      --default-chat-template-kwargs '{"preserve_thinking":false}'

Error:

ValueError: moe_backend='flashinfer_b12x' is not supported for FP8 MoE. Expected one of ['triton', 'deep_gemm', 'cutlass', 'flashinfer_trtllm', 'flashinfer_cutlass', 'marlin', 'humming', 'aiter', 'hpc'].
qwen36-moe  | [rank0]:[W716 05:15:58.851442254 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

SeriousMax

6 days ago

•

edited 6 days ago

I've been fighting for this model for the second day now 🫠 It's just stops with that log and I see no CPU usage. RAM used 32.9 from 122G according to htop.
But it is a good sign, right - Using 'FLASHINFER_B12X' NvFp4 MoE backend out of potential backends?

$ python -c "
import torch; from vllm.utils.flashinfer import has_flashinfer_b12x_gemm as g, has_flashinfer_b12x_moe as m
cap = torch.cuda.get_device_capability(); print('cap', cap, '| b12x gemm', g(), '| b12x moe', m()); assert cap[0] == 12 and g() and m(), 'b12x unavailable: serving would degrade to marlin W4A16'"
cap (12, 1) | b12x gemm True | b12x moe True

$ export CUTE_DSL_ARCH=sm_121a
vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast --moe-backend flashinfer_b12x
(APIServer pid=1691148) INFO 07-16 20:23:41 [api_utils.py:339] 
(APIServer pid=1691148) INFO 07-16 20:23:41 [api_utils.py:339]        █     █     █▄   ▄█
(APIServer pid=1691148) INFO 07-16 20:23:41 [api_utils.py:339]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.25.1
(APIServer pid=1691148) INFO 07-16 20:23:41 [api_utils.py:339]   █▄█▀ █     █     █     █  model   unsloth/Qwen3.6-35B-A3B-NVFP4-Fast
(APIServer pid=1691148) INFO 07-16 20:23:41 [api_utils.py:339]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1691148) INFO 07-16 20:23:41 [api_utils.py:339] 
(APIServer pid=1691148) INFO 07-16 20:23:41 [api_utils.py:273] non-default args: {'model_tag': 'unsloth/Qwen3.6-35B-A3B-NVFP4-Fast', 'model': 'unsloth/Qwen3.6-35B-A3B-NVFP4-Fast', 'moe_backend': 'flashinfer_b12x'}
config.json: 100%|█████████████████████████████████████████████████████████████| 22.4k/22.4k [00:00<00:00, 76.0MB/s]
processor_config.json: 100%|███████████████████████████████████████████████████| 1.19k/1.19k [00:00<00:00, 9.61MB/s]
preprocessor_config.json: 100%|████████████████████████████████████████████████████| 390/390 [00:00<00:00, 4.52MB/s]
(APIServer pid=1691148) INFO 07-16 20:23:47 [model.py:619] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=1691148) INFO 07-16 20:23:47 [model.py:1776] Using max model len 262144
(APIServer pid=1691148) INFO 07-16 20:23:47 [vllm.py:1042] Asynchronous scheduling is enabled.
(APIServer pid=1691148) INFO 07-16 20:23:47 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
tokenizer_config.json: 100%|███████████████████████████████████████████████████| 1.17k/1.17k [00:00<00:00, 4.99MB/s]
vocab.json: 100%|██████████████████████████████████████████████████████████████| 6.72M/6.72M [00:00<00:00, 55.6MB/s]
tokenizer.json: downloading bytes: █████████████████████████████████████████████████████████████| 6.23MB,  604kB/s  
tokenizer.json: reconstructing file: 100%|█████████████████████████████████████████████| 20.0MB / 20.0MB, 1.94MB/s  
chat_template.jinja: 100%|█████████████████████████████████████████████████████| 7.76k/7.76k [00:00<00:00, 43.4MB/s]
generation_config.json: 100%|██████████████████████████████████████████████████████| 214/214 [00:00<00:00, 2.39MB/s]
video_preprocessor_config.json: 100%|██████████████████████████████████████████████| 385/385 [00:00<00:00, 4.47MB/s]
(APIServer pid=1691148) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=1691422) INFO 07-16 20:24:09 [core.py:114] Initializing a V1 LLM engine (v0.25.1) with config: model='unsloth/Qwen3.6-35B-A3B-NVFP4-Fast', speculative_config=None, tokenizer='unsloth/Qwen3.6-35B-A3B-NVFP4-Fast', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=compressed-tensors, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=unsloth/Qwen3.6-35B-A3B-NVFP4-Fast, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='flashinfer_b12x', linear_backend='auto')
(EngineCore pid=1691422) INFO 07-16 20:24:12 [parallel_state.py:1607] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.20.30.45:52315 backend=nccl
(EngineCore pid=1691422) INFO 07-16 20:24:12 [parallel_state.py:1942] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=1691422) INFO 07-16 20:24:13 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=1691422) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=1691422) INFO 07-16 20:24:25 [gpu_model_runner.py:5209] Starting to load model unsloth/Qwen3.6-35B-A3B-NVFP4-Fast...
(EngineCore pid=1691422) INFO 07-16 20:24:25 [cuda.py:535] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=1691422) INFO 07-16 20:24:25 [mm_encoder_attention.py:373] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=1691422) INFO 07-16 20:24:25 [__init__.py:631] Selected CutlassFP8ScaledMMLinearKernel for CompressedTensorsW8A8Fp8
(EngineCore pid=1691422) INFO 07-16 20:24:25 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(EngineCore pid=1691422) INFO 07-16 20:24:25 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(EngineCore pid=1691422) INFO 07-16 20:24:25 [deep_gemm.py:120] DeepGEMM E8M0 enabled on current platform.
(EngineCore pid=1691422) INFO 07-16 20:24:25 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=1691422) INFO 07-16 20:24:25 [__init__.py:974] Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
(EngineCore pid=1691422) INFO 07-16 20:24:25 [nvfp4.py:239] Using 'FLASHINFER_B12X' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION'].
(EngineCore pid=1691422) INFO 07-16 20:24:25 [cuda.py:476] Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN'].
model.safetensors.index.json: downloading bytes: ███████████████████████████████████████████████|  922kB, 90.0kB/s  
model.safetensors.index.json: reconstructing file: 100%|███████████████████████████████| 14.3MB / 14.3MB, 1.40MB/s

tanbuikim7

6 days ago

remove
#--moe-backend flashinfer_b12x
my current on sparks. Limit seqs =3 to avoid prefill lockdown
deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu]


entrypoint: ["/bin/bash", "-c"]
command: >
  "python3 -m vllm.entrypoints.openai.api_server
  --model unsloth/Qwen3.6-27B-NVFP4
  --trust-remote-code
  --gpu-memory-utilization 0.45
  --max-model-len 262144
  --load-format instanttensor
  --max-num-seqs 3
  --enable-prefix-caching
  --served-model-name qwen36-27b-coder
  --enable-auto-tool-choice
  --tool-call-parser qwen3_coder
  --reasoning-parser qwen3
  --dtype bfloat16
  --kv-cache-dtype fp8
  --enable-chunked-prefill
  --max-num-batched-tokens 8192
  --override-generation-config '{\"temperature\": 0.6, \"top_p\": 0.95, \"top_k\": 20, \"presence_penalty\": 0.00, \"repetition_penalty\": 1.05}'
  --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'"

Thanks Unsloth team, works like charm. haven't seen loop yet, sessions range from 6 to 20 hours (I have multiple ~20k -50k loc apps, for each app: deep dive audit/debug --> remediation --> review)
(APIServer pid=1) INFO 07-17 05:12:16 [loggers.py:273] Engine 000: Avg prompt throughput: 183.4 tokens/s, Avg generation throughput: 37.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 94.0%, MM cache hit rate: 50.0% (APIServer pid=1) INFO 07-17 05:12:16 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.37, Accepted throughput: 26.50 tokens/s, Drafted throughput: 33.60 tokens/s, Accepted: 265 tokens, Drafted: 336 tokens, Per-position acceptance rate: 0.911, 0.768, 0.688, Avg Draft acceptance rate: 78.9% (APIServer pid=1) INFO: 172.18.0.1:45606 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-17 05:12:26 [loggers.py:273] Engine 000: Avg prompt throughput: 318.8 tokens/s, Avg generation throughput: 46.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.0%, Prefix cache hit rate: 94.0%, MM cache hit rate: 50.0% (APIServer pid=1) INFO 07-17 05:12:26 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.42, Accepted throughput: 32.69 tokens/s, Drafted throughput: 40.49 tokens/s, Accepted: 327 tokens, Drafted: 405 tokens, Per-position acceptance rate: 0.881, 0.807, 0.733, Avg Draft acceptance rate: 80.7% (APIServer pid=1) INFO: 172.18.0.1:45606 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.18.0.1:45606 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO: 172.18.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1) INFO 07-17 05:12:36 [loggers.py:273] Engine 000: Avg prompt throughput: 229.4 tokens/s, Avg generation throughput: 40.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.8%, Prefix cache hit rate: 94.0%, MM cache hit rate: 50.0% (APIServer pid=1) INFO 07-17 05:12:36 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.68, Accepted throughput: 29.79 tokens/s, Drafted throughput: 33.29 tokens/s, Accepted: 298 tokens, Drafted: 333 tokens, Per-position acceptance rate: 0.964, 0.910, 0.811, Avg Draft acceptance rate: 89.5% (APIServer pid=1) INFO 07-17 05:12:46 [loggers.py:273] Engine 000: Avg prompt throughput: 403.4 tokens/s, Avg generation throughput: 41.5 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.0%, Prefix cache hit rate: 94.0%, MM cache hit rate: 50.0% (APIServer pid=1) INFO 07-17 05:12:46 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.69, Accepted throughput: 30.09 tokens/s, Drafted throughput: 33.59 tokens/s, Accepted: 301 tokens, Drafted: 336 tokens, Per-position acceptance rate: 0.964, 0.884, 0.839, Avg Draft acceptance rate: 89.6%

SeriousMax

6 days ago

What version of vllm are you using? On v0.25.1, the unsloth nvfp4 models don't start no matter what I do.

tanbuikim7

6 days ago

(APIServer pid=1) INFO 07-17 03:11:50 [api_utils.py:339]
(APIServer pid=1) INFO 07-17 03:11:50 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=1) INFO 07-17 03:11:50 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.1
(APIServer pid=1) INFO 07-17 03:11:50 [api_utils.py:339] █▄█▀ █ █ █ █ model unsloth/Qwen3.6-27B-NVFP4
(APIServer pid=1) INFO 07-17 03:11:50 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=1) INFO 07-17 03:11:50 [api_utils.py:339]
(APIServer pid=1) INFO 07-17 03:11:50 [api_utils.py:273] non-default args: {'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'model': 'unsloth/Qwen3.6-27B-NVFP4', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_model_len': 262144, 'served_model_name': ['qwen36-27b-coder'], 'override_generation_config': {'temperature': 0.6, 'top_p': 0.95, 'top_k': 20, 'presence_penalty': 0.0, 'repetition_penalty': 1.05}, 'load_format': 'instanttensor', 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.45, 'kv_cache_dtype': 'fp8', 'enable_prefix_caching': True, 'max_num_batched_tokens': 8192, 'max_num_seqs': 3, 'enable_chunked_prefill': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 3}}
(APIServer pid=1) WARNING 07-17 03:11:50 [envs.py:2041] Unknown vLLM environment variable detected: VLLM_BUILD_URL
(APIServer pid=1) WARNING 07-17 03:11:50 [envs.py:2041] Unknown vLLM environment variable detected: VLLM_IMAGE_TAG
(APIServer pid=1) WARNING 07-17 03:11:50 [envs.py:2041] Unknown vLLM environment variable detected: VLLM_BUILD_PIPELINE
(APIServer pid=1) WARNING 07-17 03:11:50 [envs.py:2041] Unknown vLLM environment variable detected: VLLM_BUILD_COMMIT
(APIServer pid=1) INFO 07-17 03:12:04 [model.py:619] Resolved architecture: Qwen3_5ForConditionalGeneration

build with docker file (comments in Dockerfile by gg AImode)
more Dockerfile.vllm.instanttensor FROM vllm/vllm-openai:latest-ubuntu2404



	
		
	
	
		1. Build tools (Cần thiết để compile một số custom ops)
	

RUN apt-get update && apt-get install -y 
    git 
    build-essential 
    && rm -rf /var/lib/apt/lists/*

	
		
	
	
		2. Upgrade pip stack & install uv để cài đặt thư viện cực nhanh và an toàn
	

RUN pip install --no-cache-dir --upgrade pip setuptools wheel && 
    pip install --no-cache-dir uv

	
		
	
	
		3. Extra libs chính xác cho Unsloth NVFP4 theo khuyến nghị của tác giả
	


	
		
	
	
		Lưu ý: Bỏ qua auto-round (vì mô hình đã được quant trước bởi Unsloth sang NVFP4)
	


	
		
	
	
		và auto-gptq (không tương thích với cấu trúc NVFP4 Blackwell).
	

RUN uv pip install --system 
    "flashinfer-python>=0.6.13" 
    "nvidia-cutlass-dsl>=4.5.2" 
    instanttensor

	
		
	
	
		4. Lưu ý về flash-attn:
	


	
		
	
	
		Base image của vLLM ĐÃ CÓ SẴN phiên bản flash-attn tối ưu riêng cho engine vLLM.
	


	
		
	
	
		KHÔNG chạy lệnh cài đè flash-attn để tránh lỗi xung đột symbol nhị phân (binary mismatch).

lvyou

5 days ago

Note for DGX Spark / RTX 6000 / Blackwell, first get the latest vLLM:

uv venv unsloth-nvfp4-env --python 3.13
source unsloth-nvfp4-env/bin/activate
uv pip install "vllm>=0.25.0" "flashinfer-python>=0.6.13" "nvidia-cutlass-dsl>=4.5.2" \
    --torch-backend=auto

Then non DGX Spark:

export CUTE_DSL_ARCH=sm_121a
vllm serve unsloth/Qwen3.6-27B-NVFP4

For DGX Spark:

export CUTE_DSL_ARCH=sm_121a
vllm serve unsloth/Qwen3.6-27B-NVFP4 --moe-backend flashinfer_b12x

tested working. this is my mtp config:
CUTE_DSL_ARCH=sm_121a vllm serve nvidia/Qwen3.6-27B-NVFP4 --moe-backend flashinfer_b12x
--reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice
--enable-prefix-caching
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'
--max-num-batched-tokens 8192 --enable-chunked-prefill
--kv-cache-memory-bytes 15G
--max-num-seqs 30

for spark:
without mtp 11.05 t/s stable
mtp=3 20~31 t/s avg: 28t/s avg accept rate 85% (sweet point ,mtp 4 has no significant improvement for overall gen rate)

but, I also compared to nvidia/Qwen3.6-27B-NVFP4. for excact config(only change 'unsloth' to 'nvidia') and prompt (web page job)
mtp=3 20~31 t/s avg: 33t/s

nvidia is obvious faster than unsloth for non-massive parallel request :

Item	Unsloth, No MTP	NVIDIA, No MTP	Unsloth, MTP=3	NVIDIA, MTP=3
Model	`unsloth/Qwen3.6-27B-NVFP4`	`nvidia/Qwen3.6-27B-NVFP4`	`unsloth/Qwen3.6-27B-NVFP4`	`nvidia/Qwen3.6-27B-NVFP4`
Quantization Format	`compressed-tensors`	`modelopt_mixed`	`compressed-tensors`	`modelopt_mixed`
Weight / Quantization Types	Mainly NVFP4 + FP8	Mixed FP8 / NVFP4 / W4A16_NVFP4 / MXFP8	Mainly NVFP4 + FP8	Mixed FP8 / NVFP4 / W4A16_NVFP4 / MXFP8
Linear / GEMM Backend	FlashInfer CUTLASS NVFP4 + FP8 kernels	ModelOpt mixed kernels; some FP4 paths fall back to Marlin	FlashInfer CUTLASS NVFP4 + FP8 kernels	ModelOpt mixed kernels; some FP4 paths fall back to Marlin
Speculative Decoding	Disabled	Disabled	MTP, 3 speculative tokens	MTP, 3 speculative tokens
CUDA Graph Mode	FULL + PIECEWISE	FULL + PIECEWISE	PIECEWISE	PIECEWISE
Model Memory Usage	~21.34 GiB	Lower than Unsloth	~22.13 GiB	~20.80 GiB
Average Decode Throughput	~11.05 tok/s	~12.25 tok/s	~28.2 tok/s	~33.2 tok/s
Peak / High-Acceptance Throughput	~11.1 tok/s	~12.3 tok/s	~31.6 tok/s	~35.7 tok/s
MTP Draft Throughput	—	—	~23.4–23.7 tok/s	~26.4–27.3 tok/s
MTP Acceptance	—	—	Lower	Higher
Notable Backend Behavior	Native NVFP4 path via FlashInfer	Some FP4 operations fall back to weight-only FP4 via Marlin	Native NVFP4 path via FlashInfer	Some FP4 operations fall back to weight-only FP4 via Marlin

not a rigorous comparison, but I can clearly feel that nvidia is faster .

tanbuikim7

5 days ago

since this unsloth nvfp4 release, no loop in my 24/7 gh Copilot CLI sessions.
It's much better than Nvidia one (nvidia nvfp4 27b gets loop after few hours). This is the best quant for 27b sofar. Approx 30 tps is acceptable for single spark now, and with max 3 seqs , my virtual software house delivers more high quality code. Thanks again 😁

lvyou

5 days ago

since this unsloth nvfp4 release, no loop in my 24/7 gh Copilot CLI sessions.
It's much better than Nvidia one (nvidia nvfp4 27b gets loop after few hours). This is the best quant for 27b sofar. Approx 30 tps is acceptable for single spark now, and with max 3 seqs , my virtual software house delivers more high quality code. Thanks again 😁

I'm testing this now with MTP off for max prefill speed. Currently, my Spark is helping me do text reviews. I input long text, and it only outputs the problems. I split the job into chunks for parallel processing. This is a good reason to prefer unsloth over nvidia: I saw near 4k token/s prefill speed!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment