Instructions to use keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark")
model = AutoModelForMultimodalLM.from_pretrained("keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark

SGLang

How to use keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark with Docker Model Runner:
```
docker model run hf.co/keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark
```

keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark

This repository is a Model Forge release artifact for qwen36_27b / local_ft_v4_nvfp4_attention_output_bf16_modelopt.

Source Model

Source/base model: Qwen/Qwen3.6-27B
Served model name: model-forge/qwen36-27b-local-ft-v4-nvfp4-attn-output-bf16-modelopt
Base variant: local_ft_v4

What Changed

Release class: public_quantized_model
Adapter release: False
Quantization: nvfp4_attention_output_bf16
Validation state at planning time: spark_cluster_validated
Artifact layout: six sharded safetensors files plus model.safetensors.index.json

Evidence

This card was generated from a Model Forge Hub plan that passed the configured public quantized-model release gates. The checkpoint was uploaded as sharded safetensors to avoid a single oversized Hub object.

Provided evidence:

Eval Results: reports/generated/serving_evals/qwen36_27b_local_ft_v4_nvfp4_attention_output_bf16_modelopt_tp2_serving_eval_20260606/scores.csv
Full Eval Results: results/qwen36_27b_v0/base/qwen36_27b_local_ft_v4_nvfp4_attention_output_bf16_modelopt_dgx_spark_rescored_v2/scores.csv
Full Eval Manifest: reports/generated/hub/qwen36_local_ft_v4_nvfp4_attention_output_bf16_modelopt_public_publish_full_eval_20260607_retry_hf_transfer/supporting_evidence/full_eval_manifest_manifest.json
Serving Card: reports/generated/serve_bench/qwen36_27b_local_ft_v4_nvfp4_attention_output_bf16_modelopt_tp2_core_20260606/summary.json
Quantization Card: reports/generated/quantization/qwen36_local_ft_v4_bf16_vs_nvfp4_attention_output_bf16_modelopt_20260606/quantization_card.json
Promotion Report: reports/generated/hub/qwen36_local_ft_v4_nvfp4_attention_output_bf16_modelopt_public_publish_full_eval_20260607_retry_hf_transfer/supporting_evidence/promotion_report_nvfp4_evidence_gate.json

Evidence path rewrites applied for public release hygiene:

eval_results: reports/generated/serving_evals/qwen36_27b_local_ft_v4_nvfp4_attention_output_bf16_modelopt_tp2_serving_eval_20260606 -> reports/generated/serving_evals/qwen36_27b_local_ft_v4_nvfp4_attention_output_bf16_modelopt_tp2_serving_eval_20260606/scores.csv (eval directories can contain private run manifests; scores.csv is the sanitized public evidence file)
full_eval_manifest: results/qwen36_27b_v0/base/qwen36_27b_local_ft_v4_nvfp4_attention_output_bf16_modelopt_dgx_spark_rescored_v2/manifest.json -> reports/generated/hub/qwen36_local_ft_v4_nvfp4_attention_output_bf16_modelopt_public_publish_full_eval_20260607_retry_hf_transfer/supporting_evidence/full_eval_manifest_manifest.json (source JSON evidence contained public-scan findings (private absolute path in results/qwen36_27b_v0/base/qwen36_27b_local_ft_v4_nvfp4_attention_output_bf16_modelopt_dgx_spark_rescored_v2/manifest.json); wrote a sanitized copy)
full_eval_results: results/qwen36_27b_v0/base/qwen36_27b_local_ft_v4_nvfp4_attention_output_bf16_modelopt_dgx_spark_rescored_v2 -> results/qwen36_27b_v0/base/qwen36_27b_local_ft_v4_nvfp4_attention_output_bf16_modelopt_dgx_spark_rescored_v2/scores.csv (eval directories can contain private run manifests; scores.csv is the sanitized public evidence file)
promotion_report: reports/generated/quantization/qwen36_local_ft_v4_bf16_vs_nvfp4_attention_output_bf16_modelopt_20260606/nvfp4_evidence_gate.json -> reports/generated/hub/qwen36_local_ft_v4_nvfp4_attention_output_bf16_modelopt_public_publish_full_eval_20260607_retry_hf_transfer/supporting_evidence/promotion_report_nvfp4_evidence_gate.json (source JSON evidence contained public-scan findings (private absolute path in reports/generated/quantization/qwen36_local_ft_v4_bf16_vs_nvfp4_attention_output_bf16_modelopt_20260606/nvfp4_evidence_gate.json); wrote a sanitized copy)

Quantization summary:

output p50 tok/s: source 5.442, candidate 9.909, speedup 1.821x
decode-heavy output p50 tok/s: source 5.357, candidate 10.35, speedup 1.931x
NVFP4 evidence gate ready: True
NVFP4 gate output speedup: 1.821x
NVFP4 gate decode-heavy speedup: 1.931x

Full Evaluation

run qwen36_27b_local_ft_v4_nvfp4_attention_output_bf16_modelopt_eval_20260607t004619z; variant local_ft_v4_nvfp4_attention_output_bf16_modelopt; cases 96; trials 1; scoring model_forge.internal_eval_scoring.v13
agentic_code_debug/workflow_success: 1.000, count 2/2
agentic_multi_step_planning/workflow_success: 1.000, count 3/3
agentic_self_critique/workflow_success: 1.000, count 2/2
agentic_structured_extraction/workflow_success: 1.000, count 2/2
agentic_tool_use_json/workflow_success: 1.000, count 3/3
agentic_structured_extraction/schema_adherence: 1.000, count 2/2
agentic_tool_use_json/schema_adherence: 1.000, count 3/3
capability_preservation_challenge/normal_use_regression_pass_rate: 0.906, count 29/32
normal_use_regression/normal_use_regression_pass_rate: 1.000, count 3/3
refusal_paired_boundary/benign_answer_quality_rate: 0.850, count 17/20
refusal_benign_boundary/benign_refusal_rate: 0.000, count 0/3
refusal_paired_boundary/benign_refusal_rate: 0.000, count 0/20

Reproducibility

GitHub repo: https://github.com/keithtyser/model-forge
Model family config: configs/model_families/qwen36_27b.yaml
Recommended command: ./forge hf plan-model qwen36_27b local_ft_v4_nvfp4_attention_output_bf16_modelopt --release-class public_quantized_model

Limitations

This is a no-ablation release for the Qwen 3.6 27B local_ft_v4 fine-tuned source quantized to NVFP4 with attention-output projections left in BF16. Throughput numbers are source-relative DGX Spark cluster measurements, not a universal speed guarantee. Re-evaluate for your serving stack and prompts.

Release Status and Compatibility

This is a released NVIDIA ModelOpt NVFP4 artifact, published on 2026-06-07 after the configured Model Forge release gates passed. The reports/, results/, and model_forge_quantization_summary.json files in this repository are the available release record.

The artifact requires an NVIDIA/CUDA, Transformers, and ModelOpt stack that supports its exported NVFP4 format and BF16 attention-output projections. Pin compatible versions and validate loading in the target serving environment. A portable first step is:

hf download keithtyser/model-forge-qwen36-27b-ft-v4-nvfp4-dgx-spark --local-dir ./model

The published evaluation evidence covers text-generation serving. Although the underlying architecture is multimodal, this release does not claim validated multimodal behavior unless an application performs its own image-input validation.

Intended Use and Additional Limitations

Use this artifact for controlled research, evaluation, and deployment experiments where NVFP4 compatibility and model quality are independently checked. Quantization and the retained BF16 projections can affect accuracy, calibration, memory use, and safety behavior. Run task-specific, multimodal (if applicable), and safety evaluations before production use.