Instructions to use huginnfork/Qwen3.8-27B-NVFP4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use huginnfork/Qwen3.8-27B-NVFP4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="huginnfork/Qwen3.8-27B-NVFP4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("huginnfork/Qwen3.8-27B-NVFP4A16")
model = AutoModelForMultimodalLM.from_pretrained("huginnfork/Qwen3.8-27B-NVFP4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use huginnfork/Qwen3.8-27B-NVFP4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "huginnfork/Qwen3.8-27B-NVFP4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "huginnfork/Qwen3.8-27B-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/huginnfork/Qwen3.8-27B-NVFP4A16

SGLang

How to use huginnfork/Qwen3.8-27B-NVFP4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "huginnfork/Qwen3.8-27B-NVFP4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "huginnfork/Qwen3.8-27B-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "huginnfork/Qwen3.8-27B-NVFP4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "huginnfork/Qwen3.8-27B-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use huginnfork/Qwen3.8-27B-NVFP4A16 with Docker Model Runner:
```
docker model run hf.co/huginnfork/Qwen3.8-27B-NVFP4A16
```

Qwen3.8-27B-NVFP4A16 / recipe.yaml

huginnfork

Upload folder using huggingface_hub

645e83e verified about 1 month ago

Raw

History Blame Contribute Delete

1.85 kB

	name: nvfp4a16_attnbf16
	scheme: NVFP4A16 # FP4 weights, bf16 activations — skips activation quant for much lower KLD
	engine: llmcompressor

	# Variant of `nvfp4a16.yaml` that additionally keeps the ENTIRE self-attention
	# block in bf16, leaving only the MLPs quantised — the same lever measured on
	# ThinkingCap for FP8 (landmine 36), applied to the W4A16 path.
	#
	# The motivating fact is architectural, not scheme-specific: `attn_output_gate:
	# true` fuses the attention output gate into `q_proj`, making it
	# [2headshead_dim, hidden] instead of [heads*head_dim, hidden]. Half that
	# tensor is a multiplicative per-head gate on what attention writes into the
	# residual stream, and only the full_attention layers (a quarter of the stack)
	# carry long-range retrieval. Quantisation error on a multiplicative gate
	# behaves worse than on an additive projection.
	#
	# Cost/benefit differs from the FP8 case and should be re-measured, not assumed:
	# at W4A16 the attention block would otherwise compress 4x rather than 2x, so
	# holding it in bf16 costs proportionally more on-disk than the +4.6% measured
	# for FP8. The KLD win is expected to be larger too, since NVFP4A16's error
	# floor is 2-3x FP8's. Ship whichever the measured KLD/size tradeoff justifies —
	# `nvfp4a16.yaml` remains the standard build.

	calibration:
	dataset: neuralmagic/calibration
	config: LLM
	split: train
	num_samples: 128
	max_seq_length: 2048

	ignore:
	- lm_head
	- "re:.visual."
	- "re:.linear_attn." # entire SSM block kept in bf16 — same rationale as the standard build
	- "re:.self_attn." # THE VARIANT: q/k/v/o_proj too, incl. the output gate fused into q_proj
	- "re:.mtp."
	# Note: dense base. On MoE bases also add "re:.*mlp.gate$" and
	# "re:.*mlp.shared_expert_gate$" — no-ops on dense models.

	export:
	save_compressed: true