Instructions to use Qwen/Qwen3.8-27B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3.8-27B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Qwen/Qwen3.8-27B-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Qwen/Qwen3.8-27B-FP8")
model = AutoModelForMultimodalLM.from_pretrained("Qwen/Qwen3.8-27B-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/Qwen3.8-27B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3.8-27B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3.8-27B-FP8

SGLang

How to use Qwen/Qwen3.8-27B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3.8-27B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3.8-27B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3.8-27B-FP8 with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3.8-27B-FP8
```

Missing pre-tuned FP8 kernel config files for modded RTX 4090 48GB — how do you work around this?

#16

by realxumai - opened 28 days ago

Discussion

realxumai

28 days ago

I'm running a modded RTX 4090 with 48GB VRAM (upgraded from the stock 24GB), and deploying vLLM v0.27.1 using the official Docker image. I noticed the following warnings in the logs:

(EngineCore pid=1243) WARNING 08-18 00:52:03 [fp8_utils.py:851] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=5120,K=6144,device_name=NVIDIA_GeForce_RTX_4090,dtype=fp8_w8a8,block_shape=[128,128].json

(EngineCore pid=1243) WARNING 08-18 00:52:03 [fp8_utils.py:851] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=34816,K=5120,device_name=NVIDIA_GeForce_RTX_4090,dtype=fp8_w8a8,block_shape=[128,128].json

(EngineCore pid=1243) WARNING 08-18 00:52:03 [fp8_utils.py:851] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=5120,K=17408,device_name=NVIDIA_GeForce_RTX_4090,dtype=fp8_w8a8,block_shape=[128,128].json

(EngineCore pid=1243) WARNING 08-18 00:52:03 [fp8_utils.py:851] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=14336,K=5120,device_name=NVIDIA_GeForce_RTX_4090,dtype=fp8_w8a8,block_shape=[128,128].json

It seems that vLLM is falling back to the default W8A8 Block FP8 kernel config because it can't find the pre-tuned config JSON files for the RTX 4090. I suspect this might be related to the fact that my card is a 48GB VRAM mod rather than the stock 24GB version — the device name is still reported as NVIDIA_GeForce_RTX_4090, so vLLM may not have pre-tuned configs specifically for this variant.
Has anyone encountered this with a modded RTX 4090 48GB? How did you resolve it? Any tips on generating or obtaining the proper FP8 kernel configs for this setup would be greatly appreciated.

alex81k

27 days ago

I used this tool to create custom for this videocard https://github.com/massif-01/vllm_benchmark_block_fp8
You can download from my repo https://github.com/kryoz/4090-48gb-vllm-fp8

realxumai

27 days ago

•

edited 27 days ago

I used this tool to create custom for this videocard https://github.com/massif-01/vllm_benchmark_block_fp8
You can download from my repo https://github.com/kryoz/4090-48gb-vllm-fp8

Bro, thanks for the help — just starred your repo!

BTW，Were these optimized configs generated on an RTX 4090? What settings did you use when generating them? Like --out-dtype and other args.

alex81k

27 days ago

I used this tool to create custom for this videocard https://github.com/massif-01/vllm_benchmark_block_fp8
You can download from my repo https://github.com/kryoz/4090-48gb-vllm-fp8

Bro, thanks for the help — just starred your repo!

BTW，Were these optimized configs generated on an RTX 4090? What settings did you use when generating them? Like --out-dtype and other args.

That was almost 2 months ago I barely remember the details. But I generated shapes exactly on my RTX 4090 (not D) 48Gb.
I've found the script which I ran with the benchmark that can be useful for you https://gist.github.com/kryoz/81b7a130f0a5574ad4bb1688e571fb02
You should inspect vLLM logs about missing config shapes and specify in the script.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment