Instructions to use Qwen/Qwen3.8-27B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3.8-27B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Qwen/Qwen3.8-27B-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Qwen/Qwen3.8-27B-FP8")
model = AutoModelForMultimodalLM.from_pretrained("Qwen/Qwen3.8-27B-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/Qwen3.8-27B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3.8-27B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3.8-27B-FP8

SGLang

How to use Qwen/Qwen3.8-27B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3.8-27B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3.8-27B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3.8-27B-FP8 with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3.8-27B-FP8
```

look like Loops but not loops.

#11

by tanbuikim7 - opened about 1 month ago

Discussion

tanbuikim7

about 1 month ago

My setup is dual spark with CX7, serves Qwen3.5 122b fp8 or ds4 flash 0731. So I created recipe TP2 and run with eugr run-recipe.sh
#Result:

utilization .25 on each node have kv cache pool approx 500k
prefill approx 4kps
tg: 3x (max 36, avg 31)

#Agent:
Copilot CLI w reasoning effort= xhigh
Router orchestration mode: main agent only calls task tools, delgates to custom agents.

#Task:
I asked to review >10 audit report files generated by Qwen3.5 122b fp8. Initial just ok.

router delegated researcher to read all reports quickly with thinkings about 20-30s
once completed reading router thought for 803s ( outcome was todo list + delegate 5 sub agents to verify against the reports.)
4 parallel sub agents were running fine for approx. 20' with one completed. Then eventually I saw only 1 remain request to vllm and loop forever with MTP accepted always 100%. I need to shutdown at 40'.

Now I'm asking ds4 flash 0731 do the job. Will return to Qwen3.8 27b later with simpler tasks.

#My recipe, Not the best yet,
-- Recipe: Qwen3.8-27B-FP8 -- Target: dual Spark / TP=2, Copilot CLI coding worker recipe_version: "1" name: qwen3.8-27b-fp8 description: vLLM serving Qwen3.8-27B-FP8 with FlashInfer, FP8 KV, MTP, prefix cache model: Qwen/Qwen3.8-27B-FP8


container: ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest

defaults: port: 8026 host: 0.0.0.0 served_model_name: qwen38-27b-coder tensor_parallel: 2 gpu_memory_utilization: 0.25 max_model_len: 262144 max_num_seqs: 4 max_num_batched_tokens: 8192 attention_backend: flashinfer tool_call_parser: qwen3_coder reasoning_parser: qwen3 load_format: auto kv_cache_dtype: fp8 env: NVIDIA_VISIBLE_DEVICES: all NVIDIA_DRIVER_CAPABILITIES: compute,utility PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
'''

alex81k

27 days ago

try froggeric chat template

tanbuikim7

24 days ago

•

edited 24 days ago

Today , weekend, I have more time to work with this FP8, original chat template w xhigh thinking, 3 harness sessions in parallel: at certain time, 100% mtp hit (speculation 3) with 2 requests were running for a while (5 to 10 minutes, just 99% , 100%). I just let it run overnight (approx 10 hours ), then the jobs completed without error.

After this first pass all 3 sessions did well in my complex django monorepo: refactor ui ux in one module (A), review ai pipeline in another module (B), and review complex SSE mcp tools + chat UI in 3rd module (C). Now I Only need minor adjust in A with a form missing Select2 search, B hasn't been thoroughly tested, C better result than DS V4 flash 0731, understand what I need to have.

Ds4 flash struggled with mcp tool calls + chat display for some days already: backend have bugs, UI UX just messed around, That's why I decided to test again with Qwen3.8 27B FP8 (coding)+ Qwen3.6 35B A3B FP8 (in app llm) on dual sparks.

Great model so far. Thanks

tanbuikim7 changed discussion title from Loops to look like Loops but not loops. 24 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment