Instructions to use z-lab/Qwen3-4B-DFlash-b16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use z-lab/Qwen3-4B-DFlash-b16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="z-lab/Qwen3-4B-DFlash-b16", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("z-lab/Qwen3-4B-DFlash-b16", trust_remote_code=True)
model = AutoModel.from_pretrained("z-lab/Qwen3-4B-DFlash-b16", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use z-lab/Qwen3-4B-DFlash-b16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "z-lab/Qwen3-4B-DFlash-b16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3-4B-DFlash-b16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/z-lab/Qwen3-4B-DFlash-b16

SGLang

How to use z-lab/Qwen3-4B-DFlash-b16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "z-lab/Qwen3-4B-DFlash-b16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3-4B-DFlash-b16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "z-lab/Qwen3-4B-DFlash-b16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3-4B-DFlash-b16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use z-lab/Qwen3-4B-DFlash-b16 with Docker Model Runner:
```
docker model run hf.co/z-lab/Qwen3-4B-DFlash-b16
```

Benchmark Results

by beyoru - opened Mar 11

Discussion

beyoru

Mar 11

Model: Qwen/Qwen3-4B

Benchmark: sglang

Concurrency	TTFT	E2E	Tok/s	Prompt Type
1	40 ms	4.79 s	53.9	short
1	45 ms	4.85 s	53.2	long
5	45 ms	5.16 s	50.1	short
5	68 ms	5.24 s	49.5	long
10	43 ms	5.18 s	49.6	short
10	267 ms	5.57 s	48.3	long
20	1295 ms	6.75 s	47.0	short
20	2606 ms	8.16 s	46.7	long

Benchmark:sglang + dflash

Concurrency	TTFT	E2E	Tok/s	Prompt Type
1	28 ms	2.86 s	40.1	short
1	29 ms	2.27 s	39.9	long
5	42 ms	3.13 s	34.2	short
5	61 ms	3.31 s	33.2	long
10	46 ms	4.41 s	24.2	short
10	531 ms	4.61 s	26.9	long
20	1844 ms	6.35 s	23.7	short
20	2522 ms	7.21 s	23.0	long

jianchen0311

Z Lab org Mar 11

Did you turn on the thinking or not and what is the acceptance length you got? And also which hardware did you use?

beyoru

Mar 11

•

edited Mar 11

I need to measure again to get this.
Spec accept length (avg): 2.4016
Spec accept length (per DP): [2.4016]
Thinking is on.
Hardware used: RTX 4070 Super.

Here my set up:
SGLANG:

python -m sglang.launch_server \
    --host 127.0.0.1 \
    --port 30000 \
    --model-path Qwen/Qwen3-4B \
    --tensor-parallel-size 1 \
    --dtype bfloat16 \
    --attention-backend fa3 \
    --mem-fraction-static 0.75 \
    --trust-remote-code

SGLANG + DFLASH

python -m sglang.launch_server \
    --host 127.0.0.1 \
    --port 30000 \
    --model-path Qwen/Qwen3-4B \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/Qwen3-4B-DFlash-b16 \
    --tensor-parallel-size 1 \
    --dtype bfloat16 \
    --attention-backend fa3 \
    --mem-fraction-static 0.75 \
    --trust-remote-code

jianchen0311

Z Lab org Mar 11

@beyoru Thanks. I have also granted the access to Qwen3.5-4B-DFlash for you. That DFlash draft model is able to work with thinking mode turned on and the performance will be more consistent than Qwen3-4B-DFlash.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment