useful-quants/GLM-4.6V-Flash-W4A16-BF16Vision

GLM-4.6V-Flash W4A16 BF16-Vision

Ampere-oriented quantized serving build of zai-org/GLM-4.6V-Flash.

This repo packages a local-serving variant where:

• the text / MoE weights are stored as compressed-tensors W4A16
• the vision tower remains BF16
• serving is validated through the provided Dockerfile
• the Dockerfile applies a required vLLM loader patch for this package

This is intended as a practical RTX 30xx / 40xx local serving artifact, not as a general “drop into any vLLM install” checkpoint.

Why the Dockerfile is required

This package currently depends on a patched vLLM runtime.

The included Dockerfile applies a loader patch needed for this checkpoint layout, specifically to support loading split Q/K/V vision weights into combined qkv_proj parameters used by the runtime.

Use the provided Docker image unless you have independently applied and validated the same patch in your own vLLM environment.

Recommended serving path

From the root of this model repo:

docker build -t vllm-glm46v-bf16vision .

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8001:8000 \
  -v "$(pwd):/model" \
  vllm-glm46v-bf16vision \
  --model /model \
  --served-model-name GLM-4.6V-Flash-W4A16-BF16Vision \
  --trust-remote-code \
  --dtype bfloat16 \
  --quantization compressed-tensors \
  --max-model-len 8000 \
  --limit-mm-per-prompt '{"image":1}' \
  --allowed-local-media-path /

The --max-model-len 8000 value is the recommended local serving default for this package. It is not a statement about the upstream model’s full trained context length.

Smoke test

After the server is running:

python examples/smoke_image_data_url.py \
  --url http://localhost:8001/v1/chat/completions \
  --model GLM-4.6V-Flash-W4A16-BF16Vision

The smoke test sends an image as a data URL to the OpenAI-compatible /v1/chat/completions endpoint and checks that the model returns a text response.

Expected use

Best fit:

• RTX 30xx / 40xx GPUs
• local multimodal inference
• OpenAI-compatible vLLM serving
• one image per prompt
• moderate context serving defaults

Not the best fit:

• CPU inference
• Apple Silicon inference
• unpatched vLLM installs
• generic Transformers pipeline usage without testing
• long-context stress testing without adjusting memory settings

Notes

• The base model is zai-org/GLM-4.6V-Flash.
• This repo is a quantized serving package derived from the base model.
• Text / MoE weights use W4A16 compressed-tensors.
• Vision weights remain BF16.
• Runtime validation is tied to the supplied Dockerfile.
• If you use a different serving stack, treat it as unsupported until independently smoke-tested.

Files

Expected repo layout:

.
├── README.md
├── LICENSE
├── Dockerfile
├── patches/
│   └── vllm_glm46v_qkv_loader.patch
├── examples/
│   └── smoke_image_data_url.py
├── config.json
├── generation_config.json
├── tokenizer.json
├── tokenizer_config.json
├── preprocessor_config.json
├── chat_template.jinja
├── model.safetensors.index.json
└── *.safetensors

Attribution

This package is derived from zai-org/GLM-4.6V-Flash.