Instructions to use RedHatAI/GLM-5.3-Flash-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/GLM-5.3-Flash-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="RedHatAI/GLM-5.3-Flash-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("RedHatAI/GLM-5.3-Flash-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("RedHatAI/GLM-5.3-Flash-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/GLM-5.3-Flash-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/GLM-5.3-Flash-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-Flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/GLM-5.3-Flash-NVFP4

SGLang

How to use RedHatAI/GLM-5.3-Flash-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/GLM-5.3-Flash-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-Flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/GLM-5.3-Flash-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/GLM-5.3-Flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/GLM-5.3-Flash-NVFP4 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/GLM-5.3-Flash-NVFP4
```

dsikka commited on 2 days ago

Commit

bee1cbd

verified ·

1 Parent(s): 9eaeada

Add FP8-block MTP layer (layer 45): weights + index + mixed-precision quant config

Browse files

Files changed (3) hide show

config.json +53 -2
model.safetensors.index.json +2 -2
model_mtp.safetensors +3 -0

config.json CHANGED Viewed

@@ -27,7 +27,7 @@
         },
         "output_activations": null,
         "targets": [
-          "re:.*mlp\\.experts\\..*(gate|up|down)_proj$"
         ],
         "weights": {
           "actorder": null,
@@ -43,9 +43,47 @@
           "type": "float",
           "zp_dtype": null
         }
       }
     },
-    "format": "nvfp4-pack-quantized",
     "global_compression_ratio": null,
     "ignore": [
       "model.visual.blocks.0.attn.qkv",
@@ -811,6 +849,19 @@
       "model.language_model.layers.44.mlp.shared_experts.gate_proj",
       "model.language_model.layers.44.mlp.shared_experts.up_proj",
       "model.language_model.layers.44.mlp.shared_experts.down_proj",
       "lm_head"
     ],
     "kv_cache_scheme": null,

         },
         "output_activations": null,
         "targets": [
+          "re:.*\\.layers\\.(?:[3-9]|[1-3][0-9]|4[0-4])\\.mlp\\.experts\\..*(gate|up|down)_proj$"
         ],
         "weights": {
           "actorder": null,
           "type": "float",
           "zp_dtype": null
         }
+      },
+      "group_1": {
+        "format": "float-quantized",
+        "input_activations": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": true,
+          "group_size": 128,
+          "num_bits": 8,
+          "observer": null,
+          "observer_kwargs": {},
+          "scale_dtype": null,
+          "strategy": "group",
+          "symmetric": true,
+          "type": "float",
+          "zp_dtype": null
+        },
+        "output_activations": null,
+        "targets": [
+          "re:.*\\.layers\\.45\\.mlp\\.experts\\.\\d+\\.(gate_proj|up_proj|down_proj)$"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": [
+            128,
+            128
+          ],
+          "dynamic": false,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": "memoryless_minmax",
+          "observer_kwargs": {},
+          "scale_dtype": null,
+          "strategy": "block",
+          "symmetric": true,
+          "type": "float",
+          "zp_dtype": null
+        }
       }
     },
+    "format": "mixed-precision",
     "global_compression_ratio": null,
     "ignore": [
       "model.visual.blocks.0.attn.qkv",
       "model.language_model.layers.44.mlp.shared_experts.gate_proj",
       "model.language_model.layers.44.mlp.shared_experts.up_proj",
       "model.language_model.layers.44.mlp.shared_experts.down_proj",
+      "model.language_model.layers.45.eh_proj",
+      "model.language_model.layers.45.self_attn.q_a_proj",
+      "model.language_model.layers.45.self_attn.q_b_proj",
+      "model.language_model.layers.45.self_attn.kv_a_proj_with_mqa",
+      "model.language_model.layers.45.self_attn.kv_b_proj",
+      "model.language_model.layers.45.self_attn.o_proj",
+      "model.language_model.layers.45.self_attn.indexer.wq_b",
+      "model.language_model.layers.45.self_attn.indexer.wk",
+      "model.language_model.layers.45.self_attn.indexer.weights_proj",
+      "model.language_model.layers.45.mlp.gate",
+      "model.language_model.layers.45.mlp.shared_experts.gate_proj",
+      "model.language_model.layers.45.mlp.shared_experts.up_proj",
+      "model.language_model.layers.45.mlp.shared_experts.down_proj",
       "lm_head"
     ],
     "kv_cache_scheme": null,

model.safetensors.index.json CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b42e3ccb127abb24a914eeed0fcd51ccb80eecaabb7d3e5550bfb1f3cdfd2027
-size 16884180

 version https://git-lfs.github.com/spec/v1
+oid sha256:015faae91e8189c7553f1d48ec3d0694b8c02b282d7f58af2d7b4064a81ce4c0
+size 17055736

model_mtp.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0cf8dfdc936f1cbc33b9e88d0289ff3b5b4a3d0f7381158a8ccbebb9c8adc035
+size 7618560424