Instructions to use yasu-oh/gemma-4-31B-it-qat-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use yasu-oh/gemma-4-31B-it-qat-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="yasu-oh/gemma-4-31B-it-qat-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("yasu-oh/gemma-4-31B-it-qat-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("yasu-oh/gemma-4-31B-it-qat-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use yasu-oh/gemma-4-31B-it-qat-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "yasu-oh/gemma-4-31B-it-qat-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yasu-oh/gemma-4-31B-it-qat-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/yasu-oh/gemma-4-31B-it-qat-NVFP4

SGLang

How to use yasu-oh/gemma-4-31B-it-qat-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "yasu-oh/gemma-4-31B-it-qat-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yasu-oh/gemma-4-31B-it-qat-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "yasu-oh/gemma-4-31B-it-qat-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yasu-oh/gemma-4-31B-it-qat-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use yasu-oh/gemma-4-31B-it-qat-NVFP4 with Docker Model Runner:
```
docker model run hf.co/yasu-oh/gemma-4-31B-it-qat-NVFP4
```

yasu-oh commited on 7 days ago

Commit

7c3a887

1 Parent(s): 28413b5

v4

Browse files

Files changed (6) hide show

README.md +9 -7
config.json +2 -2
model-00001-of-00002.safetensors +3 -0
model-00002-of-00002.safetensors +3 -0
recipe.yaml +7 -6
tokenizer.json +3 -0

README.md CHANGED Viewed

@@ -46,11 +46,7 @@ The calibration dataset consists of:
 ```yaml
 default_stage:
   default_modifiers:
-    IMatrixGatherer:
-      targets: [Linear]
-      ignore: ['re:.*vision.*', 're:.*audio.*', lm_head, 're:.*embed.*', 're:.*router.*']
-      weight_observer: imatrix_mse
-    QuantizationModifier:
       config_groups:
         group_0:
           targets: [Linear]
@@ -62,7 +58,7 @@ default_stage:
             strategy: tensor_group
             block_structure: null
             dynamic: false
-            actorder: null
             scale_dtype: torch.float8_e4m3fn
             zp_dtype: null
             observer: imatrix_mse
@@ -98,6 +94,11 @@ default_stage:
         observer: static_minmax
         observer_kwargs: {}
       bypass_divisibility_checks: false
 ```
 ## Reproducibility
@@ -110,6 +111,7 @@ The quantization run used:
 ```text
 transformers:        5.12.1
-llmcompressor:       0.12.1.dev87+g9bbfd9564
 ```

 ```yaml
 default_stage:
   default_modifiers:
+    GPTQModifier:
       config_groups:
         group_0:
           targets: [Linear]
             strategy: tensor_group
             block_structure: null
             dynamic: false
+            actorder: static
             scale_dtype: torch.float8_e4m3fn
             zp_dtype: null
             observer: imatrix_mse
         observer: static_minmax
         observer_kwargs: {}
       bypass_divisibility_checks: false
+      requires_calibration_data: true
+      block_size: 128
+      dampening_frac: 0.01
+      actorder: static
+      offload_hessians: false
 ```
 ## Reproducibility
 ```text
 transformers:        5.12.1
+llmcompressor:       0.12.1.dev91+g4bb29d00c
+compressed-tensors:  0.17.2.dev36+g2cff344
 ```

config.json CHANGED Viewed

@@ -40,7 +40,7 @@
           "Linear"
         ],
         "weights": {
-          "actorder": null,
           "block_structure": null,
           "dynamic": false,
           "group_size": 16,
@@ -271,7 +271,7 @@
     "quantization_status": "compressed",
     "sparsity_config": {},
     "transform_config": {},
-    "version": "0.1.dev546+g4ed496d"
   },
   "text_config": {
     "attention_bias": false,

           "Linear"
         ],
         "weights": {
+          "actorder": "static",
           "block_structure": null,
           "dynamic": false,
           "group_size": 16,
     "quantization_status": "compressed",
     "sparsity_config": {},
     "transform_config": {},
+    "version": "0.17.2.dev36+g2cff344"
   },
   "text_config": {
     "attention_bias": false,

model-00001-of-00002.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2e4308e2eb7c61e4d007fb28009c93dd24571662334bac067962de5d7c2e38cf
+size 19994110824

model-00002-of-00002.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9b89e976593cf1c73f63a2618c92920a6e9383d1e91820efcbe944170fde09e9
+size 452731960

recipe.yaml CHANGED Viewed

@@ -1,10 +1,6 @@
 default_stage:
   default_modifiers:
-    IMatrixGatherer:
-      targets: [Linear]
-      ignore: ['re:.*vision.*', 're:.*audio.*', lm_head, 're:.*embed.*', 're:.*router.*']
-      weight_observer: imatrix_mse
-    QuantizationModifier:
       config_groups:
         group_0:
           targets: [Linear]
@@ -16,7 +12,7 @@ default_stage:
             strategy: tensor_group
             block_structure: null
             dynamic: false
-            actorder: null
             scale_dtype: torch.float8_e4m3fn
             zp_dtype: null
             observer: imatrix_mse
@@ -52,3 +48,8 @@ default_stage:
         observer: static_minmax
         observer_kwargs: {}
       bypass_divisibility_checks: false

 default_stage:
   default_modifiers:
+    GPTQModifier:
       config_groups:
         group_0:
           targets: [Linear]
             strategy: tensor_group
             block_structure: null
             dynamic: false
+            actorder: static
             scale_dtype: torch.float8_e4m3fn
             zp_dtype: null
             observer: imatrix_mse
         observer: static_minmax
         observer_kwargs: {}
       bypass_divisibility_checks: false
+      requires_calibration_data: true
+      block_size: 128
+      dampening_frac: 0.01
+      actorder: static
+      offload_hessians: false

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f
+size 32169626