Instructions to use DhruvalLabs/gemma-4-12B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DhruvalLabs/gemma-4-12B-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="DhruvalLabs/gemma-4-12B-GGUF")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("DhruvalLabs/gemma-4-12B-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use DhruvalLabs/gemma-4-12B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use DhruvalLabs/gemma-4-12B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DhruvalLabs/gemma-4-12B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DhruvalLabs/gemma-4-12B-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M

SGLang

How to use DhruvalLabs/gemma-4-12B-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DhruvalLabs/gemma-4-12B-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DhruvalLabs/gemma-4-12B-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DhruvalLabs/gemma-4-12B-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DhruvalLabs/gemma-4-12B-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use DhruvalLabs/gemma-4-12B-GGUF with Ollama:
```
ollama run hf.co/DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use DhruvalLabs/gemma-4-12B-GGUF with Docker Model Runner:
```
docker model run hf.co/DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M
```

Lemonade

How to use DhruvalLabs/gemma-4-12B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull DhruvalLabs/gemma-4-12B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-12B-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

Dhruval commited on Jun 19

Commit

0344b5a

verified ·

1 Parent(s): 7d10227

Upload VLM GGUF quants via quant-kit (batch 1)

Browse files

Files changed (6) hide show

.gitattributes +4 -0
README.md +14 -7
gemma-4-12B-Q2_K.gguf +3 -0
gemma-4-12B-Q3_K_L.gguf +3 -0
gemma-4-12B-Q3_K_M.gguf +3 -0
gemma-4-12B-Q3_K_S.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -38,3 +38,7 @@ gemma-4-12B-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 gemma-4-12B-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 gemma-4-12B-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
 gemma-4-12B-mmproj-f16.gguf filter=lfs diff=lfs merge=lfs -text

 gemma-4-12B-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 gemma-4-12B-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
 gemma-4-12B-mmproj-f16.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-12B-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-12B-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-12B-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-12B-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -3,18 +3,18 @@ license: apache-2.0
 base_model: google/gemma-4-12B
 pipeline_tag: image-text-to-text
 tags:
-  - safetensors
-  - gemma4_unified
   - gguf
-  - multimodal
-  - vision
-  - license:apache-2.0
   - transformers
-  - image-text-to-text
   - quantized
   - vlm
   - region:us
-  - any-to-any
 language:
   - en
 ---
@@ -51,8 +51,15 @@ Works with **[llama.cpp](https://github.com/ggerganov/llama.cpp)** · **[LM Stud
 | Filename | Size | RAM Required | Quant | Quality | Best For |
 |---|---|---|---|---|---|
 | `gemma-4-12B-Q4_K_M.gguf` | 6.87 GB | ~8.4 GB | `Q4_K_M` ✅ **Recommended** | ⭐⭐⭐⭐ | Best balance of size and quality. Recommended for most users. |
 | `gemma-4-12B-Q5_K_M.gguf` | 7.96 GB | ~9.5 GB | `Q5_K_M` | ⭐⭐⭐⭐½ | Better quality than Q4, slightly larger. Great if you have the RAM. |
 | `gemma-4-12B-Q8_0.gguf` | 11.80 GB | ~13.3 GB | `Q8_0` | ⭐⭐⭐⭐⭐ | Closest to original quality. Use when RAM is not a concern. |
 ### 🖼️ Vision Encoder — mmproj (always required, always F16)

 base_model: google/gemma-4-12B
 pipeline_tag: image-text-to-text
 tags:
   - gguf
+  - any-to-any
   - transformers
   - quantized
   - vlm
+  - multimodal
+  - image-text-to-text
+  - vision
+  - gemma4_unified
   - region:us
+  - license:apache-2.0
+  - safetensors
 language:
   - en
 ---
 | Filename | Size | RAM Required | Quant | Quality | Best For |
 |---|---|---|---|---|---|
+| `gemma-4-12B-Q2_K.gguf` | 4.50 GB | ~6.0 GB | `Q2_K` | ⭐ | Extreme compression, significant quality loss. |
+| `gemma-4-12B-Q3_K_L.gguf` | 6.12 GB | ~7.6 GB | `Q3_K_L` | ⭐⭐⭐ | Slightly better than Q3_K_M, still a compromise. |
+| `gemma-4-12B-Q3_K_M.gguf` | 5.67 GB | ~7.2 GB | `Q3_K_M` | ⭐⭐⭐ | Very small file. Quality drop noticeable. |
+| `gemma-4-12B-Q3_K_S.gguf` | 5.15 GB | ~6.6 GB | `Q3_K_S` | ⭐⭐ | Very high compression, high quality loss. |
 | `gemma-4-12B-Q4_K_M.gguf` | 6.87 GB | ~8.4 GB | `Q4_K_M` ✅ **Recommended** | ⭐⭐⭐⭐ | Best balance of size and quality. Recommended for most users. |
+| `gemma-4-12B-Q4_K_S.gguf` | 6.54 GB | ~8.0 GB | `Q4_K_S` | ⭐⭐⭐½ | Good speed/size balance, slight quality loss. |
 | `gemma-4-12B-Q5_K_M.gguf` | 7.96 GB | ~9.5 GB | `Q5_K_M` | ⭐⭐⭐⭐½ | Better quality than Q4, slightly larger. Great if you have the RAM. |
+| `gemma-4-12B-Q5_K_S.gguf` | 7.77 GB | ~9.3 GB | `Q5_K_S` | ⭐⭐⭐⭐ | Large but accurate. |
+| `gemma-4-12B-Q6_K.gguf` | 9.11 GB | ~10.6 GB | `Q6_K` | ⭐⭐⭐⭐⭐ | Near-perfect quality, very large. |
 | `gemma-4-12B-Q8_0.gguf` | 11.80 GB | ~13.3 GB | `Q8_0` | ⭐⭐⭐⭐⭐ | Closest to original quality. Use when RAM is not a concern. |
 ### 🖼️ Vision Encoder — mmproj (always required, always F16)

gemma-4-12B-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:60e7ef8d7d89616c2930bbef0741f3aa49d50c55dfb53e05a1ce3a99558b2578
+size 4830129856

gemma-4-12B-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a8ef80be99edd852a6c516ed6f00e00b31967e92d17956c17d6954c0d282e794
+size 6566301376

gemma-4-12B-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a53bdb763b5e0b79dd4fe63aa9c7e76b2d158118da2d9931b9580de1153df9a7
+size 6087069376

gemma-4-12B-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:13729853ec6b2dbe34042e18306a3a645bf1da9fd949875bca89b50813e63d0d
+size 5528211136