Instructions to use google/gemma-4-E2B-it-qat-q4_0-unquantized with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use google/gemma-4-E2B-it-qat-q4_0-unquantized with Transformers:
# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("google/gemma-4-E2B-it-qat-q4_0-unquantized") model = AutoModelForMultimodalLM.from_pretrained("google/gemma-4-E2B-it-qat-q4_0-unquantized", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README (#4)
Browse files- Update README (d906d4555894900049015fd8e72d11c59c06d170)
README.md
CHANGED
|
@@ -29,6 +29,7 @@ base_model:
|
|
| 29 |
> * **GGUF** (Q4_0): Ready-to-deploy formats for broad ecosystem compatibility. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B.
|
| 30 |
> * **Mobile-optimized** (wNa8o8): A custom schema engineered explicitly for mobile hardware efficiency. It features targeted 2-bit decoding layers, optimized KV caches, and static activations to maximize VRAM savings. Available for Gemma 4 E2B and E4B.
|
| 31 |
> * **Compressed Tensors** (w4a16): QAT checkpoints serialized in the compressed-tensors format for native, optimized inference with vLLM. Available for Gemma 4 E2B, E4B, 12B, and 31B.
|
|
|
|
| 32 |
|
| 33 |
Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages.
|
| 34 |
|
|
@@ -144,7 +145,7 @@ Once you have everything installed, you can proceed to load the model with the c
|
|
| 144 |
```python
|
| 145 |
from transformers import AutoProcessor, AutoModelForMultimodalLM
|
| 146 |
|
| 147 |
-
MODEL_ID = "google/gemma-4-
|
| 148 |
|
| 149 |
# Load model
|
| 150 |
processor = AutoProcessor.from_pretrained(MODEL_ID)
|
|
@@ -199,7 +200,7 @@ You can then load the model with the code below:
|
|
| 199 |
```python
|
| 200 |
from transformers import AutoProcessor, AutoModelForMultimodalLM
|
| 201 |
|
| 202 |
-
MODEL_ID = "google/gemma-4-
|
| 203 |
|
| 204 |
# Load model
|
| 205 |
processor = AutoProcessor.from_pretrained(MODEL_ID)
|
|
@@ -258,7 +259,7 @@ You can then load the model with the code below:
|
|
| 258 |
```python
|
| 259 |
from transformers import AutoProcessor, AutoModelForMultimodalLM
|
| 260 |
|
| 261 |
-
MODEL_ID = "google/gemma-4-
|
| 262 |
|
| 263 |
# Load model
|
| 264 |
processor = AutoProcessor.from_pretrained(MODEL_ID)
|
|
@@ -316,7 +317,7 @@ You can then load the model with the code below:
|
|
| 316 |
```python
|
| 317 |
from transformers import AutoProcessor, AutoModelForMultimodalLM
|
| 318 |
|
| 319 |
-
MODEL_ID = "google/gemma-4-
|
| 320 |
|
| 321 |
# Load model
|
| 322 |
processor = AutoProcessor.from_pretrained(MODEL_ID)
|
|
@@ -391,7 +392,7 @@ Compared to Gemma 3, the models use standard `system`, `assistant`, and `user` r
|
|
| 391 |
|
| 392 |
### 3. Multi-Turn Conversations
|
| 393 |
|
| 394 |
-
* **No Thinking Content in History**: In multi-turn conversations, the historical model output should only include the final response. Thoughts from previous model turns must *not be added* before the next user turn begins.
|
| 395 |
|
| 396 |
### 4. Modality order
|
| 397 |
|
|
|
|
| 29 |
> * **GGUF** (Q4_0): Ready-to-deploy formats for broad ecosystem compatibility. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B.
|
| 30 |
> * **Mobile-optimized** (wNa8o8): A custom schema engineered explicitly for mobile hardware efficiency. It features targeted 2-bit decoding layers, optimized KV caches, and static activations to maximize VRAM savings. Available for Gemma 4 E2B and E4B.
|
| 31 |
> * **Compressed Tensors** (w4a16): QAT checkpoints serialized in the compressed-tensors format for native, optimized inference with vLLM. Available for Gemma 4 E2B, E4B, 12B, and 31B.
|
| 32 |
+
> * **Assistant Compatibility**: When using multi-token prediction (speculative decoding) with an assistant model alongside a QAT target model, the assistant model must also be a QAT checkpoint with the same precision to ensure compatibility.
|
| 33 |
|
| 34 |
Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages.
|
| 35 |
|
|
|
|
| 145 |
```python
|
| 146 |
from transformers import AutoProcessor, AutoModelForMultimodalLM
|
| 147 |
|
| 148 |
+
MODEL_ID = "google/gemma-4-E2B-it"
|
| 149 |
|
| 150 |
# Load model
|
| 151 |
processor = AutoProcessor.from_pretrained(MODEL_ID)
|
|
|
|
| 200 |
```python
|
| 201 |
from transformers import AutoProcessor, AutoModelForMultimodalLM
|
| 202 |
|
| 203 |
+
MODEL_ID = "google/gemma-4-E2B-it"
|
| 204 |
|
| 205 |
# Load model
|
| 206 |
processor = AutoProcessor.from_pretrained(MODEL_ID)
|
|
|
|
| 259 |
```python
|
| 260 |
from transformers import AutoProcessor, AutoModelForMultimodalLM
|
| 261 |
|
| 262 |
+
MODEL_ID = "google/gemma-4-E2B-it"
|
| 263 |
|
| 264 |
# Load model
|
| 265 |
processor = AutoProcessor.from_pretrained(MODEL_ID)
|
|
|
|
| 317 |
```python
|
| 318 |
from transformers import AutoProcessor, AutoModelForMultimodalLM
|
| 319 |
|
| 320 |
+
MODEL_ID = "google/gemma-4-E2B-it"
|
| 321 |
|
| 322 |
# Load model
|
| 323 |
processor = AutoProcessor.from_pretrained(MODEL_ID)
|
|
|
|
| 392 |
|
| 393 |
### 3. Multi-Turn Conversations
|
| 394 |
|
| 395 |
+
* **No Thinking Content in History**: In multi-turn conversations, the historical model output should only include the final response. Thoughts from previous model turns must *not be added* before the next user turn begins, with the exception of tool call turns where thinking content should be preserved.
|
| 396 |
|
| 397 |
### 4. Modality order
|
| 398 |
|