Instructions to use prokopsafranek/gemma-4-26B-A4B-it-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use prokopsafranek/gemma-4-26B-A4B-it-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M

Use Docker

docker model run hf.co/prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use prokopsafranek/gemma-4-26B-A4B-it-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "prokopsafranek/gemma-4-26B-A4B-it-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prokopsafranek/gemma-4-26B-A4B-it-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M

Ollama
How to use prokopsafranek/gemma-4-26B-A4B-it-GGUF with Ollama:
```
ollama run hf.co/prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M
```
Unsloth Desktop

How to use prokopsafranek/gemma-4-26B-A4B-it-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use prokopsafranek/gemma-4-26B-A4B-it-GGUF with Docker Model Runner:
```
docker model run hf.co/prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M
```

Lemonade

How to use prokopsafranek/gemma-4-26B-A4B-it-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-26B-A4B-it-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use prokopsafranek/gemma-4-26B-A4B-it-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use prokopsafranek/gemma-4-26B-A4B-it-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "prokopsafranek/gemma-4-26B-A4B-it-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Gemma 4 26B A4B IT GGUF

License: Apache 2.0

Authors: Google DeepMind

A polished GGUF release of Gemma 4 26B A4B IT, prepared for efficient local inference with llama.cpp and compatible runtimes.

Demo: Gemma 4 26B A4B IT

Overview

Gemma 4 26B A4B IT GGUF is a quantized distribution of the original instruction-tuned base model. The goal of this release is straightforward: preserve as much of the original model quality as possible while making the model easier to run locally and more flexible to deploy.

GGUF is especially useful for users who want:

fast local inference
straightforward compatibility with modern llama.cpp-based tooling
predictable memory requirements
support across desktop and self-hosted environments
a simple model format with broad ecosystem support

Available Quantizations

The following quantizations are included in this release:

Quantization	Focus	Typical Use
Q4_K_M	Smaller footprint, strong efficiency	Best starting point for most local setups
Q5_K_M	Balanced quality and speed	Good default for general use
Q6_K	Higher quality with moderate memory cost	For users who want a stronger output profile
Q8_0	Near-maximal fidelity among common quantizations	Best when memory is less constrained
IQ4_XS	Compact and efficient	Useful for smaller systems and experimentation

If you are unsure which one to choose, Q5_K_M is usually the most practical middle ground. It keeps the model relatively light while preserving a solid amount of the original behavior.

Why This Release Exists

Large models often become impractical the moment they leave hosted inference environments. Memory, throughput, startup time, and runtime compatibility all become real constraints. GGUF solves part of that problem by making the model more accessible for local execution without changing the fundamental character of the underlying model.

This release is aimed at users who want a model that feels polished in practice:

easy to load
easy to run
easy to integrate
easy to reason about

The emphasis is not only on compression, but on usability.

Supported Runtimes

This model is compatible with:

ModelD
llama.cpp
LM Studio
Ollama
Open WebUI
KoboldCpp
other GGUF-compatible tooling

For best results, use a recent version of the runtime you prefer. GGUF support continues to improve quickly, and newer builds generally provide better performance and more stable behavior.

Recommended Use Cases

This model is well suited for:

chat and instruction following
local assistant workflows
writing support
summarization
idea generation
code-adjacent reasoning
offline experimentation
evaluation and prototyping

It is especially useful when you want strong general-purpose behavior without depending on hosted APIs.

Getting Started

A typical local workflow looks like this:

Download the GGUF file that matches your hardware and runtime requirements.
Load it in your preferred GGUF-compatible application.
Use the instruction-tuned model as you would in a chat or assistant setting.
Adjust context, sampling, and runtime parameters to match your use case.

Exact launch commands depend on the runtime you use, but most GGUF tools follow a similar setup pattern: model path, context size, threads or GPU offload settings, and a chat template or system prompt.

Notes on Performance

Actual performance depends on:

the selected quantization
CPU and GPU capabilities
available RAM and VRAM
context length
runtime configuration
sampling settings
batching and offload behavior

In practice, there is no single best configuration. The best choice depends on whether you value responsiveness, memory efficiency, or output quality most.

Model Identity

This repository contains quantizations based on:

google/gemma-4-26b-a4b-it

The original model defines the behavior and overall capabilities. This repository focuses on packaging and deployment convenience through GGUF quantization.

ModelD Deployment

If you want to serve this model locally with a polished self-hosted interface, use ModelD:

Repository: https://github.com/prokopsafranek/modeld
Overview: a local AI model server for managing models, APIs, auth, and monitoring
Documentation: see the ModelD README in that repository for installation and deployment details

ModelD is a good fit when you want a single place to upload, manage, and run GGUF models behind a clean web UI.

Attribution

Quantized by:

Prokop Šafránek

Based on:

google/gemma-4-26b-a4b-it

License

This repository follows the Apache 2.0 license.

Downloads last month: 81

GGUF

Model size

25B params

Architecture

gemma4

Hardware compatibility

4-bit

5-bit

6-bit

8-bit

prokopsafranek
/

gemma-4-26B-A4B-it-GGUF

Gemma 4 26B A4B IT GGUF

Overview

Available Quantizations

Why This Release Exists

Supported Runtimes

Recommended Use Cases

Getting Started

Notes on Performance

Model Identity

ModelD Deployment

Attribution

License

Space using prokopsafranek/gemma-4-26B-A4B-it-GGUF 1