Instructions to use KikoCis/gemma-4-31b-it-IQ3_XS-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use KikoCis/gemma-4-31b-it-IQ3_XS-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS
# Run inference directly in the terminal:
llama cli -hf KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS
# Run inference directly in the terminal:
llama cli -hf KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS
# Run inference directly in the terminal:
./llama-cli -hf KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS

Use Docker

docker model run hf.co/KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS

LM Studio
Jan

vLLM

How to use KikoCis/gemma-4-31b-it-IQ3_XS-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KikoCis/gemma-4-31b-it-IQ3_XS-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KikoCis/gemma-4-31b-it-IQ3_XS-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS

Ollama
How to use KikoCis/gemma-4-31b-it-IQ3_XS-GGUF with Ollama:
```
ollama run hf.co/KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS
```
Unsloth Desktop

How to use KikoCis/gemma-4-31b-it-IQ3_XS-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use KikoCis/gemma-4-31b-it-IQ3_XS-GGUF with Docker Model Runner:
```
docker model run hf.co/KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS
```

Lemonade

How to use KikoCis/gemma-4-31b-it-IQ3_XS-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS

Run and chat with the model

lemonade run user.gemma-4-31b-it-IQ3_XS-GGUF-IQ3_XS

List all available models

lemonade list

Hermes Agent

How to use KikoCis/gemma-4-31b-it-IQ3_XS-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use KikoCis/gemma-4-31b-it-IQ3_XS-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "KikoCis/gemma-4-31b-it-IQ3_XS-GGUF:IQ3_XS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Gemma 4 31B IT — IQ3_XS GGUF

Gemma 4 31B Instruct quantized to IQ3_XS (3.40 BPW, 13.1GB) using llama.cpp with importance matrix calibration on CLI/package management data.

Matches f16 baseline on Char-F1 (84.76%) at 1/4.7 the size (50-example NL2Bash benchmark).

💡 Looking for the smallest possible variant?

Try our gemma-4-31b-it-IQ2_M-GGUF — 10.17 GB with F1 84.71% and BLEU-4 22.39 (beats f16 at 6× smaller). Custom CLI-tuned imatrix on IQ2_M.

Key Stats

Metric	Value
Base model	google/gemma-4-31b-it
Quantization	IQ3_XS (3.40 BPW)
Size	13.1 GB
Layers	60 (full model, no pruning)
NL2Bash Char-F1	84.76% (= f16 baseline)
CLI 7/7	7/7 with thinking enabled

NL2Bash Benchmark — Full Comparison

50 examples from the official Stanford/Tellina test split, reasoning OFF, max_tokens=200, temp=0.1, sorted by Char-F1.

Model	Size	BPW	Char-F1	BLEU-1	BLEU-2	BLEU-4	EM
Unsloth UD-IQ3_XXS	11.84 GB	~2.98	85.06%	46.30	34.64	20.26	12%
Base f16 (full precision)	61.4 GB	16.0	84.76%	43.94	33.84	21.02	12%
Base IQ3_XS (this model)	13.1 GB	3.40	84.76%	42.77	31.46	18.95	8%
✨ Sibling IQ2_M (CLI imatrix)	10.17 GB	2.84	84.71%	44.72	34.36	22.39	12%
Unsloth UD-IQ2_M	10.75 GB	~2.70	84.02%	42.38	31.73	18.64	10%
Unsloth UD-IQ2_XXS	8.53 GB	~2.06	76.82%	33.03	22.17	11.49	0%
Base Q2_K (3rd party)	11.0 GB	2.70	58.60%	18.82	12.79	6.92	0%

Key observations

This IQ3_XS ties f16 on Char-F1 (84.76% vs 84.76%) and comes within 2 F1 of the best available quantization
Unsloth Dynamic 2.0 is the Pareto frontier: UD-IQ2_M (10.75 GB) is strictly better than plain Q2_K (+25.4 F1 points at same BPW) thanks to per-layer adaptive bit allocation and Gemma-4-specific imatrix
Q2_K at 2.7 BPW collapses on Gemma 4 (F1 58.6%, produces repetitive output). Standard 2-bit scalar quantization is not viable for this architecture
f16 only wins on token-exact metrics (BLEU-4, EM) — IQ3_XS produces semantically equivalent commands with minor stylistic variations (different flag order, quoting style)

Recommendation

Smallest working quant: Unsloth UD-IQ2_M (10.75 GB, F1 84%)
Best overall: Unsloth UD-IQ3_XXS (11.84 GB, F1 85%)
This model (IQ3_XS, 13.1 GB): simpler imatrix calibration (CLI-focused), slightly larger but within 0.3 F1 of UD-IQ3_XXS

Full per-question predictions + layer analysis study: https://huggingface.co/datasets/KikoCis/gemma4-31b-layer-study

CLI Benchmark Results (7/7 with thinking)

Tested with llama-cli -cnv --reasoning on --reasoning-budget 512:

Test	Result
Install neofetch on Void Linux	`sudo xbps-install -S neofetch`
Install htop on Ubuntu	`sudo apt install htop`
Search ripgrep on Arch	`pacman -Ss ripgrep`
Search packages on Void	`xbps-query -S <package_name>`
Add cargo to PATH in zsh	`echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> ~/.zshrc`
Install jq on macOS	`brew install jq`
Grep TODO in /var/www	`grep -r "TODO" /var/www`

Usage

# llama.cpp with thinking
llama-cli -m gemma4-31b-IQ3_XS.gguf -cnv -ngl 99 --ctx-size 8192 \
  --reasoning on --reasoning-budget 512

# Ollama
cat > Modelfile << 'EOF'
FROM ./gemma4-31b-IQ3_XS.gguf
PARAMETER temperature 0.1
PARAMETER num_ctx 8192
EOF
ollama create gemma4-31b-iq3xs -f Modelfile

Quantization Details

Tool: llama.cpp (build 0d049d6)
Imatrix: Computed from 200 chunks of synthetic CLI/package management data
Source: Converted from google/gemma-4-31b-it safetensors via convert_hf_to_gguf.py
Architecture: Gemma4 with sliding + full attention pattern (every 6th layer is full attention)

Hardware Requirements

Minimum RAM: 16GB (with partial offload)
Recommended: Apple Silicon with 32GB+ unified memory
Performance: ~21 tok/s on Apple Silicon (128GB unified) with full GPU offload

Benchmark Methodology

NL2Bash test set: 50 examples from the official deterministic split (RANDOM_SEED=100, fold 11 from TellinaTool/nl2bash)
Inference: Each question runs as an isolated llama-cli subprocess with -p prompt, --reasoning off, --ctx-size 4096, --max-tokens 200, --temp 0.1
Scoring: NLTK BLEU corpus-level + character-level F1 + exact match
Reproducibility: All scripts and JSONL predictions are public at the dataset repo above

Real-World Agent Test Warning (April 2026)

Benchmark scores do not predict agent capability. In Docker-based autonomous testing, fine-tuned E4B models (95% BFCL) scored 0/10 while the unfine-tuned base scored 6/10. Fine-tuning for BFCL destroyed general reasoning (error recovery, strategy adaptation, anti-repetition). Fine-tuned E4B models have been withdrawn.

For autonomous agent tasks, use the base Gemma 4 model or a larger model at higher BPW. See: The Benchmark Trap — Full Study

Downloads last month: 145

GGUF

Model size

31B params

Architecture

gemma4

Hardware compatibility

3-bit