Instructions to use FINAL-Bench/POCKET-26B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FINAL-Bench/POCKET-26B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="FINAL-Bench/POCKET-26B-GGUF",
	filename="POCKET-26B-Q2_K.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use FINAL-Bench/POCKET-26B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FINAL-Bench/POCKET-26B-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf FINAL-Bench/POCKET-26B-GGUF:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FINAL-Bench/POCKET-26B-GGUF:Q2_K
# Run inference directly in the terminal:
llama cli -hf FINAL-Bench/POCKET-26B-GGUF:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf FINAL-Bench/POCKET-26B-GGUF:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf FINAL-Bench/POCKET-26B-GGUF:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf FINAL-Bench/POCKET-26B-GGUF:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf FINAL-Bench/POCKET-26B-GGUF:Q2_K

Use Docker

docker model run hf.co/FINAL-Bench/POCKET-26B-GGUF:Q2_K

LM Studio
Jan

vLLM

How to use FINAL-Bench/POCKET-26B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FINAL-Bench/POCKET-26B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FINAL-Bench/POCKET-26B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/FINAL-Bench/POCKET-26B-GGUF:Q2_K

Ollama
How to use FINAL-Bench/POCKET-26B-GGUF with Ollama:
```
ollama run hf.co/FINAL-Bench/POCKET-26B-GGUF:Q2_K
```

Unsloth Studio

How to use FINAL-Bench/POCKET-26B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FINAL-Bench/POCKET-26B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FINAL-Bench/POCKET-26B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for FINAL-Bench/POCKET-26B-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use FINAL-Bench/POCKET-26B-GGUF with Docker Model Runner:
```
docker model run hf.co/FINAL-Bench/POCKET-26B-GGUF:Q2_K
```

Lemonade

How to use FINAL-Bench/POCKET-26B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull FINAL-Bench/POCKET-26B-GGUF:Q2_K

Run and chat with the model

lemonade run user.POCKET-26B-GGUF-Q2_K

List all available models

lemonade list

POCKET-26B-GGUF / README.md

SeaWolf-AI

POCKET-Gemma-KR model card

1553409 verified 6 days ago

5 kB

license: apache-2.0
library_name: llama.cpp
pipeline_tag: text-generation
base_model:
  - google/gemma-4-26B-A4B-it
tags:
  - gguf
  - llama.cpp
  - conversational
  - on-device
  - mobile
  - korean
  - korean-llm
  - cpu
  - local-llm
  - edge
  - gemma
  - gemma4
  - mixture-of-experts
  - moe
  - imatrix
  - pocket
  - vidraft

📚 Collections

▶ POCKET Models — this family (on-device, no GPU) Darwin Family · Aether Foundation · VKAE Accelerated · Metacognition Adapters

POCKET-Gemma-KR-GGUF · 한국어

A Gemma4-26B-A4B-based pocket model that loads in any app today — Ollama, LM Studio, PocketPal — with no bleeding-edge runtime needed. Korean-tuned, GPU-optional.

🚀 Try the family live, no install →

Pick your build →

Why this one?

POCKET-Gemma-KR takes Google's Gemma4-26B-A4B (25.2B total, ~4B active MoE, Apache-2.0) and re-quantizes it with our Korean imatrix + MoE mixed-precision recipe — unpruned, so quality holds. Unlike our Qwen-based POCKET (which needs a very recent llama.cpp build for its qwen35moe architecture), Gemma4 loads in every mainstream runtime today: Ollama, LM Studio, PocketPal, koboldcpp, and the browser.

Quality — GPQA-Diamond, greedy, 198 questions (our harness)

Build	GPQA-Diamond	vs base
Gemma4-26B-A4B (base)	67.7%	—
POCKET-Gemma-KR `Q4_K_M`	67.7%	= base (lossless)
POCKET-Gemma-KR `Q2_K` (mixed) ⭐	67.2%	−0.5pp (≈ lossless)

Single greedy pass, 198 items → ±~3 pp noise. Our Korean-imatrix + mixed-precision quantization is statistically lossless vs the base.

Files in this repo

File	Size	Runs on	Best for
`POCKET-Gemma-KR-Q4_K_M.gguf`	17 GB	PC / high-RAM	top quality
`POCKET-Gemma-KR-Q2_K.gguf` ⭐	11 GB	12 GB phone / PC / browser	universal daily driver

Mixed precision = shared path (attention, dense MLP, router) kept high-bit, only routing experts at 2-bit. That is why Q2_K holds 67.2% while a plain uniform Q2 collapses to ~44%.

Quickstart — loads anywhere

# stock llama.cpp — brew / winget / apt, or LM Studio / Ollama / PocketPal
llama-cli -m POCKET-Gemma-KR-Q2_K.gguf -p "대한민국의 수도는?" -ngl 0 -t 8

No fork, no bleeding-edge build — Gemma4 support has shipped in every mainstream runtime since April 2026.

Lineage (honest)

Based on google/gemma-4-26B-A4B-it (Apache-2.0). We do not re-host it unchanged — we add Korean-imatrix calibration + MoE mixed-precision quantization (VIDRAFT). We deliberately do not prune it: Gemma4's low-bit robustness collapses under pruning (measured), so we keep all 128 experts and win on quality + universal compatibility instead.

Limitations

For 8 GB phones (~5 GB budget), use POCKET-KR-GGUF (5.1 GB) — Gemma4 cannot be shrunk that far without collapse.
On-device iPhone/Mac throughput not yet measured by us — community reports welcome.

License

Apache-2.0 — use, modify, redistribute freely.

POCKET is a VIDRAFT model family. Runs anywhere, no GPU.