Instructions to use islamassanov/Kimi-K3-Mixed-IQ2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use islamassanov/Kimi-K3-Mixed-IQ2-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16
# Run inference directly in the terminal:
llama cli -hf islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16
# Run inference directly in the terminal:
./llama-cli -hf islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16

Use Docker

docker model run hf.co/islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16

LM Studio
Jan

vLLM

How to use islamassanov/Kimi-K3-Mixed-IQ2-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "islamassanov/Kimi-K3-Mixed-IQ2-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "islamassanov/Kimi-K3-Mixed-IQ2-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16

Ollama
How to use islamassanov/Kimi-K3-Mixed-IQ2-GGUF with Ollama:
```
ollama run hf.co/islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16
```
Unsloth Desktop
Docker Model Runner
How to use islamassanov/Kimi-K3-Mixed-IQ2-GGUF with Docker Model Runner:
```
docker model run hf.co/islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16
```

Lemonade

How to use islamassanov/Kimi-K3-Mixed-IQ2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull islamassanov/Kimi-K3-Mixed-IQ2-GGUF:BF16

Run and chat with the model

lemonade run user.Kimi-K3-Mixed-IQ2-GGUF-BF16

List all available models

lemonade list

Atomic Chat

Kimi K3 2-bit Mixed-IQ2 GGUF

This repository provides a 2-bit mixed-IQ2 quantization of MoonshotAI/Kimi-K3, packaged as a standard split GGUF for llama.cpp.

Highlights

2.449913 bpw quantized checkpoint tensor rate
2.450403 bpw complete GGUF rate, including metadata and alignment
10 numbered GGUF shards with a BF16 vision projector
Per-file SHA-256 hashes in verification-receipt.json

The routed expert weights use 146,048 IQ2_XXS matrices and 101,248 IQ2_XS matrices. Another 2,628 tensors remain in protected higher-precision formats.

Why this quantization is different

This checkpoint uses a decoder-aligned IQ2_XXS scale fit (iq2_decoded_grid_scale_uniform_canonical_v1). Stock IQ2_XXS fits scales against the idealized magnitudes 1, 3, 5, although its decoder actually reconstructs 1, 3.125, 5.375. The decoder-aligned encoder optimizes for those real reconstruction values without changing the block layout, decoder, or storage size.

Equal-byte fidelity measurement	Stock IQ2_XXS	Decoder-aligned IQ2_XXS	Improvement
Activation-weighted matrix SQNR	9.72418 dB	9.99648 dB	+0.27230 dB
Bounded train expert replay	9.99486 dB	10.57623 dB	+0.58136 dB
Bounded dev expert replay	10.01970 dB	10.46470 dB	+0.44500 dB
Source reconstruction error	948,896.68174	896,088.61273	-5.5652%
Matrices improved	—	1,104 / 1,104	No sampled regressions

Decoder-aligned IQ2 quantization delivered superior dequantized matrix-multiplication fidelity versus stock IQ2 at equal bytes.

The final checkpoint combines this improved IQ2_XXS encoder with selective IQ2_XS upgrades for the routed matrices that benefit most from additional precision. These measurements describe reconstruction and expert-replay fidelity rather than task-accuracy percentage points.

Quick start

Download the complete repository:

hf download islamassanov/Kimi-K3-Mixed-IQ2-GGUF \
  --local-dir Kimi-K3-Mixed-IQ2-GGUF

Start an OpenAI-compatible llama.cpp server by loading the first shard:

llama-server \
  -m Kimi-K3-Mixed-IQ2-GGUF/Kimi-K3-Mixed-IQ2-00001-of-00010.gguf \
  --mmproj Kimi-K3-Mixed-IQ2-GGUF/mmproj-Kimi-K3-BF16.gguf

llama.cpp discovers the remaining shards automatically. The projector is required for image inputs.

Benchmark comparison

Benchmark	This GGUF	Base Kimi K3	Score retention
OCRBench	871 / 1,000 (87.1%)	890 / 1,000 (89.0%)	97.9%

This quantization retains 97.9% of the base model's OCRBench score, using Moonshot AI's published 89.0% result as the reference.

License

Use of this model is subject to the Kimi K3 license.

Contacts

If you have any questions or suggestions please feel free to reach out: islam@uni.minerva.edu

Downloads last month: 361

GGUF

Model size

2.8T params

Architecture

kimi-k3

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Inference Providers NEW

Image-Text-to-Text

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support