Instructions to use Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K
# Run inference directly in the terminal:
llama cli -hf Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K
# Run inference directly in the terminal:
llama cli -hf Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K

Use Docker

docker model run hf.co/Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K

LM Studio
Jan

vLLM

How to use Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K

Ollama
How to use Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED with Ollama:
```
ollama run hf.co/Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K
```

Unsloth Studio

How to use Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED to start chatting

How to use Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED with Docker Model Runner:
```
docker model run hf.co/Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K
```

Lemonade

How to use Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:Q2_K

Run and chat with the model

lemonade run user.KIMI-K3-Q2_K-GGUF-ABLITERATED-Q2_K

List all available models

lemonade list

KIMI-K3-DERISKED-Q2_K-GGUF

All-Q2_K GGUF of de-risked Kimi K3 · 38 shards · llama.cpp PR-only

Built by Blackfrost · Las Vegas, NV

⚠️ EXPERIMENTAL

Stock llama.cpp cannot load this. kimi-k3 support exists only in an unmerged pull request — ggml-org/llama.cpp#26185. Released llama.cpp builds, and everything downstream of them (LM Studio, Ollama, koboldcpp, brew install llama.cpp), will fail to load these files.

Bugs have been observed. Expect sharp edges. Please report them — open a Community discussion with GPU SKU, driver, llama.cpp commit, full serve flags, prompt, sampler settings, and failure mode (load-time / loop / empty content / OOM).

Why this model exists

Full Kimi K3 in GGUF is ~1.5 TB. This is the same de-risked checkpoint requantized to Q2_K so it fits a single 8×B200 node with room for KV — with the refusal surface already reduced at the weight level in the parent.

The single-node GGUF footprint is the product; the refusal behaviour change is inherited from the parent and is the contract.

Specifications


Architecture	Kimi K3 LatentMoE + KDA · `general.architecture = kimi-k3`
Parent	`BlackfrostAI/KIMI-K3-DERISKED-MXFP4` — lossless MXFP4
Transform	Requantization to all-Q2_K · built to PR #26185's KV contract
Experts	896 of 896 routed retained — no pruning
On-disk	~940 GiB (1,009 GB) · 38 shards · 2,573 tensors
Context	8,192 validated · higher untested
Serve shape	1× node · 8× B200 · `-ngl 99 -nr`
Status	EXPERIMENTAL · free, ungated

What "Q2_K" costs here

The experts were already MXFP4 4-bit QAT in the source checkpoint. This is therefore a requantization of already-quantized weights, not a quantization of BF16. That is a different and harsher operation than a normal Q2_K build, and its quality relative to the MXFP4 parent has not been measured.

Blackfrost's own quant ladder for K3 starts at IQ4_XS for exactly this reason. This Q2_K exists to prove single-node GGUF viability, not as a recommended quality point.

Full-precision and higher-quality builds

This Q2_K is the openly-distributed variant. As set out above, it is a requantization of already-4-bit-QAT experts and is not the quality point Blackfrost recommends for production work.

Full-precision and higher-quality builds of this checkpoint are available for purchase under a separate licence agreement, including:

Build	What it is
Lossless MXFP4	Routed experts bit-exact from the parent packs — no requantization at all
Coding-calibrated REAP-320	320 of 896 experts, fits a single 8× RTX 6000 node
Custom	Other expert budgets, non-uniform keep sets, or calibration against your own workload and threat model

Access to these builds is granted on purchase. Their repositories are gated, and approval follows a completed licence agreement — the gate is the transaction. Requesting access to a premium repository without a licence in place will not be approved.

Purchase link coming soon. Until then, @Blackfrost_AI DMs are the fastest route to a human.

Lineage


Base	Official `moonshotai/Kimi-K3`
Applied	Refusal-direction de-risk at the weight level (in parent) · Q2_K requantization
Not applied	Expert pruning · additional SFT/DPO
Format	GGUF · llama.cpp PR #26185 KV contract

On refusal behaviour: this checkpoint inherits the parent's deliberately reduced refusal surface. It is a Blackfrost de-risked model. Do not evaluate or rate-limit it as if it were a safety-stock derivative of upstream Kimi K3.

Measured behaviour

Verified on 8× NVIDIA B200, llama.cpp PR #26185 branch @ cf67f0d, llama-server:


Load	healthy in ~120 s, ~956 GiB across 8 GPUs at `-c 8192`
Decode	~16.3 tok/s single-stream
Termination	5/5 clean at `temperature 1.0 / top_p 0.95`

Quality — not yet measured

Benchmark	Parent (MXFP4)	This (Q2_K)	Retention
pending	—	—	—%

Requantizing already-QAT experts is a capability trade, and a card that ships a Q2_K of a 4-bit-QAT parent without publishing what it cost is asking the reader to take the trade on faith. Harness, conditions and retention figures will be stated here — including any benchmark where retention is poor.

⚠️ Sampling — read before filing a bug

Do not use greedy decoding. At temperature 0 / top_k 1 the model fails to terminate: it emits its answer, re-opens a response segment and restates it on an exact 40-token cycle, never producing an end-of-generation token.

Follow Moonshot's published guidance:

{ "temperature": 1.0, "top_p": 0.95 }

Measured on this file, five seeds each, identical prompt:

sampler	clean terminations
`temperature 1.0, top_p 0.95`	5 / 5 — Moonshot's spec, recommended
`temperature 0.6, top_p 0.95`	4 / 5
`temperature 0, top_k 1`	0 / 5 — loops indefinitely
`temperature 0, top_k 1, repeat_penalty 1.1`	stops — use if you need determinism

This is greedy-decoding degeneration, not a defect in the quantization. Moonshot's card specifies temperature = 1.0; generation_config.json sets eos_token_id: 163586 (<|end_of_msg|>). Greedy was never a supported operating point for K3.

Deployment notes

Build the PR branch — mainline will not work:

git clone https://github.com/pwilkin/llama.cpp.git && cd llama.cpp
git fetch origin kimi-k3-text && git checkout cf67f0d24511864d2d3da0769108fd6fc16d00d1
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=100 \
      -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF
cmake --build build --target llama-server -j

./build/bin/llama-server \
  -m KIMI-K3-MXP4-DERISKED-Q2_K-00001-of-00038.gguf \
  -ngl 99 -nr -c 8192 --jinja --host 0.0.0.0 --port 8080

-nr / --no-repack is strongly recommended. Repack is on by default, runs single-threaded (100% of one core, 0% GPU) and allocates a second full copy of the weights.
Point -m at shard 00001 — llama.cpp resolves the remaining 37 automatically.
max_tokens. Thinking is always on; the answer lands in content, the chain of thought in reasoning_content. Budget generously or content comes back empty with finish_reason: length.
Integrity. Verify shard count and byte totals after download before attributing a load failure to the weights.

Benign warning at load

load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect

Ignore it. llama-vocab.cpp emits the warning and inserts the token on the next line — it is self-healing. This file's vocab metadata is correct: [BOS] 163584 · [EOS] 163585 · <|end_of_msg|> 163586 · [EOT] 163593. Structural markers are <|open|> 163587 · <|close|> 163588 · <|sep|> 163589.

File naming

Shards carry the legacy KIMI-K3-MXP4-DERISKED-Q2_K-* prefix, which contains a historic MXP4 typo. Filenames are deliberately not renamed so existing download paths and scripts keep working.

⚠️ Security: structural markers in untrusted input

K3 builds prompts in Python via encoding_k3.py::build_chat_segments, where each segment carries its own allow_special flag (default False). A Jinja template cannot express that, and llama.cpp tokenizes the rendered prompt in a single pass with parse_special=true.

Consequence: a literal <|end_of_msg|> in user content becomes control token 163586 in llama.cpp, where Moonshot's own tokenizer keeps it as ordinary text. Untrusted input can forge chat structure.

This is a general llama.cpp property, not specific to K3 or this quant. If you feed untrusted text to this model, strip or escape these four markers first: <|open|> · <|sep|> · <|close|> · <|end_of_msg|>

Disclaimer

Refusal behaviour in this checkpoint has been deliberately modified at the weight level. It is not a safety-stock model and must not be deployed, marketed, or evaluated as one.

No warranty of any kind. Provided "as is", without warranty express or implied, including fitness for a particular purpose. Nothing here guarantees that any given input will be accepted or refused, that any capability is retained, or that any category of output is unreachable.

Measurements describe what was measured. Throughput, termination and retention figures reflect specific harnesses under specific conditions. They are not safety proofs and do not generalise to multimodal, tool-use, long-context or multi-turn adversarial settings.

Modification by a recipient voids this characterization. Blackfrost's obligations attach at the point of release. Any further ablation, fine-tuning, merging, quantization or alteration by a recipient produces an artifact Blackfrost has not evaluated and does not stand behind — responsibility for that artifact transfers entirely to whoever produced it.

Operator-owned policy. Open weights mean the operator sets and enforces policy. Deploy only in controlled environments with access control, independent logging and review. Do not deploy where refusal behaviour equivalent to upstream Kimi K3 is assumed or required.

Access & licensing

This repository is free and ungated — no request, no review, no purchase. It is published openly for the community to evaluate. The full-precision builds listed above are the paid tier; those repositories are gated and their gates open on a completed licence agreement.

Base licence: Kimi K3 — Moonshot AI's terms apply to this derivative and travel with it.
Redistribution: do not redistribute weights outside your grant.
Evaluation recommendation: should not be evaluated by processes that assume refusal behaviour equivalent to the parent.
Ask us about other quant points, or calibration against your own threat model.