P4pps3n/KAT-Coder-V2.5-Dev-MTP-exl3-5bpw-hq

KAT-Coder-V2.5-Dev MTP EXL3 5.08 bpw HQ

This is a community EXL3 quantization of Kwaipilot/KAT-Coder-V2.5-Dev for ExLlamaV3.

The main decoder was quantized from the original BF16 model. The MTP component was sourced from SpectreSystems/KAT-Coder-V2.5-Dev-MTP, which contains the corresponding BF16 MTP tensors. Existing MTP tensors are preserved and quantized at approximately 8 bpw. This conversion does not synthesize a new MTP head.

This is not an official Kwaipilot release. It is an independently converted community quantization.

Default sampling

The published generation_config.json now defaults to deterministic coding behavior:

{
  "temperature": 0.0,
  "top_k": 20,
  "top_p": 0.95,
  "do_sample": true
}

Clients may override these values per request. For debugging path handling, tool calls, or malformed output, start with temperature: 0 and enable_thinking: false.

Parameter count note

The upstream KAT-Coder-V2.5-Dev model is a 35B total parameter MoE with approximately 3B activated parameters per token. The Hugging Face file viewer may display a lower number such as 12B for this repository because its generic parameter counter does not understand EXL3 packed tensors and counts their storage representation incorrectly. That generated UI value is not the model's logical parameter count and does not indicate missing weights.

Size and quantization

• Total local package size: 24.046 GB (22.395 GiB)
• EXL3 safetensors size: 23.966 GB (22.320 GiB)
• Quantizer: ExLlamaV3
• Quantizer revision: 0b9745c526a13d5b30f1b58a864efc1932d3d9eb
• Method: EXL3 mul1 codebook with HQ allocation
• Target bitrate: 5.0 bpw
• Measured main decoder bitrate: 5.08 bpw
• Output head: 8 bpw
• MTP tensors: approximately 8.04 bpw
• MTP hidden layers: 1
• Calibration: 250 rows, 2048 columns

Context and memory requirements

• Safe tested context: 258,048 tokens
• Q8 KV cache at the full tested context: approximately 10 GiB (10.737 GB) of KV memory, before other runtime workspaces
• Recommended total GPU memory: two 24 GiB cards
• Recommended cache: Q8
• Recommended batch size: 1

The context value above is the validated operating limit for the documented TabbyAPI setup. Lower context limits are recommended on smaller or more imbalanced GPU configurations.

Quick setup with TabbyAPI and ExLlamaV3

The following is a minimal agent-friendly setup for a fresh Linux machine. Adapt the CUDA device IDs if the two 24 GiB inference GPUs use different IDs.

1. Download the model

hf download P4pps3n/KAT-Coder-V2.5-Dev-MTP-exl3-5bpw-hq \
  --local-dir /models/KAT-Coder-V2.5-Dev-MTP-exl3-5bpw-hq

The repository is public, so no token is required for download.

2. Install or enter the serving environment

Use a Python environment containing TabbyAPI, ExLlamaV3, PyTorch, and the matching CUDA runtime. For an existing TabbyAPI checkout:

cd /path/to/tabbyAPI
source /path/to/venv/bin/activate

Install the serving stack according to the TabbyAPI and ExLlamaV3 instructions for the target CUDA/PyTorch combination. Do not mix an unrelated system Python with the environment containing TabbyAPI dependencies.

3. Create model-local TabbyAPI overrides

Save this as tabby_config.yml in the model directory, or pass equivalent settings through the TabbyAPI configuration:

model:
  backend: exllamav3
  max_seq_len: 258048
  cache_size: 258048
  cache_mode: Q8
  max_batch_size: 1
  chunk_size: 2048
  output_chunking: true
  draft_mode: mtp
  prompt_template: chat_template
  tool_format: qwen3_coder

4. Start TabbyAPI

Expose only the two inference GPUs and bind locally while testing:

CUDA_VISIBLE_DEVICES=1,2 \
python main.py \
  --model-dir /models \
  --model-name KAT-Coder-V2.5-Dev-MTP-exl3-5bpw-hq \
  --host 127.0.0.1 \
  --port 5000

The OpenAI-compatible endpoint is then:

http://127.0.0.1:5000/v1

Do not expose an unauthenticated TabbyAPI instance to an untrusted network.

5. Test the endpoint

curl http://127.0.0.1:5000/v1/models

Example thinking-disabled request:

curl http://127.0.0.1:5000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"KAT-Coder-V2.5-Dev-MTP-exl3-5bpw-hq","messages":[{"role":"user","content":"Reply with exactly OK."}],"enable_thinking":false,"temperature":0,"max_tokens":32}'

Thinking modes and chat template

The model uses its native chat_template. Thinking is selected per request.

Thinking disabled:

{"enable_thinking": false}

Thinking enabled:

{"enable_thinking": true}

With thinking disabled, the response is clean assistant content without visible thinking markers. With thinking enabled, the raw response contains reasoning followed by </think> and the final answer. Clients should either render the reasoning separately or remove <think>...</think> before displaying only the final answer.

For tool-calling clients, preserve the Qwen3 coder tool format and the native chat template:

tool_format: qwen3_coder
prompt_template: chat_template

Hermes Agent setup

For Hermes Agent, create two local OpenAI-compatible provider entries:

kat_5bpw:
  api: http://127.0.0.1:5000/v1
  transport: openai_chat
  discover_models: false
  models: KAT-Coder-V2.5-Dev-MTP-exl3-5bpw-hq
  extra_body:
    enable_thinking: false

kat_5bpw_think:
  api: http://127.0.0.1:5000/v1
  transport: openai_chat
  discover_models: false
  models: KAT-Coder-V2.5-Dev-MTP-exl3-5bpw-hq
  extra_body:
    enable_thinking: true

Useful model aliases are:

kat-coder-5bit
kat-coder-5bit-think

Expected performance

These measurements were made with two RTX 3090 cards, Q8 KV cache, MTP draft mode, one active sequence, and TabbyAPI/ExLlamaV3. Actual speed depends on prompt length, generated length, cache state, GPU split, sampling settings, and whether thinking is enabled.

Long-context performance

Content tokens	End-to-end	Prefill	Generation	Draft acceptance
32,256	0.15 s	2,848 t/s	135.91 t/s	2 / 2
65,024	1.40 s	1,623.62 t/s	111.95 t/s	2 / 2
130,560	48.86 s	1,347.66 t/s	82.29 t/s	2 / 2
196,096	71.30 s	923.37 t/s	64.91 t/s	2 / 2
258,048	88.25 s	706.0 t/s	26.86 t/s	1 / 5

For ordinary short requests, expect roughly 80-136 generated tokens/second in this setup when thinking is disabled. Long-context generation becomes slower as KV memory grows; near the full tested context, approximately 27 tokens/second was measured.

Thinking-mode microbenchmark:

• Thinking disabled: 3/3 checks passed, average 0.115 seconds
• Thinking enabled: 3/3 checks passed, average 0.172 seconds
• Tool calls: valid qwen3_coder calls in both modes
• Replacement characters: 0 in tested responses

These are local independent measurements, not upstream benchmark scores. No perplexity or public benchmark score is claimed for this quantization.

Attribution and license

This is a quantized derivative and is not an official Kwaipilot release. Preserve the upstream model attribution and Apache-2.0 license declaration.

• Upstream model: https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev
• MTP source: https://huggingface.co/SpectreSystems/KAT-Coder-V2.5-Dev-MTP
• ExLlamaV3: https://github.com/turboderp-org/exllamav3
• License: Apache-2.0

Included files

The repository contains the EXL3 safetensor shards and index, tokenizer, chat template, model configuration, quantization configuration, MTP tensors, TabbyAPI overrides, conversion provenance, license attribution, benchmark scripts, raw benchmark results, and quality-check artifacts.