ekrtprsnsp/typhoon2.5-qwen3-30b-a3b-GGUF

Typhoon 2.5 Qwen3 30B-A3B — GGUF (Q6_K)

GGUF quantization (Q6_K) of typhoon-ai/typhoon2.5-qwen3-30b-a3b, a Thai-focused Mixture-of-Experts language model by SCB 10X built on the Qwen3 MoE architecture (31B total parameters, ~3B active).

This repository provides a Q6_K build for running the model locally with llama.cpp. At the time of upload, no Q6_K GGUF of this model was publicly available, so this is shared to fill that gap.

Note on quantization: This is a standard Q6_K quant produced with llama-quantize. It does not use an importance matrix (imatrix). At the 6-bit level the quality difference between plain Q6_K and imatrix-based variants is small, and a generic/incorrect calibration set can do more harm than good — so plain Q6_K was chosen deliberately for reliability.

Files

File	Quant	Bits	Size	Notes
typhoon2.5-qwen3-30b-a3b-Q6_K.gguf	Q6_K	6	24 GB	No imatrix. Good balance of quality and size.

Fits comfortably on 32 GB+ unified/VRAM setups; runs well on systems like the NVIDIA DGX Spark / GB10 with 128 GB unified memory.

How to run (llama.cpp)

./llama.cpp/build/bin/llama-server \
    -m typhoon2.5-qwen3-30b-a3b-Q6_K.gguf \
    --host 0.0.0.0 --port 8000 \
    -ngl 999 -fa on \
    -c 8192 \
    --temp 0.6 --top-p 0.95 --repeat-penalty 1.05 \
    --jinja

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.

Sampling — follow the base model's official recommendation. SCB 10X recommends a low temperature and repetition_penalty = 1.05 to improve performance and reduce repetition. The example above uses temp=0.6, top_p=0.95, repeat-penalty=1.05, matching the base model card. Please refer to the original model card for the authoritative usage guidance.

Context length: the base model supports up to 256K context. Set -c to suit your task and available memory (the example uses a modest 8192; increase as needed).

Quantization details

Produced from the original safetensors using llama.cpp:

# 1) Convert HF model -> GGUF F16
python convert_hf_to_gguf.py typhoon2.5-qwen3-30b-a3b/ \
    --outfile typhoon2.5-f16.gguf --outtype f16

# 2) Quantize F16 -> Q6_K
./llama.cpp/build/bin/llama-quantize \
    typhoon2.5-f16.gguf \
    typhoon2.5-qwen3-30b-a3b-Q6_K.gguf \
    Q6_K

• Architecture: qwen3moe
• Method: standard Q6_K (no importance matrix)
• llama.cpp: built with CUDA; please use a recent build that supports the qwen3moe architecture.

Credits & license

• Base model: typhoon-ai/typhoon2.5-qwen3-30b-a3b by SCB 10X — all model capabilities and training credit belong to them.
• License: Apache 2.0 (inherited from the base model). Please review the base model's card for the full terms and any usage notes.
• This repository only redistributes a quantized format; it does not modify the model's weights beyond quantization.

If you use this model, please cite and credit the original Typhoon / SCB 10X work.