Ternary Bonsai 2 27B vs Qwen3.8-27B quants: benchmark summary

Hardware: Tesla PG500-216 (V100-class, Volta, sm_70, 32 GB), Dell PowerEdge, Unraid
Runtime: PrismML llama.cpp fork, build bdc23b56b (10728), CUDA 12.8.1, CUDA_DOCKER_ARCH=70
Models tested:

• Qwen3.8-27B-BF16 (Unsloth): reference
• Qwen3.8-27B-Q8_0 (Unsloth)
• Qwen3.8-27B-UD-Q4_K_XL (Unsloth)
• Ternary-Bonsai-2-27B-PQ2_0 (PrismML): 2.13 bpw, group-128 ternary
• Ternary-Bonsai-2-27B-PTQ1_0 (PrismML): 1.75 bpw dense ternary (speed only)

Summary

Q8_0 is effectively identical to BF16, and Q4_K_XL is an excellent quant (within about 1% on every measure). Bonsai PQ2_0 is about 38% faster at generation and uses 10 GB less VRAM than Q4_K_XL, but it diverges from the original model far more than normal quantization does: roughly 40× the Q4's KL divergence, and it picks a different top token about 1 in 4.5 times.

Bonsai is a retrained ternary model, not a rounded copy of Qwen3.8-27B, so these numbers show it behaves differently from the original, not necessarily worse on real tasks. Chat-template task evals (gsm8k, mmlu_pro) are the remaining test.

Current pick: Q4_K_XL as the default for fidelity; Bonsai PQ2_0 only where speed or VRAM matters more than matching Qwen's behavior. Avoid PTQ1_0 on Volta.

Speed (llama-bench)

-ngl 99 -fa 1 -p 512 -n 128 -d 0,8192 -r 5

Notes:

• Prompt processing is roughly tied across all three; decode speed is the only real difference.
• Bonsai PQ2_0 is 2.4× smaller than Q4_K_XL but only 1.38× faster. Effective decode bandwidth is about 335 GB/s vs about 590 GB/s for Q4_K, so the ternary kernels are unpacking-bound on Volta rather than memory-bound.
• PTQ1_0's denser trit packing costs more ALU per weight, making it slower than both others on this card. Its only advantage is 1.2 GB less VRAM.
• Both models slow down very little at 8K context (hybrid attention).

Perplexity (wikitext-2 test, full 145 chunks, -c 2048)

Bonsai is about 32% higher. All models share the same tokenizer (gpt2 / qwen35, 248,320 tokens, BOS 248044, EOS 248046), so these are directly comparable.

KL divergence vs BF16 (wikitext-2, 50 chunks, -c 2048)

Reference logits from BF16 with partial offload (-ngl 34, about 26.9 GB VRAM, 51.5 s/chunk, about 43 min). BF16 PPL over these 50 chunks: 6.1345 ± 0.0654.

Interpretation

• Q8_0: Indistinguishable from BF16 for practical purposes (mean KLD 0.0006, 99.3% top-token agreement).
• Q4_K_XL: A strong quant. Its slightly lower PPL than BF16 is a common artifact, not an improvement; KLD shows small real drift. Greedy output differs from BF16 about once every 32 tokens.
• Bonsai PQ2_0: Divergence is broad, not limited to a few outlier tokens: its median KLD (0.18) exceeds the Q4's 99th percentile (0.066). It disagrees with BF16's top pick about once every 4.5 tokens. Its negative mean Δp shows it is systematically less confident in the correct next token. The lower per-chunk PPL correlation (92%) means it finds different passages hard, consistent with a retrained sibling model rather than a lossy copy.
• Ratios: Bonsai mean KLD is about 39× Q4_K_XL and about 530× Q8_0.

Caveats

• Perplexity and KLD score raw Wikipedia text without a chat template or reasoning. They measure closeness to Qwen3.8-27B's distribution, not task ability.
• PrismML's "~98% of benchmark performance" claim is based on task benchmarks in thinking mode, which this suite doesn't test.
• Bonsai's kernels are not tuned or benchmarked by PrismML on Volta; speed results may not reflect newer GPUs.
• The BF16 base run disabled fused Gated Delta Net ops because of the CPU/GPU split. This affects speed only, not results.

Next steps

• Run task evals through llama-server with the chat template, e.g. lm_eval --model local-chat-completions --tasks gsm8k,mmlu_pro --apply_chat_template --limit 200, with matching sampling and reasoning effort for Q4_K_XL and Bonsai PQ2_0.
• Optionally, HellaSwag / Winogrande via llama-perplexity --hellaswag / --winogrande for quick accuracy-based checks.

Commands used

# prism helper
prism() { docker run --rm --init --gpus '"device=1"' -v /mnt/user/appdata/llama_cpp/model:/models local/llama.cpp-prism:full "$@"; }

# speed
prism --bench -m <model> -ngl 99 -fa 1 -p 512 -n 128 -d 0,8192 -r 5

# perplexity
prism --perplexity -m <model> -f /models/wiki.test.raw -ngl 99 -c 2048

# KLD base (BF16, partial offload)
prism --perplexity -m <bf16-00001-of-00002.gguf> -f /models/wiki.test.raw -ngl 34 -c 2048 --chunks 50 \
  --kl-divergence-base /models/qwen-bf16.kld

# KLD compare
prism --perplexity -m <model> -f /models/wiki.test.raw -ngl 99 -c 2048 --chunks 50 \
  --kl-divergence-base /models/qwen-bf16.kld --kl-divergence