---
license: apache-2.0
pipeline_tag: image-text-to-text
base_model:
- Qwen/Qwen3.6-35B-A3B
---

# Qwen3.6-35B-A3B IQ4_KS GGUF

[ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) is required to run this model.

`ik_llama.cpp` imatrix quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B). 

This quant uses a mixed-precision recipe to maximize quality while fitting entirely into 24GB VRAM for full GPU offloading. Attention, SSM, and shared expert layers are kept at `Q8_0`/`F32`, while the massive routed MoE expert layers are quantized to `IQ5_KS` and `IQ4_KS`.

## Perplexity

Measured against `wiki.test.raw` with `n_ctx=512`:

| Quant | Size | PPL |
| :--- | :--- | :--- |
| **IQ4_KS** | ~19.8 GiB | **6.7401 +/- 0.04381** |

*Note: Lower is better.

## Quantization Recipe

The "Secret Recipe" used for the mixed-precision tensor overrides is based on the methodology used by [ubergarm](https://huggingface.co/ubergarm/Qwen3.5-35B-A3B-GGUF):

```bash
custom="
# 60 Repeating Layers [0-59]
## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=q8_0
blk\..*\.attn_qkv\.weight=q8_0
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0
blk\..*\.ssm_alpha\.weight=f32
blk\..*\.ssm_beta\.weight=f32
blk\..*\.ssm_out\.weight=q8_0
# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=iq5_ks
blk\..*\.ffn_(gate|up)_exps\.weight=iq4_ks
# Non-Repeating Layers
token_embd\.weight=q8_0
output\.weight=q8_0
"

custom=$(
  echo "$custom" | grep -v '^#' | \
  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)
```

## How This Was Made (Reproduction Steps)

Because generating an imatrix directly from a ~65GB BF16 model requires massive system RAM, we use a Q8_0 intermediate step to generate the imatrix without running out of memory.

**1. Convert HF Safetensors to BF16 GGUF**
```bash
python llama.cpp/convert_hf_to_gguf.py \
  --outtype bf16 \
  --split-max-size 50G \
  --outfile ./Qwen3.6-35B-A3B-BF16.gguf \
  /path/to/Qwen3.6-35B-A3B/
```

**2. Quantize to Q8_0 (For Imatrix Generation)**
```bash
./ik_llama.cpp/build/bin/llama-quantize \
  ./Qwen3.6-35B-A3B-BF16-00001-of-00002.gguf \
  ./Qwen3.6-35B-A3B-Q8_0.gguf \
  Q8_0 16
```

**3. Generate the Imatrix**
*Note: `GGML_CUDA_NO_PINNED=1` is used to prevent system RAM exhaustion on 24GB VRAM setups.*
```bash
GGML_CUDA_NO_PINNED=1 ./ik_llama.cpp/build/bin/llama-imatrix \
  -m ./Qwen3.6-35B-A3B-Q8_0.gguf \
  -f /path/to/ubergarm-imatrix-calibration-corpus-v02.txt \
  -o Qwen3.6-35B-A3B-imatrix.dat \
  --ctx-size 512 \
  -t 11 \
  --fit
```

**4. Quantize BF16 to IQ4_KS**
*We use the original BF16 model here with the Q8_0-generated imatrix for maximum fidelity.*
```bash
./ik_llama.cpp/build/bin/llama-quantize \
  --imatrix ./Qwen3.6-35B-A3B-imatrix.dat \
  --custom-q "$custom" \
  ./Qwen3.6-35B-A3B-BF16-00001-of-00002.gguf \
  ./Qwen3.6-35B-A3B-IQ4_KS.gguf \
  IQ4_KS 16
```

**5. Test Perplexity**
```bash
wget https://huggingface.co/datasets/ikawrakow/validation-datasets-for-llama.cpp/resolve/main/wiki.test.raw.gz
gunzip wiki.test.raw.gz

./ik_llama.cpp/build/bin/llama-perplexity \
  -m ./Qwen3.6-35B-A3B-IQ4_KS.gguf \
  -f ./wiki.test.raw \
  -c 512 \
  -ngl 99 \
  -t 1 \
  -fa
```

## Quick Start Inference

Requires [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp).

```bash
./ik_llama.cpp/build/bin/llama-server \
  -m ./Qwen3.6-35B-A3B-IQ4_KS.gguf \
  -c 131072 \
  -ngl 99
```