---
quantized_by: ubergarm
pipeline_tag: text-generation
base_model: Qwen/Qwen3.5-397B-A17B
base_model_relation: quantized
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.5-397B-A17B/blob/main/LICENSE
tags:
- imatrix
- conversational
- qwen3_5_moe
- ik_llama.cpp
---

## WIP

There is not yet support in [ik_llama.cpp though an open issue](https://github.com/ikawrakow/ik_llama.cpp/issues/1229).

For now to help out with testing, used mainline llama.cpp to make imatrix (gguf format) if others would like to use it to make their own imatrix custom quants.

Check the `logs/` directory for details on imatrix calculation.

I'll upload more if/when ik_llama.cpp support is merged.

It seems to inference very slowly on CPU-only and probably requires at least one GPU to handle attention/kv-cache/delta-net stuff as it is much faster even hybrid CPU+GPU.

## Q3_K 179.97 GiB (3.90 BPW)
TODO Perplexity Calculations

<details>

<summary>👈 Secret Recipe</summary>

```bash
./build/bin/llama-quantize \
    --tensor-type ffn_down_exps=q4_K \
    --tensor-type ffn_gate_exps=q3_K \
    --tensor-type ffn_up_exps=q3_K \
    --token-embedding-type q4_K \
    --output-tensor-type q6_K \
    --imatrix /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/imatrix-Qwen3.5-397B-A17B-BF16-mainline.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-BF16-00001-of-00017.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-Q3_K.gguf \
    Q8_0 \
    128
```

</details>

## References
* [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp)
* [ubergarm on quantizing LLMs and tuning GPUs with aifoundry.org](https://blog.aifoundry.org/p/adventures-in-model-quantization)
* [ubergarm-imatrix-calibration-corpus-v02.txt](https://gist.github.com/ubergarm/edfeb3ff9c6ec8b49e88cdf627b0711a?permalink_comment_id=5682584#gistcomment-5682584)
* [Getting Started Guide (out of date)](https://github.com/ikawrakow/ik_llama.cpp/discussions/258)
* [Quant Cookers Guide (out of date)](https://github.com/ikawrakow/ik_llama.cpp/discussions/434)
* [ik_llama.cpp Qwen3Next Issue](https://github.com/ikawrakow/ik_llama.cpp/issues/1229)