---
license: apache-2.0
base_model:
- ShyliaSafetensors/Ariel-Alloy-V1-24B-Heretic
library_name: ik_llama.cpp
pipeline_tag: text-generation
tags:
- gguf
- quantized
- ik_llama.cpp
- roleplay
- trellis
- mixed precision
quantized_by: Koshkasa
base_model_relation: quantized
---

## What's that?  
    
**The goal**: Make a quality quant of ShyliaSafetensors/Ariel-Alloy-V1-24B-Heretic using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit into 16gb VRAM with KVO, accounting for system overhead.   
  
This time the wheel was reinvented one spoke at a time - blocks in proximity to input and output are treated with premium precision, while the more tolerant middle blocks are compressed in a more aggressive 4-bit precision. Attention is kept high in all blocks.  
This is in line of how popular Q4_K_S/M quants are made for Mistral 24B, except we use IQ_K quants and utilize trellis for middle ffn_up and ffn_gate blocks.  
Three versions were cooked. Two lost marginally to Q4_K_S in hellaswag and winogrande, though perhaps they would perform better in NIHS-tests (due to higher attn), which I don't have the patience to set up. For what they offered, they were slow - they had even more trellis, and it shows.  
  
Option three, which is this one, compromised some attn precision for better protection of ffn_down layers. Lacking a conventional naming scheme for these, I went with IQ4_K_M. It looks stupid. I don't know how to fix that.

**The result**: Mixed precision quantization of [ShyliaSafetensors/Ariel-Alloy-V1-24B-Heretic](https://huggingface.co/ShyliaSafetensors/Ariel-Alloy-V1-24B-Heretic)  

quantized with ik_llama.cpp build: 9d07d868    
  
**incompatible with mainline llama.cpp**  

<details> <summary><strong>Layout: IQ4_K_M</h4></summary>
    
| Layer | Dims | Dims | Quant | 
|--|--|--|--|
| token\_embd | 5120 | 131072.0 | **iq5\_ks** |

<div style="display: flex; flex-wrap: wrap; gap: 20px; justify-content: center; ">

  
  <div style="flex: 1; min-width: 200px;">
    <h4>Blocks 0, 1, 38, 39</h4>
    <table>
      <thead>
        <tr><th>Layer</th><th>Dims</th><th>Dims</th><th>Quant</th></tr>
      </thead>
      <tbody>
        <tr><td>attn_k</td><td>5120</td><td>1024</td><td><strong>iq6_k</strong></td></tr>
        <tr><td>attn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>
        <tr><td>attn_q</td><td>5120</td><td>4096</td><td><strong>iq6_k</strong></td></tr>
        <tr><td>attn_v</td><td>5120</td><td>1024</td><td><strong>iq6_k</strong></td></tr>
        <tr><td>ffn_down</td><td>32768</td><td>5120</td><td><strong>iq6_k</strong></td></tr>
        <tr><td>ffn_gate</td><td>5120</td><td>32768</td><td><strong>iq6_k</strong></td></tr>
        <tr><td>ffn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>
        <tr><td>ffn_up</td><td>5120</td><td>32768</td><td><strong>iq6_k</strong></td></tr>
        <tr><td>attn_output</td><td>4096</td><td>5120</td><td><strong>iq6_k</strong></td></tr>
      </tbody>
    </table>
  </div>

  
  <div style="flex: 1; min-width: 200px;">
    <h4>Blocks 2, 3, 37, 34-37</h4>
    <table>
      <thead>
        <tr><th>Layer</th><th>Dims</th><th>Dims</th><th>Quant</th></tr>
      </thead>
      <tbody>
        <tr><td>attn_k</td><td>5120</td><td>1024</td><td><strong>iq6_k</strong></td></tr>
        <tr><td>attn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>
        <tr><td>attn_q</td><td>5120</td><td>4096</td><td><strong>iq5_k</strong></td></tr>
        <tr><td>attn_v</td><td>5120</td><td>1024</td><td><strong>iq6_k</strong></td></tr>
        <tr><td>ffn_down</td><td>32768</td><td>5120</td><td><strong>iq5_k</strong></td></tr>
        <tr><td>ffn_gate</td><td>5120</td><td>32768</td><td><strong>iq5_ks</strong></td></tr>
        <tr><td>ffn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>
        <tr><td>ffn_up</td><td>5120</td><td>32768</td><td><strong>iq5_ks</strong></td></tr>
        <tr><td>attn_output</td><td>4096</td><td>5120</td><td><strong>iq5_k</strong></td></tr>
      </tbody>
    </table>
  </div>

  
  <div style="flex: 1; min-width: 200px;">
    <h4>Blocks 6–33</h4>
    <table>
      <thead>
        <tr><th>Layer</th><th>Dims</th><th>Dims</th><th>Quant</th></tr>
      </thead>
      <tbody>
        <tr><td>attn_k</td><td>5120</td><td>1024</td><td><strong>iq5_k</strong></td></tr>
        <tr><td>attn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>
        <tr><td>attn_q</td><td>5120</td><td>4096</td><td><strong>iq5_ks</strong></td></tr>
        <tr><td>attn_v</td><td>5120</td><td>1024</td><td><strong>iq6_k</strong></td></tr>
        <tr><td>ffn_down</td><td>32768</td><td>5120</td><td><strong>iq4_k</strong></td></tr>
        <tr><td>ffn_gate</td><td>5120</td><td>32768</td><td><strong>iq4_kt</strong></td></tr>
        <tr><td>ffn_norm</td><td>5120</td><td>1</td><td><strong>f32</strong></td></tr>
        <tr><td>ffn_up</td><td>5120</td><td>32768</td><td><strong>iq4_kt</strong></td></tr>
        <tr><td>attn_output</td><td>4096</td><td>5120</td><td><strong>iq5_k</strong></td></tr>
      </tbody>
    </table>
  </div>


| Layer | Dims | Dims | Quant | 
|--|--|--|--|
| output | 5120 | 131072 | **iq6\_k** | 
| output\_norm | 5120 | 1 | **f32** |  

  </div>
</details>

  
using [imatrix by mradermacher](https://huggingface.co/mradermacher/Ariel-Alloy-V1-24B-Heretic-i1-GGUF)

## Rationale  
Hopefully the beefed up attention will help over contexts this quant is intended to run (16k-32k). It is not statistically dumber than its main competitor, Q4_K_S, outsmarted by 3-4 responses over 1267 winogrande and 800 hellaswag questions (s = 123). While the difference is not statistically significant, it's there. A NIHS test would probably be this quant's stronger suit, but I lack quality data to test it. Experimental quant. WYSIWYG.      
  
## Cheers  
**[MistralAI](https://huggingface.co/mistralai)** - the beloved base model(s).  
**[ikawrakow and contributors of ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp)** - I probably misused your wonderful creation.   
**[ShyliaSafetensors](https://huggingface.co/ShyliaSafetensors/)** - for the merge effort.    
**Everyone whose finetunes were included in the merge!**  
**[mradermacher](https://huggingface.co/mradermacher)** - for the imatrix + the myriad of quants we all benefit from.