Koshkasa/ShyliaSafetensors_Ariel-Alloy-V1-24B-Heretic-IQ4_K_M.GGUF

What's that?

The goal: Make a quality quant of ShyliaSafetensors/Ariel-Alloy-V1-24B-Heretic using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit into 16gb VRAM with KVO, accounting for system overhead.

This time the wheel was reinvented one spoke at a time - blocks in proximity to input and output are treated with premium precision, while the more tolerant middle blocks are compressed in a more aggressive 4-bit precision. Attention is kept high in all blocks.
This is in line of how popular Q4_K_S/M quants are made for Mistral 24B, except we use IQ_K quants and utilize trellis for middle ffn_up and ffn_gate blocks.
Three versions were cooked. Two lost marginally to Q4_K_S in hellaswag and winogrande, though perhaps they would perform better in NIHS-tests (due to higher attn), which I don't have the patience to set up. For what they offered, they were slow - they had even more trellis, and it shows.

Option three, which is this one, compromised some attn precision for better protection of ffn_down layers. Lacking a conventional naming scheme for these, I went with IQ4_K_M. It looks stupid. I don't know how to fix that.

The result: Mixed precision quantization of ShyliaSafetensors/Ariel-Alloy-V1-24B-Heretic

quantized with ik_llama.cpp build: 9d07d868

incompatible with mainline llama.cpp

Layout: IQ4_K_M

Layer	Dims	Dims	Quant
token_embd	5120	131072.0	iq5_ks

Blocks 0, 1, 38, 39

Layer	Dims	Dims	Quant
attn_k	5120	1024	iq6_k
attn_norm	5120	1	f32
attn_q	5120	4096	iq6_k
attn_v	5120	1024	iq6_k
ffn_down	32768	5120	iq6_k
ffn_gate	5120	32768	iq6_k
ffn_norm	5120	1	f32
ffn_up	5120	32768	iq6_k
attn_output	4096	5120	iq6_k

Blocks 2, 3, 37, 34-37

Layer	Dims	Dims	Quant
attn_k	5120	1024	iq6_k
attn_norm	5120	1	f32
attn_q	5120	4096	iq5_k
attn_v	5120	1024	iq6_k
ffn_down	32768	5120	iq5_k
ffn_gate	5120	32768	iq5_ks
ffn_norm	5120	1	f32
ffn_up	5120	32768	iq5_ks
attn_output	4096	5120	iq5_k

Blocks 6–33

Layer	Dims	Dims	Quant
attn_k	5120	1024	iq5_k
attn_norm	5120	1	f32
attn_q	5120	4096	iq5_ks
attn_v	5120	1024	iq6_k
ffn_down	32768	5120	iq4_k
ffn_gate	5120	32768	iq4_kt
ffn_norm	5120	1	f32
ffn_up	5120	32768	iq4_kt
attn_output	4096	5120	iq5_k

Layer	Dims	Dims	Quant
output	5120	131072	iq6_k
output_norm	5120	1	f32

using imatrix by mradermacher

Rationale

Hopefully the beefed up attention will help over contexts this quant is intended to run (16k-32k). It is not statistically dumber than its main competitor, Q4_K_S, outsmarted by 3-4 responses over 1267 winogrande and 800 hellaswag questions (s = 123). While the difference is not statistically significant, it's there. A NIHS test would probably be this quant's stronger suit, but I lack quality data to test it. Experimental quant. WYSIWYG.

Cheers

MistralAI - the beloved base model(s).
ikawrakow and contributors of ik_llama.cpp - I probably misused your wonderful creation.
ShyliaSafetensors - for the merge effort.
Everyone whose finetunes were included in the merge!
mradermacher - for the imatrix + the myriad of quants we all benefit from.