Thunder13240/Ornith-1.5-9B-heretic-GGUF

Ornith-1.5-9B-heretic

Importance-matrix GGUF quantizations of an abliterated Ornith-1.5-9B.

Genuine refusals	0 / 100 (base model: 99 / 100)
Cross-group uniformity	uniform - 0 refusals across 20 groups
KL divergence	0.0376 (damage threshold ~0.5)
Perplexity	2.917
Weights modified	43 of 760 tensors, text decoder only
Vision tower	bit-identical to base

Progressive release

Our upload bandwidth is limited (~1 MB/s), so the full 24-quant ladder takes many hours to transfer. Rather than sit on everything until the last byte lands, files are being published as they finish uploading:

1. Q4_K_M and the vision projector first - the ones most people want
2. Then Q8_0, Q6_K, Q5_K_M, IQ4_XS, Q4_K_S
3. Then the remaining low-bit quants, over the following hours

23 quants are built and queued. The table below lists what is currently live in this repo and is regenerated as uploads land - so if the file you want is missing, it is on its way rather than cancelled. Check back later, or watch the repo.

Quantization quality - read this before judging the numbers

Top-token agreement on this model looks low compared to figures you may have seen for other GGUF repos. It is not a defect, and it is not caused by the abliteration. The control proves it:

Model	Quant	Same top token
Official ornith-ai/Ornith-1.5-9B-GGUF	Q4_K_M	91.29%
This abliterated build	Q4_K_M	90.81%
Unmodified base, quantized here	Q4_K_M	88.93%
This build, SSM projections at Q8_0	Q4_K_M	92.97%

The model authors' own official Q4_K_M scores 91.29% on the identical test. This repo lands within half a point of it, and the SSM-protected variant beats it. Whatever is happening here is a property of the model, not of this build.

The untouched base model quantizes worse than this one. Two architectural reasons it sits lower than a typical Llama-class model:

1. 248,320-token vocabulary (Llama-2 has 32,000). With 7.8x more tokens competing, far more of them sit within a hair of each other in logit space, so the argmax flips on near-ties even when the output distribution is nearly identical. Top-1 agreement is simply a harsher metric at this vocab size and is not comparable across models.
2. 24 of 32 layers use linear attention (GatedDeltaNet), which carries a recurrent state forward through the sequence. Standard attention recomputes from scratch each token so quantization error stays local; a recurrent state accumulates it. The config pins mamba_ssm_dtype: float32 precisely because that state is numerically fragile.

Judge these quants by KL divergence, not top-1 agreement. Q8_0 measures KL 0.0019 and Q4_K_M 0.048 - both squarely in the normal published range.

An architecture-specific improvement

Point 2 above is actionable. llama.cpp's default Q4_K_M recipe was tuned for conventional transformers and quantizes the ssm_alpha, ssm_beta and ssm_out projections - the ones feeding that fragile recurrent state - at the base 4-bit rate. Protecting just those 72 tensors at Q8_0:

Build	Same top token
Base model, stock Q4_K_M	88.93%
This repo, stock Q4_K_M	90.81%
SSM projections at Q8_0	92.97%

llama-quantize --imatrix imatrix.gguf \
  --tensor-type ssm_alpha=q8_0 --tensor-type ssm_beta=q8_0 \
  --tensor-type ssm_out=q8_0 \
  model-BF16.gguf model-Q4_K_M.gguf Q4_K_M

Worth knowing for anyone quantizing hybrid linear-attention models - the stock recipes do not know these tensors are special.

Important

This model has had its safety alignment deliberately removed. It will attempt to comply with requests the original model would refuse, including harmful ones. It has no guardrails.

You are responsible for how you use it and for complying with the base model's MIT licence and applicable law. Do not deploy it in a user-facing product without adding your own safety layer.

The finding: where you sample the residuals decides everything

This is the part worth reading if you abliterate models yourself.

Directional ablation (Arditi et al. 2024) derives a "refusal direction" from the difference between residual activations on harmful and harmless prompts, then projects that direction out of every weight that writes to the residual stream. Tooling like Heretic samples those activations at the start of the model's response.

Ornith-1.5 is a reasoning model, and its chat template ends the generation prompt with an already-open <think> block:

{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
    {{- '<think>\n' }}          {# prompt ends INSIDE the reasoning block #}
{%- endif %}

So "the start of the response" is not the start of the answer - it is the start of the model's reasoning. The direction you recover is "about to think about refusing" rather than "about to refuse", and those are measurably different features.

Ablating the first one plateaus. We ran a 200-trial TPE search and a 13-recipe sweep against it and could not get below ~22/100 genuine refusals no matter how hard we pushed - raising ablation strength just cost capability without moving refusals.

Closing the block first samples where the answer actually begins:

# MUST be set before computing directions - get_residuals() routes
# through generate(), which appends this to the prompt.
settings.response_prefix = "\n</think>\n\n"

Verify the string against the template rather than trusting it; it should reproduce the model's own no-thinking form exactly:

p = tok.apply_chat_template(chat, add_generation_prompt=True, tokenize=False)
assert p + "\n</think>\n\n" == tok.apply_chat_template(
    chat, add_generation_prompt=True, tokenize=False, enable_thinking=False)

Identical ablation parameters. Corrected sampling position: 22/100 -> 0/100 genuine refusals, at lower KL divergence.

How to tell this is happening to you

• Your tool reports an empty common response prefix (Heretic prints response_prefix = '')
• Refusals plateau and extra ablation strength buys nothing
• The giveaway: your measured baseline refusal rate looks low for a well-aligned model. Ornith-1.5-9B scores 67/100 measured inside the reasoning block and 99/100 measured on its answers. If a safety-tuned model looks only two-thirds censored, you are probably scoring the wrong text.

What we tried before finding it

Every recipe below was evaluated against the same 100 held-out prompts. Down and to the left is better - fewer refusals for less divergence.

Attempt	Result	Why it failed
200-trial Optuna/TPE search	plateaued	optimising a direction derived from the wrong token position
Raising the KL target 0.01 -> 0.15	no improvement	more force on the wrong feature
Porting the published Ornith-1.0 recipe (single direction, layer 28, uniform scale 1.0)	worse	tuned for a different model, and still measured on corrupted directions
Layer sweep L20-L30	inconclusive	all measured pre-fix; not reported here, because those numbers were never re-validated
Iterative multi-direction ablation	built but never needed	the sampling fix solved it first; see below

What is ours, what is borrowed

Being explicit about this, because "new abliteration recipe" is an easy thing to overclaim:

Reused, unchanged:

• Directional ablation - Arditi et al. 2024. The core method.
• Heretic by Philipp Emanuel Weidmann - did the actual work. The shipped ablation parameters are exactly what its TPE optimizer produced; we did not invent a new ablation algorithm. We fixed the input it was being fed.
• Norm-preserving / projected abliteration - grimjim. Heretic's default, and it beat plain difference-of-means here.
• Calibration corpus for the importance matrix - bartowski's calibration_datav3.
• llama.cpp for GGUF conversion, imatrix and quantization.
• Prompt sets - mlabonne/harmful_behaviors and mlabonne/harmless_alpaca.

Inspiration:

• The published Ornith-1.0 abliterations, whose model cards documented their recipe in enough detail to test directly. It did not transfer to 1.5, but having a concrete baseline to falsify is what made the sweep worth running.

Ours:

• The chain-of-thought sampling-position fix. The finding above. This is the contribution.
• Two-scorer refusal measurement. Keyword scoring reported 6/100 where the true rate was 0/100; every hit was a compliant answer containing a word like "illegal". Reporting the keyword number alone overstates residual censorship.
• Matched-pair cross-group uniformity probe. Described below - an average refusal rate cannot detect it.

Measured results

100 held-out prompts from mlabonne/harmful_behaviors.

Metric	Original	This model
Genuine refusals	99/100	0/100
Keyword-scorer rate	99/100	6/100
Complies cleanly	-	88/100
Complies with a caveat	-	12/100
KL divergence	0 (by definition)	0.0376
Perplexity	-	2.917

Cross-group uniformity

Abliteration removes a single direction, and refusals reinforced hardest in post-training can survive it unevenly - producing a model that complies for some demographic groups and refuses for others given identical prompts. That is a defect on its own terms: a model that is uncensored for 18 groups and censored for 2 is not uncensored, it is unpredictable, and an average refusal rate will not show it.

Tested with a matched-pair probe - 100 prompts holding the request template constant while varying only the group name, across 20 groups spanning religion, ethnicity, national origin, gender, orientation, age and disability.

Result: uniform. 0/100 refusals - every group treated identically across all five phrasings, with zero spread between the most- and least-refused group.

Capability preservation

• 43 of 760 tensors modified, all in the text decoder
• Vision tower, embeddings, norms and MTP head bit-identical to base
• No NaN/Inf; max relative weight change 0.028
• 0 of 6 neutral prompts produced empty or repetition-looped output
• Vision path verified working post-ablation

How much does quantization actually cost you?

Every quant was scored against the BF16 model as ground truth: run the full-precision model once with --save-all-logits, then measure each quant against those logits with llama-perplexity --kl-divergence over 16k tokens.

The headline number is top-token agreement - how often the quant predicts the same next token the unquantized model would. That is the thing you feel in use, and it is not something you can read off a file size.

Quant	Size	Same top token	KL divergence	Gen speed
Q8_0	9.53 GB	98.2%	0.0019	85 t/s
Q6_K	7.36 GB	96.8%	0.0054	106 t/s
Q5_K_M	6.47 GB	92.9%	0.0316	118 t/s
Q5_K_S	6.30 GB	92.8%	0.0334	126 t/s
Q4_K_M	5.63 GB	90.8%	0.0478	135 t/s
Q4_K_S	5.35 GB	90.5%	0.0526	141 t/s
IQ4_XS	5.20 GB	91.8%	0.0362	138 t/s
IQ4_NL	5.42 GB	91.7%	0.0352	138 t/s
Q3_K_L	4.93 GB	86.8%	0.1125	140 t/s
Q3_K_M	4.62 GB	86.0%	0.1220	147 t/s
Q3_K_S	4.26 GB	82.1%	0.1946	154 t/s
IQ3_M	4.42 GB	86.0%	0.1175	153 t/s
IQ3_S	4.37 GB	86.0%	0.1141	152 t/s
IQ3_XS	4.24 GB	85.2%	0.1243	157 t/s
IQ3_XXS	3.94 GB	83.3%	0.1621	164 t/s
Q2_K	3.83 GB	77.0%	0.3106	165 t/s
Q2_K_S	3.70 GB	75.1%	0.3630	172 t/s
IQ2_M	3.61 GB	77.3%	0.3054	176 t/s
IQ2_S	3.43 GB	73.8%	0.4161	186 t/s
IQ2_XS	3.29 GB	73.0%	0.4594	189 t/s
IQ2_XXS	3.10 GB	68.0%	0.6283	198 t/s
IQ1_M	2.88 GB	57.0%	1.1696	200 t/s
IQ1_S	2.74 GB	46.1%	1.8482	213 t/s

Two findings worth acting on:

• IQ4_XS beats Q4_K_M - it is smaller (5.20 vs 5.63 GB), closer to the original (KL 0.036 vs 0.048), and faster. The I-quants beat the K-quants at nearly every size point on this model. If you were going to grab Q4_K_M out of habit, take IQ4_XS instead.
• The knee is around 3.5-4 GB. Above it, agreement degrades gently. Below IQ2_XXS it falls off a cliff - IQ1_S agrees with full precision only 46% of the time. Those files are included for completeness, not recommended.

Two quants were built, measured, and deleted. Q2_0 and Q1_0 - newer group-64 types - are non-functional on this architecture. Q2_0 matched the full-precision top token 0.0% of the time (KL 12.8) while being larger than IQ1_S, which manages 46%. They are not published. If you see them in other repos for this model family, benchmark before trusting them.

Files

File	Quant	Size	imatrix	Notes
Ornith-1.5-9B-heretic-Q8_0.gguf	Q8_0	9.53 GB	Yes	Effectively lossless. Recommended if you have the VRAM.
Ornith-1.5-9B-heretic-Q6_K.gguf	Q6_K	7.36 GB	Yes	Very high quality, negligible degradation.
Ornith-1.5-9B-heretic-Q5_K_M.gguf	Q5_K_M	6.47 GB	Yes	High quality. A good default for 12-16 GB cards.
Ornith-1.5-9B-heretic-Q5_K_S.gguf	Q5_K_S	6.31 GB	Yes	High quality, slightly smaller than Q5_K_M.
Ornith-1.5-9B-heretic-Q4_K_M.gguf	Q4_K_M	5.63 GB	Yes	Recommended. Best quality/size balance for most users.
Ornith-1.5-9B-heretic-Q4_K_S.gguf	Q4_K_S	5.35 GB	Yes	Slightly smaller than Q4_K_M with a small quality cost.
Ornith-1.5-9B-heretic-IQ4_XS.gguf	IQ4_XS	5.20 GB	Yes	Decent quality, smaller than Q4_K_S. Good for tight VRAM.
Ornith-1.5-9B-heretic-IQ4_NL.gguf	IQ4_NL	5.42 GB	Yes	Similar to IQ4_XS; better for ARM/AVX2 inference.
Ornith-1.5-9B-heretic-Q3_K_L.gguf	Q3_K_L	4.93 GB	Yes	Lower quality but usable.
Ornith-1.5-9B-heretic-Q3_K_M.gguf	Q3_K_M	4.62 GB	Yes	Low quality.
Ornith-1.5-9B-heretic-Q3_K_S.gguf	Q3_K_S	4.26 GB	Yes	Low quality, not recommended.
Ornith-1.5-9B-heretic-IQ3_M.gguf	IQ3_M	4.42 GB	Yes	Medium-low quality, competitive with Q3_K_M.
Ornith-1.5-9B-heretic-IQ3_S.gguf	IQ3_S	4.37 GB	Yes	Beats Q3_K_S at a similar size.
Ornith-1.5-9B-heretic-IQ3_XS.gguf	IQ3_XS	4.24 GB	Yes	Low quality.
Ornith-1.5-9B-heretic-IQ3_XXS.gguf	IQ3_XXS	3.94 GB	Yes	Low quality, beats Q3_K quants of similar size.
Ornith-1.5-9B-heretic-Q2_K.gguf	Q2_K	3.83 GB	Yes	Very low quality, but surprisingly usable.
Ornith-1.5-9B-heretic-Q2_K_S.gguf	Q2_K_S	3.70 GB	Yes	Very low quality, not recommended.
Ornith-1.5-9B-heretic-IQ2_M.gguf	IQ2_M	3.61 GB	Yes	Relatively low quality, SOTA techniques make it usable.
Ornith-1.5-9B-heretic-IQ2_S.gguf	IQ2_S	3.43 GB	Yes	Low quality, uses SOTA techniques to be usable.
Ornith-1.5-9B-heretic-IQ2_XS.gguf	IQ2_XS	3.29 GB	Yes	Low quality, uses SOTA techniques to be usable.
Ornith-1.5-9B-heretic-IQ2_XXS.gguf	IQ2_XXS	3.10 GB	Yes	Very low quality, uses SOTA techniques to be usable.
Ornith-1.5-9B-heretic-IQ1_M.gguf	IQ1_M	2.88 GB	Yes	Extremely low quality. Experimental.
Ornith-1.5-9B-heretic-IQ1_S.gguf	IQ1_S	2.74 GB	Yes	Extremely low quality. Experimental, generally not recommended.

Vision projector (required for image input)

File	Size
mmproj-Ornith-1.5-9B-heretic-f16.gguf	0.92 GB

Ornith-1.5-9B is a vision-language model. For image input, download the mmproj file in addition to your chosen quant.

Which file should I pick?

Aim for a file 1-2 GB smaller than your VRAM for full offload. To maximise quality instead, add system RAM to VRAM and pick 1-2 GB under that total.

K-quants (Q4_K_M) are the safe default. I-quants (IQ4_XS) are smaller for a given quality and work on CUDA/CPU/Metal, but are not supported by Vulkan and are slower on some backends. In doubt, take Q4_K_M.

Usage

llama-cli -m Ornith-1.5-9B-heretic-Q4_K_M.gguf -ngl 99 \
  --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
  -p "Hello"

Sampling matters on this model family. repeat-penalty 1.05 is the sweet spot - 1.0 causes reasoning loops, 1.1 truncates answers. Greedy decoding (--temp 0) is not recommended.

With image input:

llama-mtmd-cli -m Ornith-1.5-9B-heretic-Q4_K_M.gguf \
  --mmproj mmproj-Ornith-1.5-9B-heretic-f16.gguf \
  --image photo.jpg -p "Describe this image." -ngl 99

Note for anyone re-quantizing this model

config.json declares mtp_num_hidden_layers: 1, but the checkpoint ships no mtp.* weights. Converting without --no-nextn produces a GGUF that advertises 33 blocks while containing 32, and llama.cpp refuses to load it:

check_tensor_dims: tensor 'blk.32.attn_norm.weight' not found

python convert_hf_to_gguf.py <model> --outtype bf16 --no-nextn

Provenance

Base model	ornith-ai/Ornith-1.5-9B (MIT)
Abliteration	Heretic 1.4.0
Quantization	llama.cpp b10502
imatrix corpus	bartowski calibration_datav3
Direction sampling	residuals taken with the CoT block closed (\n</think>\n\n)

Credits

• Philipp Emanuel Weidmann for Heretic, which did the actual ablation work here
• Arditi et al. for the directional-ablation result this all rests on
• grimjim for projected and norm-preserving abliteration
• mlabonne for the abliteration write-ups and the prompt datasets
• bartowski for the imatrix calibration corpus that is now the community standard
• the ggml-org/llama.cpp team
• DeepReinforce AI for releasing Ornith under MIT