Amazing. New meta?
Looks like the way abliteration should really be done in practice, not messing with existing weights. Also consumer-friendly 'heretic'—does that really work on the same config llama.cpp can run the inference? Advantages are so many I can't count.
The only downside I noticed is the significant slowdown of text generation with LoRAs (except tiny t256). ~7-8t/s against almost ~10t/s.
Yes, heretic-gguf works on the same llama.cpp that I use to run the inference. It need a harness that you have to compile yourself and it's currently only verified against a specific version of llama.cpp but I plan to verify it works fine against the latest build once I have time.
And yes, the downside is very true. While I think a LoRA is perfect for trying out a model to see how it performs you don't really want a slowdown of text generation for long term use. I have a suspicion that the slowdown is caused by the mismatch in precision (the LoRA is exported in f32/f16). This is also something that I'll look into when I have a bit more time. What quant of the base model did you use?
Could be something with 3D tensors on CPU. Mine is DeepSeek-V4-Flash-0731-UD-Q8_K_XL. But as I said, t256 works without any noticeable slowdown.
I use pretty heavy offloading to GPU and I get the same slowdown and I'm running the same quant. I went from 20t/s to around 15t/s which feels pretty bad. Secondly, did you notice any degradation in model quality on the model quality in the trials with higher KL divergence? I've mostly played around with the ones that kept it low myself.
I tested few prompts on the t264, it seemed to produce correct answers. But can't say I tested it extensively.
Anyway I think ideally there shouldn't be any slowdown, it must be some problem on llama.cpp side because LoRA computation complexity is very low in principle.
I have ~0 clue what I am talking about, but this is probably because (at least partially) every LoRA here except tiny t256 contains tensors with shapes incompatible with any of the optimized CUDA kernels in llama.cpp (for ggml_mul_mat and ggml_mul_mat_id). With LoRA applied I think all matrices need the non-quantized FP path, and mul_mat_vec_f or mul_mat_f seem to both require that the number of columns are even. Only t256 seem to satisfy this condition everywhere.
Only noticed this because I attempted to apply these LoRAs on llama.cpp w/ ROCm, all of which except t256 crashed by failing the MMVF shape assertions because the ROCm CUDA backend did not handle fallback correctly in this case. There is a slow path based on cuBLAS which the normal CUDA code path falls back to, but that one seems to require much more copying and synchronization between host and GPU, which probably also explains why it slows things down.
I think the same tensors, they're slow on CPU too.
I put K3 to see if it could locate what's causing the slowdown, and here is what it said.
What's causing the slowdown
It's not precision mismatch — it's two compounding llama.cpp problems, both specific to adapters that cover routed
expert tensors (ffn_down_exps). That's why t256 (routed MLP off) is fast: it has no 3-D expert LoRA tensors at all.
Adapter buffers aren't tagged as weights. Model weight buffers get GGML_BACKEND_BUFFER_USAGE_WEIGHTS
(llama-model.cpp:1606); LoRA adapter buffers keep USAGE_ANY (llama-adapter.cpp never sets it). The ggml scheduler's
"run the op where the weights live" rule requires that tag, so the LoRA mul_mat_id ops were pulled onto the GPU
even for experts your -ot flags put on CPU — with per-token host→device copies of the LoRA tensors and stream syncs
attached. This was the dominant cost in your config (~25% at batch 1 decode).No fast CUDA kernel for non-quantized mul_mat_id on NVIDIA. The mmvf vector path is AMD-only
(ggml-cuda.cu:1877), mmf requires ne1 % 32 == 0 (rank 1 fails), mmq/mmvq are quantized-only. So every F16
expert-LoRA matmul took a fallback doing 2 stream syncs per call, and its mere presence disabled CUDA graphs for
the whole decode graph. This dominates for users with GPU-resident experts (like the "20→15 t/s" reporter). It's
also the ROCm crash the commenter hit: on AMD the mmvf path is entered unconditionally and our lora_b (ne0 = rank =
1, odd) violates its ncols % 2 == 0 assert.
I verified upstream master (2026-08-16): #26802 fixed the blanket graph-disable but the mmvf gate is still AMD-only
and adapter buffers are still untagged — upgrading llama.cpp alone does not fix this.
The whole thing is soooo eye-opening. I had no idea you can capture tensors so easily via cb_eval during plain llama.cpp inference. This changes a lot in enterprise-cloud-only-gatekept stuff...