I sanity-checked this locally on the full 50-instance swe_bench_verified_mini set using the available GGUFs. I wasn't able to reproduce the metrics.

Setup:

Dataset: MariusHobbhahn/swe-bench-verified-mini, split test
Agent: mini-SWE-agent
Scoring: official swebench.harness.run_evaluation
Runtime: llama.cpp OpenAI-compatible server
Settings: temperature=1.0, top_p=1.0, thinking off, max_tokens=4096, tool_choice=required
Context used: 131072
Per-instance generation wall limit: 420s

Results I got:

Quant Resolved Score Empty patches
━━━━━━━━━━━━━━━━━━
Q3_K_M 32/50 64.0% 3
──────────────────
IQ4_XS 30/50 60.0% 5
──────────────────
IQ4_XS MTP graft 30/50 60.0% 2
──────────────────
Q4_K_M 31/50 62.0% 2
──────────────────
Q5_K_M 32/50 64.0% 1
──────────────────
Q6_K 33/50 66.0% 2
──────────────────
Q8_0 29/50 58.0% 3

The repetition plus visible while “thinking off” is a red flag. I would check:

• Whether the runner is leaking / refeeding reasoning text into the transcript.

• Whether the chat template is the intended Ornith/Qwen-style template.

• Whether tool calls are being forced/parsed correctly. I used tool_choice=required.

• Whether the model is being run with enough context. The GGUF advertises/train context around 262144; my run used 131072, and even that produced a few context/timeout empty patches on Sphinx tail cases. For a cleaner run, use -c 262144 or lower the agent’s compaction trigger.

• Whether stop sequences are accidentally clipping or preserving in a bad way.

The rough reproduction shape was:

serve the quant through llama.cpp OpenAI-compatible server

llama-server
-m /path/to/ornith-1.0-35b-IQ4_XS.gguf
--host 127.0.0.1
--port 8040
-c 131072
--jinja
--alias Ornith-IQ4_XS-mini

Then run mini-SWE-agent against that endpoint:

export HOSTED_VLLM_API_BASE=http://127.0.0.1:8040/v1
export HOSTED_VLLM_API_KEY=dummy

mini-extra swebench \
  --subset MariusHobbhahn/swe-bench-verified-mini \
  --split test \
  --model hosted_vllm/Ornith-IQ4_XS-mini \
  --workers 1 \
  --output ./predictions/iq4_xs \
  --config swebench.yaml \
  --config model.model_kwargs.max_tokens=4096 \
  --config model.model_kwargs.temperature=1.0 \
  --config model.model_kwargs.top_p=1.0 \
  --config model.model_kwargs.tool_choice=required \
  --config agent.wall_time_limit_seconds=420

Then score with the official harness:

python -m swebench.harness.run_evaluation \
  --dataset_name MariusHobbhahn/swe-bench-verified-mini \
  --split test \
  --predictions_path ./predictions/iq4_xs/preds.json \
  --max_workers 8 \
  --timeout 1800 \
  --run_id ornith_iq4_xs_verified_mini \
  --namespace none

Thanks so much for taking the time to help me debug my setup. I am currently re-testing all benchmarks with mini-extra instead of inspect_evals and I'll get back to you with my updated results (it might take a day or two).

I'm using froggeric's v20 template (https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates) in place of the included one - I'll investigate whether that is causing issues as well, but I suspect my test harness was the culprit here.

I ended up doing way more testing than I thought I would, but there were so many variables to check.
Here's the data I've collected over the past few days:

Data

swe_bench_verified_mini                                    ------------ tasks -----------
model (:IQ4_XS)                 config thinking temp top-p limit empty completed resolved score hours
unsloth/Qwen3.6-35B-A3B-MTP     A*  16 off      1.0  0.95  0     1     49        24       0.48  2.2   # establish Qwen3.6 baseline
unsloth/Qwen3.6-35B-A3B-MTP     A*  4  off      1.0  1.0   0     2     41        23       0.46  2.6
unsloth/Qwen3.6-35B-A3B-MTP     H   4  off      1.0  0.95  1     1     46        23       0.46  2.4   # Qwen with llama.cpp and default template
unsloth/Qwen3.6-35B-A3B-MTP     F  i4  off      1.0  0.95  0     1     46        22       0.44  3.4   # Qwen with ik_llama and default template; not better

LordNeel/Ornith-1.0-35B-MTP     A*  16 off      1.0  0.95  2     2     48        25       0.50  4.0   # slight improvement, but still low
LordNeel/Ornith-1.0-35B-MTP     A*  16 off      1.0  1.0   3     4     46        25       0.50  3.6
LordNeel/Ornith-1.0-35B         B   4  off      1.0  0.95  0     3     46        27       0.54  2.7   # base template and no MTP
LordNeel/Ornith-1.0-35B         B   8  off      1.0  1.0   2     2     48        27       0.54  2.6   
LordNeel/Ornith-1.0-35B         B*  8  off      1.0  0.95  6     7     43        24       0.48  5.0   # the froggeric template is worse
LordNeel/Ornith-1.0-35B         C ^ 8  off      1.0  1.0   4     4     46        26       0.52  3.0   # increase kv cache quant; no change
LordNeel/Ornith-1.0-35B         E  i8  off      1.0  1.0   3     3     47        29       0.58  3.0   # try ik_llama
LordNeel/Ornith-1.0-35B-MTP     F  i4  off      1.0  0.95  7     7     43        24       0.48  4.5   # retry MTP; terrible, ik_llama was no help

Tribbler/ornith-apex:I-Compact  G  i4  off      1.0  0.95  1     1     49        25       0.50  2.8   # smaller Ornith quant; ik_llama not actually better?
LordNeel/Agents-A1              G  i8  off      1.0  1.0   0     1     49        20       0.40  3.3   # seems low to me, but I don't know this model

* = froggeric jinja template, otherwise default chat template from gguf
^ = -ctk f16 -ctv q8_0, otherwise -ctk q8_0 -ctv q5_0
i = ik_llama.cpp, otherwise llama.cpp
# = max_tokens (kibi tokens)
'limit' means a time/turn/context_window limit was exceeded, aborting the task without creating a patch

Discussion (ramblings)

I wasn't sure what the effect of top-p=1.0 (differing from Qwen3.6 default of 0.95) might be, so I tested both.
They are quite similar, so it's a bit like having two near-replicates for those tests. I should have just kept this fixed, but I wanted to test 0.95 because it's the config I'll actually use in practice.

I somewhat inadvertently varied max_tokens more than I intended. I was getting a lot of LimitExceeded errors with 4096 tokens at one point, so I raised it thinking it helped, but in hindsight that was not the remedy.
It does seem that max_tokens doesn't really affect the scores much, but that was some sloppy testing on my part.
Edit: now that I've seen all the data I realize I saw more LimitExceeded messages because the 4k token limit allowed the 250 turn limit to be reached more often, so I shouldn't have been concerned about it, as those tasks were likely to fail anyway.

All tests used --parallel 1 and either 128k or 256k context. I did see a context_length_exceeded message at least once, but my suspicion is that any task that blows through 128k of context is not on a successful trajectory anyway.

MTP does seem to be worse, which is unfortunate. I'm unsure if it underperforms due to speed issues or because it enters into repetition. I read a reddit post about MTP causing loops which led me to test it specifically.
Then I wondered if kv cache quantization was the issue. k/v of 16/8 did not improve anything, and I don't have the VRAM to run unquantized.
The froggeric template seems to be worse for Ornith, likely causing looping/repetition/nonsense as evidenced by the extended times.
I actually walked though the two templates with ChatGPT before applying it to Ornith and it seemed like there would be no issues, but that was apparently wrong - I'll step through it more carefully, because I do want to root-cause the template incompatibility.
As a sanity check I served the model with ik_llama.cpp just to rule out any issues with the specific release of llama.cpp used. I was not expecting an improvement, and it raises more questions than it answers.
It's now seeming like the ik_llama 58% score is more of an outlier than a consistent score increase. That would actually be a relief so I'm not chained to ik_llama, because I'd rather just stick with the mainline fork.

The froggeric template does not seem to hinder Qwen3.6, just Ornith.

Even with my best-scoring result, I'm just barely matching the worst score reported for any quant in LordNeel's testing.
I find it hard to believe that kv cache quantization makes up that defecit, and I don't see any other settings differences, so I'm still at a bit of a loss.
At least I'm now measuring a better score with Ornith than base Qwen (as the model card indicates I should expect), but I still don't know why I'm missing about 5-10% accuracy on this test.

Configuration

Running on an RTX 4090, CUDA Toolkit 13.3, NVIDIA drivers 610.62

llama.cpp version:

version: 9837 (b3fed31b9)
built with Clang 20.1.8 for Windows x86_64
(via winget: https://github.com/ggml-org/llama.cpp/releases/download/b9837/llama-b9837-bin-win-vulkan-x64.zip)

ik_llama.cpp version:

version: 4666 (f96eaddb)
built with MSVC 19.51.36248.0 for x64
(locally compiled)

Extra configuration settings:
A:

set LLAMA_ARG_CHAT_TEMPLATE_FILE=D:\model_cache\chat_templates\Qwen3_chat_template_froggeric_v20.jinja
set LLAMA_SPEC_MAX_DRAFTING_SLOTS=1
set LLAMA_MTP_FAST_BACKEND_SAMPLE=1
set LLAMA_MTP_DRAFT_TOP_K=1
set LLAMA_MTP_DRAFT_TOP_P=1
set LLAMA_MTP_DRAFT_TEMP=1

rem The unsloth configuration only changes the --model param

llama-server ^
  --model D:\model_cache\models--LordNeel--Ornith-1.0-35B-GGUF-llamacpp-tp1\ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf ^
  --n-gpu-layers 99 ^
  --fit off ^
  --parallel 1 ^
  --ctx-size 262144 ^
  --spec-type draft-mtp --spec-draft-n-max 2 ^
  --flash-attn on ^
  --temp 1.0 ^
  --min-p 0.0 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --repeat-penalty 1.0 ^
  --presence-penalty 0.0 ^
  --cache-type-k q8_0 ^
  --cache-type-v q5_0 ^
  --cache-type-k-draft q8_0 ^
  --cache-type-v-draft q5_0 ^
  --mlock ^
  --no-mmap ^
  --jinja ^
  --reasoning off

B: Identical to A except for these changes:

rem No env variables are set in variation B
rem set LLAMA_ARG_CHAT_TEMPLATE_FILE=D:\model_cache\chat_templates\Qwen3_chat_template_froggeric_v20.jinja
rem set LLAMA_SPEC_MAX_DRAFTING_SLOTS=1
rem set LLAMA_MTP_FAST_BACKEND_SAMPLE=1
rem set LLAMA_MTP_DRAFT_TOP_K=1
rem set LLAMA_MTP_DRAFT_TOP_P=1
rem set LLAMA_MTP_DRAFT_TEMP=1

rem The model is the non-MTP version
llama-server ^
  --model D:\model_cache\models--LordNeel--Ornith-1.0-35B-GGUF-llamacpp-tp1\ornith-1.0-35b-IQ4_XS.gguf ^

C: Identical to B except for these changes:

llama-server ^
  --ctx-size 131072 ^
  --cache-type-k f16 ^
  --cache-type-v q8_0 ^

D: Identical to C except for this change:

set LLAMA_ARG_CHAT_TEMPLATE_FILE=D:\model_cache\chat_templates\Qwen3_chat_template_froggeric_v20.jinja

E: Identical to C except for these changes:

C:\Progra~1\ik_llama.cpp\bin\llama-server.exe ^
  --k-cache-hadamard --cache-type-k q8_0 ^
  --v-cache-hadamard --cache-type-v q5_0 ^

F: Identical to A except for these changes:

rem (The unsloth configuration only changes the --model param)

C:\Progra~1\ik_llama.cpp\bin\llama-server.exe ^
  --model D:\model_cache\models--LordNeel--Ornith-1.0-35B-GGUF-llamacpp-tp1\ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf ^
  --spec-type mtp:n_max=2,p_min=0.0 ^
  --k-cache-hadamard --cache-type-k q8_0 ^
  --v-cache-hadamard --cache-type-v q5_0 ^

G: Identical to E except for these changes:

rem (The Agents-A1 configuration only changes the --model param)

C:\Progra~1\ik_llama.cpp\bin\llama-server.exe ^
  --model D:\model_cache\models--Tribbler--ornith-1.0-apex\ornith-1.0-35b-APEX-I-Compact.gguf ^
  --ctx-size 262144 ^

H: Identical to A except for this change:

rem No chat template override was set
rem set LLAMA_ARG_CHAT_TEMPLATE_FILE=D:\model_cache\chat_templates\Qwen3_chat_template_froggeric_v20.jinja

Benchmark script

I ran the benchmark with these settings under wsl:

run_swe_bench_mini_p1() {
  local LOG_DIR="${OUTPUT_ROOT}/swe_bench_mini"
  mkdir -p "${LOG_DIR}"

  local DATASET="MariusHobbhahn/swe-bench-verified-mini"

  export HOSTED_VLLM_API_BASE="http://127.0.0.1:8080/v1"
  export HOSTED_VLLM_API_KEY="dummy"
  export MSWEA_COST_TRACKING="ignore_errors"

  echo "Running SWE-bench-mini Temp: ${TEMP}"

  local TARGET_MODEL="hosted_vllm/${RAW_MODEL_ID}:temp_${TEMP}${OPTS_SUFFIX}"
  local TEMP_DIR="${LOG_DIR}/temp_${TEMP}_p1"
  local PREDS_PATH="${TEMP_DIR}/preds.json"
  local RUN_ID="swe_bench_mini_${MODEL_NAME}_temp_${TEMP}"

  mkdir -p "${TEMP_DIR}"

  # Phase 1: Generate Predictions
  echo "Executing agent trajectories..."

  mini-extra swebench \
    --subset "${DATASET}" \
    --split test \
    --model "${TARGET_MODEL}" \
    --workers 1 \
    --output "${TEMP_DIR}" \
    --config swebench.yaml \
    --config agent.wall_time_limit_seconds=1800 \
    --config model.model_kwargs.max_tokens=<4096 to 16384> \
    --config model.model_kwargs.temperature="${TEMP}" \
    --config model.model_kwargs.tool_choice=required \
    --config model.model_kwargs.top_p=1.0

  # Phase 2: Evaluate and Score
  if [ -f "${PREDS_PATH}" ]; then
    echo "Predictions generated successfully at ${PREDS_PATH}. Starting official harness evaluation..."

    python -m swebench.harness.run_evaluation \
      --dataset_name "${DATASET}" \
      --split "test" \
      --predictions_path "${PREDS_PATH}" \
      --max_workers 8 \
      --timeout 1800 \
      --run_id "${RUN_ID}" \
      --namespace none

    echo "Temp ${TEMP} complete."
  else
    echo "Error: Prediction file not found at ${PREDS_PATH}. Aborting evaluation."
    exit 1
  fi
}

Earlier testing

Earlier inspect_evals harness testing (llama.cpp version was not consistent across this testing):

model                        thinking      temp  quant        aime2025       aime2026       bbh            gpqa_diamond    humaneval      ifeval         scicode (bad)  swe_bench_verified_mini
---------------------------  ----------  ------  -----------  -------------  -------------  -------------  --------------  -------------  -------------  -------------  -------------------------
Ornith-1.0-35B-llamacpp-tp1  off              0  IQ4_XS:kq85  80.0% (0.20h)  83.3% (0.19h)  83.6% (2.54h)  83.7% (4.19h)   95.1% (0.08h)  81.3% (0.39h)  21.0% (0.52h)  —
Ornith-1.0-35B-llamacpp-tp1  off              1  IQ4_XS:kq85  —              —              —              —               —              —              —              ~30% score when aborted
---------------------------  ----------  ------  -----------  -------------  -------------  -------------  --------------  -------------  -------------  -------------  -------------------------
Qwen3.6-27B                  off              0  IQ4_XS:kq85  —              —              —              —               92.1% (0.20h)  84.5% (1.48h)  26.9% (2.04h)  —
Qwen3.6-27B @ b9553?         off              1  IQ4_XS:      —              —              —              —               —              —              —              66.0% (5.94h)            # I have yet to reproduce this score
Qwen3.6-27B                  on               1  IQ4_XS:kq85  —              —              —              —               —              —              —              60.0% (7.63h) (low??)
---------------------------  ----------  ------  -----------  -------------  -------------  -------------  --------------  -------------  -------------  -------------  -------------------------
Qwen3.6-35B-A3B              off              0  IQ4_XS:kq85  73.3% (0.48h)  73.3% (0.20h)  82.5% (2.17h)  —               96.3% (0.07h)  81.5% (0.40h)  25.8% (0.64h)  —
Qwen3.6-35B-A3B              off              1  IQ4_XS:kq8   —              —              —              —               —              —              —              50.0% (3.38h)

kq85 = -ctk q8_0 -ctv q5_0
kq8 = -ctk q8_0 -ctv q4_0
none = -ctk q4_0 -ctv q4_0
The scicode benchmark had issues with scoring, so the scores are depressed from what they should be. These benchmarks are all run with inspec_evals, which I don't have much faith in any more.

TL;DR: all discussion is in the Discussion section ^^