Add model card and per-leg benchmark results

Model card for the four shoehorn budget-exact fits, plus the six-leg accuracy benchmark: summary JSON, per-case CSV and the exact server config for each leg. Raw model outputs are deliberately excluded (they quote benchmark questions verbatim).

README.md

@@ -0,0 +1,276 @@
+---
+base_model: Qwen/Qwen3.8-27B
+license: apache-2.0
+library_name: gguf
+pipeline_tag: text-generation
+tags:
+  - gguf
+  - quantized
+  - llama.cpp
+  - shoehorn
+---
+
+# Qwen3.8-27B — shoehorn budget-exact fits
+
+Four GGUF quantizations of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B),
+produced with [shoehorn](https://github.com/notactuallytreyanastasio/shoehorn),
+plus the accuracy benchmark that checks whether they still think.
+
+A preset quant like `Q8_0` or `IQ4_XS` applies one recipe to every tensor, and
+whatever size falls out is the size you get. shoehorn works the other way round:
+you give it a VRAM budget, a context length and a KV cache type, and it solves
+for a per-tensor mix of quant formats that fills whatever is left after
+inference overhead. The point is to land on a specific card at a specific
+context, not to hit a particular bits-per-weight number.
+
+So the obvious question is whether solving for a size costs you accuracy. We
+benchmarked all four against the two stock quants. The short answer is in the
+table below; the long answer is that at 4.9 bits per weight and up, this
+benchmark could not measure a difference, and at 2.9 bits it very clearly could.
+
+## The files
+
+| file | bpw | size | solved for | GiB |
+|---|---|---|---|---|
+| `qwen38-27b-fit-128k-q4kv.gguf` | 6.099 | 20,838,706,944 B | 131072 ctx, q4_0 KV, 23.5 GiB budget | 19.41 |
+| `qwen38-27b-fit-192k-q4kv.gguf` | 5.509 | 18,825,459,584 B | 196608 ctx, q4_0 KV, 23.5 GiB budget | 17.53 |
+| `qwen38-27b-fit-256k-q4kv.gguf` | 4.920 | 16,812,197,504 B | 262144 ctx, q4_0 KV, 23.5 GiB budget | 15.66 |
+| `qwen38-27b-fit-256k-17g.gguf`  | 2.876 |  9,832,877,184 B | 262144 ctx, q4_0 KV, **17 GiB** budget | 9.16 |
+
+Every budget above is a budget for weights **plus** KV cache plus overhead, at
+the stated context. That is what "solved for" means here — the 17 GiB file was
+built to fit a 24 GB card together with its speculative-decoding draft model
+and a full 256K context, and it does: 21.6 GiB, measured in earlier testing of that configuration.
+
+These are text-only. The base model is multimodal; no vision projector
+(`mmproj`) is included here and the vision path was neither converted nor
+tested.
+
+### Exactly which bytes were benchmarked
+
+Every number in this card was produced by these exact files. The published
+weights are byte-identical to the ones that ran the benchmark — the hashes
+below were taken on the machine that ran the benchmark, and match what
+HuggingFace stores for each of the four published files.
+
+| file | size (bytes) | sha256 |
+|---|---|---|
+| `qwen38-27b-fit-128k-q4kv.gguf` | 20838706944 | `f30987d23ed027d4ecf5e49aed5a8479162e8d4f796c6e8a541d15e16d08384a` |
+| `qwen38-27b-fit-192k-q4kv.gguf` | 18825459584 | `2faadd35212ae9132e147eb22e423a7d708aed19da1bbc5a5492029806ad99f6` |
+| `qwen38-27b-fit-256k-q4kv.gguf` | 16812197504 | `b156c0c1aa0a2659e4d0ed203f7e6e7f8ff847a081f77f07aebf683cea80eb7a` |
+| `qwen38-27b-fit-256k-17g.gguf` | 9832877184 | `787b2bfafc39177d051b13e5c2359c5950c92496b9ed1fd3f51df750ec78dd6b` |
+
+Verify with `sha256sum <file>` after downloading.
+
+The comparison files, which are **not** in this repo but which the benchmark
+numbers depend on:
+
+| file | size (bytes) | sha256 |
+|---|---|---|
+| `Qwen3.8-27B-Q8_0.gguf` | 29047085024 | `57484e3196aaff8dbbd71c666158b00206baa2bc134e4f16ae90fc8cbdfbeea2` |
+| `Qwen3.8-27B-IQ4_XS.gguf` | 15705860064 | `8b29da0e6cbc5032d478cd2333f5071b5195b4fa903c1d51d86704386c18fb29` |
+| `Qwen3.8-27B-MTP-ONLY-Q8_0.gguf` | 4503398688 | `674d0fc3b2b09c48cf77fbab0aba39b9c4ee538bd240fa87c1f13044260f7d7b` |
+
+The two stock quants are from
+[unsloth/Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF),
+and the revision matters. Both files match that repo exactly at any revision
+**before** commit `408fcc1807ab` (2026-08-14, "Add Unsloth style chat
+template"). That commit embedded a chat template and grew each file by exactly
+1024 bytes, so the current `main` copies hash differently. If you download them
+today you will get the +1024-byte versions.
+
+We checked whether that mattered, because it is the kind of difference that
+quietly breaks a comparison. The fits were built from our own BF16 conversion
+and carry a different embedded chat template (9993 characters) than the
+baselines (8945). Different prompt formatting between the two groups would
+invalidate the whole table.
+
+It does not, for this eval. The harness sends exactly one system message and
+one user message and uses no tools. Rendering both templates on that input
+produces byte-identical output — 563 characters, same hash, both resolving
+`reasoning_effort` to `xhigh`. Every difference between the two templates lives
+in a path this benchmark never takes: multi-system-message merging, tool-call
+argument handling, and a `high`→`xhigh` alias.
+
+## Benchmark
+
+92 questions, four sources: AIME2025 (25), GPQA Diamond (25), SuperGPQA (25),
+COMPSEC (17).
+
+Every leg ran on the same RTX PRO 6000 Blackwell at a 250 W cap, the same
+llama.cpp build (22b8e31), the same MTP draft model for speculative decoding,
+`q4_0` KV cache, flash attention on, temperature 0, seed 42, 32768 max tokens
+per answer.
+
+One caveat you will notice in the published CSVs: 24 rows are tagged
+`GPQA Diamond` and one is tagged `GPQA Diamond (modified)`. That marker is
+inherited verbatim from the upstream ds4 corpus — it is antirez's annotation,
+not a change of ours — and the item is counted within the 25. So a reader
+tallying `source == "GPQA Diamond"` gets 24, not 25.
+
+| leg | bpw | ctx | accuracy | pass | wrong | exhausted | median case | median completion tok |
+|---|---|---|---|---|---|---|---|---|
+| `Qwen3.8-27B-Q8_0` (stock) | 8.500 | 131072 | 83.7% | 77 | 7 | 8 | 51s | 3475 |
+| `Qwen3.8-27B-IQ4_XS` (stock) | 4.596 | 131072 | 84.8% | 78 | 5 | 9 | 42s | 3721 |
+| fit-128k | 6.099 | 131072 | 88.0% | 81 | 5 | 6 | 53s | 3612 |
+| fit-192k | 5.509 | 196608 | 88.0% | 81 | 4 | 7 | 56s | 4178 |
+| fit-256k | 4.920 | 262144 | 87.0% | 80 | 5 | 7 | 63s | 4436 |
+| fit-17g | 2.876 | 131072 | 67.4% | 62 | 9 | 21 | 64s | 6464 |
+
+By source:
+
+| leg | AIME2025 | COMPSEC | GPQA Diamond | SuperGPQA |
+|---|---|---|---|---|
+| Q8_0 | 84% (21/25) | 100% (17/17) | 80% (20/25) | 76% (19/25) |
+| IQ4_XS | 88% (22/25) | 100% (17/17) | 76% (19/25) | 80% (20/25) |
+| fit-128k | 96% (24/25) | 88% (15/17) | 84% (21/25) | 84% (21/25) |
+| fit-192k | 92% (23/25) | 94% (16/17) | 84% (21/25) | 84% (21/25) |
+| fit-256k | 92% (23/25) | 100% (17/17) | 80% (20/25) | 80% (20/25) |
+| fit-17g | 64% (16/25) | 100% (17/17) | 48% (12/25) | 68% (17/25) |
+
+Note that the three 23.5 GiB fits each ran at the context they were solved for,
+while the stock quants and fit-17g ran at 131072. That is deliberate — running
+a fit at its target context is the thing the fit exists to do.
+
+The comparison worth looking at twice is fit-256k against stock IQ4_XS: 4.920
+bpw against 4.596, 15.66 GiB against 14.63, and 87.0% against 84.8% — but
+fit-256k carried a 262144-token context while IQ4_XS ran at 131072.
+
+### Read the top five rows as one cluster
+
+The fits come out a few points above Q8_0. **That is not a result.** The
+comparison is paired, so only questions where two legs disagree carry
+information, and there are very few of them: fit-128k versus Q8_0 is 8
+discordant questions out of 92, splitting 6–2 in the fit's favour, McNemar
+exact p = 0.29. Every pairwise comparison among those five legs is
+non-significant.
+
+Simulating the exact test at the discordance rate these legs actually produce,
+the power at 92 questions is **0.17**. Eighty percent power would need about
+400 questions. Even a perfect effect — every discordant question falling the
+same way — only reaches 0.82 at this sample size.
+
+So the defensible claim is **no measurable accuracy loss against Q8_0 down to
+4.920 bpw**, and explicitly *not* that the fits are better than Q8_0. If you
+want a ranking among those five, this benchmark cannot give you one.
+
+### fit-17g is different, and the reason matters
+
+The 2.876 bpw file is the one comparison here that clears the bar. Against
+Q8_0: 19-vs-4 discordant, p = 0.003, power 0.88. Against fit-256k: 20-vs-2,
+p = 0.0001.
+
+But *how* it degrades is more useful than the headline. Its answered-only
+accuracy is 87.3% (62/71) against Q8_0's 91.7% (77/84) — a gap of about 4
+points, not 16. The headline gap is mostly **exhaustion**: 21 cases ran out of
+the 32768-token budget before committing to an answer, against 6–9 for every
+other leg, and its median completion length is 6464 tokens against 3475–4436.
+
+It is not mostly answering wrongly, and it is not uniformly slower to think
+either. On the questions it finishes, its median completion is 3818 tokens
+against Q8_0's 2956 — about 29% longer, not double. The doubled headline median
+is the *upper tail*: its third quartile is 28824 tokens, against 11k–13.5k for
+every other leg (exclusive quartiles). Most questions look normal; a subset runs away and hits the
+wall.
+
+The collapse is concentrated where the reasoning is longest: GPQA Diamond falls
+to 48% while COMPSEC stays at 100%.
+
+A larger token budget would likely recover part of that 16 points. We have not
+tested that, and until someone does, treat 67.4% as "this configuration, at
+32768 tokens" rather than as the model's ceiling.
+
+### Failure follows the question, not the quantization
+
+Across all six legs, 58 exhaustion events land on only 22 distinct questions,
+and 4 questions exhaust in **all six** legs — including Q8_0. Under
+independence with the same per-leg counts you would expect 0.000 questions to
+exhaust everywhere, and a permutation test over question ids gives p < 5e-6
+(200,000 trials).
+
+Some questions are simply long-reasoning traps, and that property survives
+across quant families and across a 3x range in bits per weight.
+
+## Running them
+
+```
+llama-server \
+  -m qwen38-27b-fit-128k-q4kv.gguf \
+  -md Qwen3.8-27B-MTP-ONLY-Q8_0.gguf --spec-draft-n-max 3 \
+  -c 131072 -ctk q4_0 -ctv q4_0 -fa on \
+  -ngl 99 -ngld 99 --host 127.0.0.1 --port 8083
+```
+
+Set `-c` to the context the file was solved for. The KV type is not optional:
+these were solved against `q4_0` KV, and running f16 KV changes the memory
+arithmetic the fit was built around.
+
+`-ngld 99` matters — without it the draft model runs on the CPU and speculative
+decoding ends up slower than not doing it at all.
+
+The draft model used here is an MTP-only Q8_0 extraction from the same base.
+It is not included in this repo.
+
+## How the fits were made
+
+```
+shoehorn --ctx 131072 --kv q4_0 --budget 23.5GiB --calibrate
+shoehorn --ctx 196608 --kv q4_0 --budget 23.5GiB --calibrate
+shoehorn --ctx 262144 --kv q4_0 --budget 23.5GiB --calibrate --force-calibrate
+shoehorn --ctx 262144 --kv q4_0 --budget 17GiB   --calibrate --force-calibrate
+```
+
+Starting from a BF16 GGUF and an imatrix generated with `llama-imatrix` on
+bartowski's calibration text.
+
+`--calibrate` is the part that does the work. It writes the file, launches
+llama.cpp once, reads back the *measured* allocations and re-solves against
+them. At 128K the KV estimate was 9.14 GiB and the real allocation was 2.25 GiB
+— most layers in a hybrid model don't keep a full-attention cache — and the
+re-solve handed 5.75 GiB back to the weights.
+
+`--force-calibrate` is our own patch, on a fork. Three estimate-based gates
+bail out before `--calibrate` ever runs, which means the tool can refuse a job
+on the strength of an estimate it is about to discover is wrong. The flag turns
+those gates into warnings. Both 256K fits need it.
+
+On a multi-GPU box, pass `--budget` explicitly — the NVML probe reads device 0
+only.
+
+## What's in this repo
+
+Per-leg results for all six benchmark legs live under `bench/`: the summary
+JSON, the per-case CSV (question id, source, domain, pass/fail, token counts,
+timings) and a `config-public.txt` recording the exact server command, sampler
+settings, build and driver version for that leg.
+
+The evaluation harness itself is ours and is not public, so the eval is not
+reproducible from this repo — the inference side is (the server command is
+recorded per leg, and the question sources are named above).
+
+The raw model outputs are **not** published. They quote the benchmark questions
+verbatim, and 25 of the 92 come from GPQA Diamond, whose authors ask that
+examples not be reproduced online. The per-case CSVs carry question ids and
+results but no question text.
+
+## Credits and licences
+
+- **Base model**: [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B)
+  by the Qwen team, Apache-2.0. These are derived weights and the base model's
+  licence terms govern them; the quantization does not change what you may do
+  with the model.
+- **Quantizer**: [shoehorn](https://github.com/notactuallytreyanastasio/shoehorn),
+  MIT. The budget solver, the calibration loop and the per-tensor format
+  assignment are all its work. The `--force-calibrate` flag is a local patch on
+  a fork, not upstream.
+- **Evaluation**: [antirez/ds4](https://github.com/antirez/ds4) ("DwarfStar")
+  by Salvatore Sanfilippo, MIT — an inference engine for DeepSeek V4 Flash and
+  GLM 5.2 on consumer hardware. Our harness is a Rust port of its `ds4_bench.c`
+  and `ds4_eval.c`, and the 92-question selection is antirez's. These numbers
+  would not exist without it.
+- **Inference**: [llama.cpp](https://github.com/ggml-org/llama.cpp), MIT.
+
+Benchmark questions originate with their respective authors: AIME2025, GPQA
+Diamond, SuperGPQA, and a computer-security set. Because the question set is
+public, treat these numbers as a *relative* comparison between quantizations
+run under identical conditions — not as leaderboard-comparable scores.

bench/fit-128k/config-public.txt

@@ -0,0 +1,25 @@
+run:        fit-128k
+recorded:   2026-08-16T01:23:21+02:00 RETROACTIVELY — see caveat below
+source:     bin/bench-fits.sh (file last changed in ab44c65, i.e. unchanged
+            across all three fit legs) — the command below is
+            reconstructed from that script, which is version controlled and was
+            unchanged across all three fit legs. This is the authoritative record.
+            The serve log does NOT contain the config — verified 2026-08-16:
+            no type_k/type_v, no flash-attn, no build version at this verbosity.
+serve_cmd:  ~/llama.cpp/build-mtp/bin/llama-server -m ~/models/qwen38-27b-fit-128k-q4kv.gguf -md ~/models/Qwen3.8-27B-MTP-ONLY-Q8_0.gguf --spec-draft-n-max 3 -c 131072 -ctk q4_0 -ctv q4_0 -fa on -ngl 99 -ngld 99 --port 8083 --host 0.0.0.0
+client:     bs4 qa-eval --base-url http://<redacted>:8083/v1 --model fit-128k
+sampler:    temperature 0.0, seed 42, max_tokens 32768 (bs4, sent in request body)
+--- desktop side (captured 2026-08-16, not at run time) ---
+llama.cpp:  22b8e31
+model:      20838706944 bytes  ~/models/qwen38-27b-fit-128k-q4kv.gguf
+gpu:        NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 250.00 W
+driver_ver: 610.57.04
+--- caveat ---
+Written after the fact. Trustworthy because the driver script is in git and
+unchanged since the runs; NOT equivalent to a config captured before the run.
+
+--- publication note ---
+Scrubbed for publication by sanitize-config.py: hostnames, internal
+repo paths and home paths removed, GPU index omitted (immaterial —
+one card, one power cap), chat template omitted for size. Everything
+that affects the measurement is kept verbatim.

bench/fit-128k/qa-eval-1786818550.csv

@@ -0,0 +1,93 @@
+id,source,domain,result,reasoning_chars,answer_chars,completion_tokens,finish_reason,ttft_ms,total_ms
+recNu3MXkvWUzHZr9,GPQA Diamond,Physics,pass,17606,392,7047,stop,400.5,90297.2
+001b51d76b4d422988f2c11f104a2c6c,SuperGPQA,Agronomy,pass,2328,9,558,stop,496.1,7497.7
+aime2025-01,AIME2025,Number Theory,pass,1843,418,980,stop,446.1,11925.3
+recoiTJPGUmzAkief,GPQA Diamond,Physics,pass,8151,465,1886,stop,571.1,30781.2
+b7e20eac98764fb0bf30e8366d951daa,SuperGPQA,Engineering,pass,17331,450,4382,stop,670.6,58830.4
+aime2025-16,AIME2025,Geometry,pass,2940,768,1491,stop,571.2,19670.4
+rec4UqStf9WUVif1f,GPQA Diamond,Physics,pass,1944,585,1230,stop,556.7,15139.9
+4a1d1780a93f4093b6fb7d3c314cbea8,SuperGPQA,Engineering,fail,7910,9,1665,stop,683.0,25679.1
+aime2025-02,AIME2025,Geometry,pass,28249,1406,11789,stop,1028.4,159785.4
+recgI6tUQ7RLJRWGx,GPQA Diamond,Physics,pass,5374,663,2295,stop,716.5,32177.8
+6082513c8dba4ec68aa68f1bf5854d09,SuperGPQA,Law,pass,2420,9,490,stop,520.7,7058.2
+aime2025-03,AIME2025,Combinatorics,pass,2046,423,1006,stop,559.8,13059.6
+recDytVnNYZe2HuUU,GPQA Diamond (modified),Physics,pass,9308,415,4276,stop,545.8,58811.2
+bebf1ed45ae14ad7b4f205f3909cb58a,SuperGPQA,Literature and Arts,pass,1146,9,273,stop,662.5,4326.1
+aime2025-18,AIME2025,Combinatorics,pass,18335,1428,6884,stop,665.7,95518.7
+recNFJjE5PPTqVJGv,GPQA Diamond,Chemistry,pass,1387,438,838,stop,820.8,11266.7
+7ca71b86327744b78e93185a45bc5cef,SuperGPQA,Medicine,pass,2124,9,521,stop,556.3,7313.9
+aime2025-04,AIME2025,Algebra,pass,2730,457,1401,stop,495.3,18458.3
+rec2UlKqC6RFHdcro,GPQA Diamond,Physics,pass,2023,448,872,stop,687.9,12094.6
+d44b94f7749345a39a65f6312bda8764,SuperGPQA,Engineering,pass,8254,323,2246,stop,558.5,35600.3
+aime2025-19,AIME2025,Algebra,pass,1693,1024,1263,stop,682.4,15963.5
+recv7GsQg3f0fvB1f,GPQA Diamond,Chemistry,pass,15177,420,3676,stop,568.4,55520.3
+febe406f44d74a40b50bb5b7c69d5dc1,SuperGPQA,Sociology,pass,28536,9,6506,stop,612.6,88599.8
+aime2025-05,AIME2025,"Combinatorics, Number Theory",pass,11415,1370,5418,stop,634.1,70548.0
+recrHBEJJoDTV05JR,GPQA Diamond,Chemistry,pass,65159,774,22432,stop,635.4,310700.3
+31950dc80ded400a9181f50626d1f75c,SuperGPQA,Science,pass,971,369,560,stop,680.8,7480.8
+aime2025-06,AIME2025,Geometry,pass,4189,709,1863,stop,464.3,25779.5
+recb80OwMgNnceA9t,GPQA Diamond,Chemistry,exhausted,95076,0,32768,length,504.8,488278.3
+0f14cd17be174618af6d60227e7dca9f,SuperGPQA,Engineering,pass,9209,954,3721,stop,1171.5,52322.5
+aime2025-21,AIME2025,"Algebra, Geometry",pass,17225,1376,6357,stop,896.8,87763.4
+recA1i5ZAh0Uzclxp,GPQA Diamond,Chemistry,pass,55590,884,17601,stop,599.2,261851.2
+cef9bcc087434cc2b4e354a9baef55eb,SuperGPQA,Engineering,pass,813,589,805,stop,790.7,9923.6
+aime2025-07,AIME2025,Combinatorics,pass,20659,1913,6650,stop,507.6,99698.6
+recqGD3fxPCI59vPQ,GPQA Diamond,Physics,pass,21916,729,7057,stop,559.1,104354.7
+9f93aa2cfdb547b5b3a4623f80f7fff6,SuperGPQA,Engineering,pass,1064,147,314,stop,565.0,4691.1
+aime2025-22,AIME2025,Number Theory,pass,6000,899,2719,stop,468.5,35268.8
+rechKl68Uc6H7vU0N,GPQA Diamond,Chemistry,pass,74392,522,24594,stop,690.0,349015.1
+97ad69dda7b2462c98638b79e78aea0b,SuperGPQA,Medicine,pass,3302,9,801,stop,714.2,12331.5
+aime2025-08,AIME2025,Algebra,pass,4135,885,2167,stop,495.9,28137.8
+rec1zl5LvaatzGhFt,GPQA Diamond,Physics,pass,2097,364,803,stop,645.3,11301.3
+e78e4e539d6f4e379ac140d923d7b1be,SuperGPQA,Military Science,fail,7256,9,1486,stop,471.0,21903.9
+aime2025-09,AIME2025,Algebra,pass,14188,1167,7290,stop,480.6,93998.9
+recTs7qzfJs6kfLUK,GPQA Diamond,Biology,pass,27479,9,9115,stop,837.1,128131.8
+8483667a25e74fdfa3188de4ea734f03,SuperGPQA,Science,pass,83439,864,24825,stop,569.4,360496.5
+aime2025-24,AIME2025,Algebra,pass,8589,1223,3775,stop,703.2,52751.2
+rec32C1ZEapBnCC0E,GPQA Diamond,Physics,pass,2810,381,1119,stop,539.3,16393.1
+e5ed76ef98144f06843125daf1bccd35,SuperGPQA,Science,pass,1245,439,812,stop,512.3,10355.4
+aime2025-10,AIME2025,Combinatorics,pass,57364,2560,19152,stop,697.7,275600.6
+recZWeueB7lSPR6wN,GPQA Diamond,Chemistry,exhausted,103323,0,32768,length,665.1,494772.6
+fd7924876c4845cd83d95d61dfa0b236,SuperGPQA,Sociology,pass,10320,9,2044,stop,750.0,30335.5
+aime2025-25,AIME2025,Combinatorics,pass,15604,869,5090,stop,466.3,74222.3
+recVvpD8miVjmmyfe,GPQA Diamond,Biology,exhausted,70673,0,32768,length,504.9,473515.6
+6bfe7d19299d4b3184636e1f51694306,SuperGPQA,Agronomy,pass,52354,461,16689,stop,1000.1,234227.7
+aime2025-26,AIME2025,Combinatorics,pass,26649,1335,8724,stop,604.3,128327.3
+recAAJoHMW45Lv5je,GPQA Diamond,Chemistry,pass,43395,825,13841,stop,588.7,189924.4
+e1825d70c5844c22933087eafa89e39c,SuperGPQA,Engineering,pass,1683,447,1097,stop,819.7,13734.8
+aime2025-12,AIME2025,Algebra,pass,37595,1223,14305,stop,499.8,202018.7
+reckEnrOPFT9Ru7tW,GPQA Diamond,Chemistry,pass,46333,684,14369,stop,681.4,207876.5
+ab430ac3f18e4e02a2cb3f35498c6b30,SuperGPQA,Engineering,pass,8732,976,3547,stop,680.7,49403.6
+aime2025-27,AIME2025,Geometry,pass,21496,1213,9093,stop,602.1,125949.7
+rec8nshandHARTkrg,GPQA Diamond,Chemistry,pass,48381,513,15188,stop,817.3,215610.2
+e8c5da5ca40647158b0c91dd695c6243,SuperGPQA,Science,pass,3709,412,1497,stop,856.6,20659.4
+aime2025-13,AIME2025,Combinatorics,pass,83457,2227,23105,stop,505.3,343957.8
+recFaL6j8UMhutXrc,GPQA Diamond,Chemistry,pass,31485,9,10543,stop,836.9,145068.5
+05efdc6fb2404ddc8dd5729bb68c74e5,SuperGPQA,Science,pass,1877,597,1423,stop,626.0,17509.7
+aime2025-28,AIME2025,Number Theory,pass,57716,2191,31186,stop,497.4,422570.1
+reczQ4I0VpENdMtIj,GPQA Diamond,Biology,pass,34384,240,7340,stop,804.1,109156.9
+ba52e06cbe1a4310a77a7d12cd1db943,SuperGPQA,Science,pass,9432,328,3299,stop,815.1,44636.8
+aime2025-29,AIME2025,Geometry,pass,33002,2672,14345,stop,555.5,195123.1
+recWxGU8Q4YReJ1tb,GPQA Diamond,Chemistry,exhausted,98549,0,32768,length,994.9,483260.7
+591a77df21324272914be82ac6583399,SuperGPQA,Science,fail,1139,353,612,stop,952.0,8220.1
+aime2025-15,AIME2025,Number Theory,exhausted,75947,0,32768,length,481.9,478804.8
+recMicVBcqy1xM1jq,GPQA Diamond,Physics,pass,50090,652,17445,stop,825.3,249383.7
+e780f37a5baa4fe094cd9c157486664d,SuperGPQA,Sociology,fail,9626,311,2310,stop,645.2,34629.9
+aime2025-30,AIME2025,Algebra,pass,61789,1850,25912,stop,509.7,371581.9
+compsec-076,COMPSEC,FreeBSD / librpcsec_gss,pass,15728,10,3922,stop,872.2,59311.5
+compsec-077,COMPSEC,GNU Inetutils telnetd,exhausted,47409,0,32768,length,637.1,357547.7
+compsec-078,COMPSEC,Botan,pass,1698,10,473,stop,923.5,7329.8
+compsec-079,COMPSEC,Botan,pass,8512,10,1946,stop,655.9,30315.2
+compsec-080,COMPSEC,Botan,pass,14966,9,3413,stop,690.8,54112.5
+compsec-081,COMPSEC,uds-c,pass,16205,10,3935,stop,741.1,59342.3
+compsec-082,COMPSEC,Firebird,pass,50188,12,12641,stop,617.6,196910.4
+compsec-083,COMPSEC,Firebird,fail,35894,0,11531,stop,694.2,142400.1
+compsec-084,COMPSEC,PHP,pass,2890,9,787,stop,725.9,11617.0
+compsec-085,COMPSEC,PHP PDO Firebird,pass,2518,9,688,stop,500.9,11093.2
+compsec-086,COMPSEC,libexpat,pass,5944,9,1494,stop,614.8,23278.3
+compsec-087,COMPSEC,Mbed TLS,pass,23030,9,6682,stop,675.0,94928.8
+compsec-088,COMPSEC,Mbed TLS,pass,4726,10,1191,stop,658.3,18610.2
+compsec-089,COMPSEC,Mbed TLS,pass,2721,10,725,stop,579.2,11213.1
+compsec-090,COMPSEC,Linux kernel,pass,4129,10,1188,stop,605.8,16941.8
+compsec-091,COMPSEC,Linux kernel USB gadget storage,pass,1508,9,401,stop,523.2,6244.0
+compsec-092,COMPSEC,Linux kernel AppArmor,pass,9825,10,2410,stop,591.1,35765.1

bench/fit-128k/qa-eval-1786818550.json

@@ -0,0 +1,1028 @@
+{
+  "accuracy": 0.8804347826086957,
+  "bench_version": "0.1.0",
+  "cases": 92,
+  "cases_detail": [
+    {
+      "answer_chars": 392,
+      "completion_tokens": 7047,
+      "finish_reason": "stop",
+      "id": "recNu3MXkvWUzHZr9",
+      "reasoning_chars": 17606,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 90297.17975,
+      "ttft_ms": 400.517583
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 558,
+      "finish_reason": "stop",
+      "id": "001b51d76b4d422988f2c11f104a2c6c",
+      "reasoning_chars": 2328,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 7497.739208,
+      "ttft_ms": 496.0825
+    },
+    {
+      "answer_chars": 418,
+      "completion_tokens": 980,
+      "finish_reason": "stop",
+      "id": "aime2025-01",
+      "reasoning_chars": 1843,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 11925.285625,
+      "ttft_ms": 446.104125
+    },
+    {
+      "answer_chars": 465,
+      "completion_tokens": 1886,
+      "finish_reason": "stop",
+      "id": "recoiTJPGUmzAkief",
+      "reasoning_chars": 8151,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 30781.231209,
+      "ttft_ms": 571.118792
+    },
+    {
+      "answer_chars": 450,
+      "completion_tokens": 4382,
+      "finish_reason": "stop",
+      "id": "b7e20eac98764fb0bf30e8366d951daa",
+      "reasoning_chars": 17331,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 58830.3535,
+      "ttft_ms": 670.6246249999999
+    },
+    {
+      "answer_chars": 768,
+      "completion_tokens": 1491,
+      "finish_reason": "stop",
+      "id": "aime2025-16",
+      "reasoning_chars": 2940,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 19670.371457999998,
+      "ttft_ms": 571.234208
+    },
+    {
+      "answer_chars": 585,
+      "completion_tokens": 1230,
+      "finish_reason": "stop",
+      "id": "rec4UqStf9WUVif1f",
+      "reasoning_chars": 1944,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 15139.857750000001,
+      "ttft_ms": 556.6582500000001
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1665,
+      "finish_reason": "stop",
+      "id": "4a1d1780a93f4093b6fb7d3c314cbea8",
+      "reasoning_chars": 7910,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 25679.061291,
+      "ttft_ms": 682.954708
+    },
+    {
+      "answer_chars": 1406,
+      "completion_tokens": 11789,
+      "finish_reason": "stop",
+      "id": "aime2025-02",
+      "reasoning_chars": 28249,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 159785.437041,
+      "ttft_ms": 1028.397875
+    },
+    {
+      "answer_chars": 663,
+      "completion_tokens": 2295,
+      "finish_reason": "stop",
+      "id": "recgI6tUQ7RLJRWGx",
+      "reasoning_chars": 5374,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 32177.827166,
+      "ttft_ms": 716.48325
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 490,
+      "finish_reason": "stop",
+      "id": "6082513c8dba4ec68aa68f1bf5854d09",
+      "reasoning_chars": 2420,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 7058.177917,
+      "ttft_ms": 520.721083
+    },
+    {
+      "answer_chars": 423,
+      "completion_tokens": 1006,
+      "finish_reason": "stop",
+      "id": "aime2025-03",
+      "reasoning_chars": 2046,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 13059.566291,
+      "ttft_ms": 559.802
+    },
+    {
+      "answer_chars": 415,
+      "completion_tokens": 4276,
+      "finish_reason": "stop",
+      "id": "recDytVnNYZe2HuUU",
+      "reasoning_chars": 9308,
+      "result": "pass",
+      "source": "GPQA Diamond (modified)",
+      "total_ms": 58811.169417000005,
+      "ttft_ms": 545.814125
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 273,
+      "finish_reason": "stop",
+      "id": "bebf1ed45ae14ad7b4f205f3909cb58a",
+      "reasoning_chars": 1146,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 4326.139,
+      "ttft_ms": 662.489667
+    },
+    {
+      "answer_chars": 1428,
+      "completion_tokens": 6884,
+      "finish_reason": "stop",
+      "id": "aime2025-18",
+      "reasoning_chars": 18335,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 95518.73208300001,
+      "ttft_ms": 665.6728750000001
+    },
+    {
+      "answer_chars": 438,
+      "completion_tokens": 838,
+      "finish_reason": "stop",
+      "id": "recNFJjE5PPTqVJGv",
+      "reasoning_chars": 1387,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 11266.735999999999,
+      "ttft_ms": 820.827583
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 521,
+      "finish_reason": "stop",
+      "id": "7ca71b86327744b78e93185a45bc5cef",
+      "reasoning_chars": 2124,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 7313.874292,
+      "ttft_ms": 556.275667
+    },
+    {
+      "answer_chars": 457,
+      "completion_tokens": 1401,
+      "finish_reason": "stop",
+      "id": "aime2025-04",
+      "reasoning_chars": 2730,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 18458.330916000003,
+      "ttft_ms": 495.294666
+    },
+    {
+      "answer_chars": 448,
+      "completion_tokens": 872,
+      "finish_reason": "stop",
+      "id": "rec2UlKqC6RFHdcro",
+      "reasoning_chars": 2023,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 12094.629792,
+      "ttft_ms": 687.8919999999999
+    },
+    {
+      "answer_chars": 323,
+      "completion_tokens": 2246,
+      "finish_reason": "stop",
+      "id": "d44b94f7749345a39a65f6312bda8764",
+      "reasoning_chars": 8254,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 35600.282417,
+      "ttft_ms": 558.460625
+    },
+    {
+      "answer_chars": 1024,
+      "completion_tokens": 1263,
+      "finish_reason": "stop",
+      "id": "aime2025-19",
+      "reasoning_chars": 1693,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 15963.453708,
+      "ttft_ms": 682.380958
+    },
+    {
+      "answer_chars": 420,
+      "completion_tokens": 3676,
+      "finish_reason": "stop",
+      "id": "recv7GsQg3f0fvB1f",
+      "reasoning_chars": 15177,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 55520.344792,
+      "ttft_ms": 568.395125
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 6506,
+      "finish_reason": "stop",
+      "id": "febe406f44d74a40b50bb5b7c69d5dc1",
+      "reasoning_chars": 28536,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 88599.846583,
+      "ttft_ms": 612.65
+    },
+    {
+      "answer_chars": 1370,
+      "completion_tokens": 5418,
+      "finish_reason": "stop",
+      "id": "aime2025-05",
+      "reasoning_chars": 11415,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 70547.959208,
+      "ttft_ms": 634.0662500000001
+    },
+    {
+      "answer_chars": 774,
+      "completion_tokens": 22432,
+      "finish_reason": "stop",
+      "id": "recrHBEJJoDTV05JR",
+      "reasoning_chars": 65159,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 310700.26125,
+      "ttft_ms": 635.402917
+    },
+    {
+      "answer_chars": 369,
+      "completion_tokens": 560,
+      "finish_reason": "stop",
+      "id": "31950dc80ded400a9181f50626d1f75c",
+      "reasoning_chars": 971,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 7480.801041000001,
+      "ttft_ms": 680.802
+    },
+    {
+      "answer_chars": 709,
+      "completion_tokens": 1863,
+      "finish_reason": "stop",
+      "id": "aime2025-06",
+      "reasoning_chars": 4189,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 25779.481874999998,
+      "ttft_ms": 464.28324999999995
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recb80OwMgNnceA9t",
+      "reasoning_chars": 95076,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 488278.307667,
+      "ttft_ms": 504.836375
+    },
+    {
+      "answer_chars": 954,
+      "completion_tokens": 3721,
+      "finish_reason": "stop",
+      "id": "0f14cd17be174618af6d60227e7dca9f",
+      "reasoning_chars": 9209,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 52322.547583,
+      "ttft_ms": 1171.4924170000002
+    },
+    {
+      "answer_chars": 1376,
+      "completion_tokens": 6357,
+      "finish_reason": "stop",
+      "id": "aime2025-21",
+      "reasoning_chars": 17225,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 87763.420208,
+      "ttft_ms": 896.7724999999999
+    },
+    {
+      "answer_chars": 884,
+      "completion_tokens": 17601,
+      "finish_reason": "stop",
+      "id": "recA1i5ZAh0Uzclxp",
+      "reasoning_chars": 55590,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 261851.24550000002,
+      "ttft_ms": 599.2252500000001
+    },
+    {
+      "answer_chars": 589,
+      "completion_tokens": 805,
+      "finish_reason": "stop",
+      "id": "cef9bcc087434cc2b4e354a9baef55eb",
+      "reasoning_chars": 813,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 9923.589708,
+      "ttft_ms": 790.676916
+    },
+    {
+      "answer_chars": 1913,
+      "completion_tokens": 6650,
+      "finish_reason": "stop",
+      "id": "aime2025-07",
+      "reasoning_chars": 20659,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 99698.61370799999,
+      "ttft_ms": 507.58579199999997
+    },
+    {
+      "answer_chars": 729,
+      "completion_tokens": 7057,
+      "finish_reason": "stop",
+      "id": "recqGD3fxPCI59vPQ",
+      "reasoning_chars": 21916,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 104354.741792,
+      "ttft_ms": 559.123875
+    },
+    {
+      "answer_chars": 147,
+      "completion_tokens": 314,
+      "finish_reason": "stop",
+      "id": "9f93aa2cfdb547b5b3a4623f80f7fff6",
+      "reasoning_chars": 1064,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 4691.068625,
+      "ttft_ms": 564.9937080000001
+    },
+    {
+      "answer_chars": 899,
+      "completion_tokens": 2719,
+      "finish_reason": "stop",
+      "id": "aime2025-22",
+      "reasoning_chars": 6000,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 35268.769709,
+      "ttft_ms": 468.462125
+    },
+    {
+      "answer_chars": 522,
+      "completion_tokens": 24594,
+      "finish_reason": "stop",
+      "id": "rechKl68Uc6H7vU0N",
+      "reasoning_chars": 74392,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 349015.086375,
+      "ttft_ms": 689.960416
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 801,
+      "finish_reason": "stop",
+      "id": "97ad69dda7b2462c98638b79e78aea0b",
+      "reasoning_chars": 3302,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 12331.464625,
+      "ttft_ms": 714.2458750000001
+    },
+    {
+      "answer_chars": 885,
+      "completion_tokens": 2167,
+      "finish_reason": "stop",
+      "id": "aime2025-08",
+      "reasoning_chars": 4135,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 28137.832167,
+      "ttft_ms": 495.911292
+    },
+    {
+      "answer_chars": 364,
+      "completion_tokens": 803,
+      "finish_reason": "stop",
+      "id": "rec1zl5LvaatzGhFt",
+      "reasoning_chars": 2097,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 11301.261959,
+      "ttft_ms": 645.257125
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1486,
+      "finish_reason": "stop",
+      "id": "e78e4e539d6f4e379ac140d923d7b1be",
+      "reasoning_chars": 7256,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 21903.941458999998,
+      "ttft_ms": 471.018459
+    },
+    {
+      "answer_chars": 1167,
+      "completion_tokens": 7290,
+      "finish_reason": "stop",
+      "id": "aime2025-09",
+      "reasoning_chars": 14188,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 93998.935917,
+      "ttft_ms": 480.561459
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 9115,
+      "finish_reason": "stop",
+      "id": "recTs7qzfJs6kfLUK",
+      "reasoning_chars": 27479,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 128131.84391699999,
+      "ttft_ms": 837.1487079999999
+    },
+    {
+      "answer_chars": 864,
+      "completion_tokens": 24825,
+      "finish_reason": "stop",
+      "id": "8483667a25e74fdfa3188de4ea734f03",
+      "reasoning_chars": 83439,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 360496.513042,
+      "ttft_ms": 569.386459
+    },
+    {
+      "answer_chars": 1223,
+      "completion_tokens": 3775,
+      "finish_reason": "stop",
+      "id": "aime2025-24",
+      "reasoning_chars": 8589,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 52751.235875,
+      "ttft_ms": 703.242708
+    },
+    {
+      "answer_chars": 381,
+      "completion_tokens": 1119,
+      "finish_reason": "stop",
+      "id": "rec32C1ZEapBnCC0E",
+      "reasoning_chars": 2810,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 16393.146417,
+      "ttft_ms": 539.2782090000001
+    },
+    {
+      "answer_chars": 439,
+      "completion_tokens": 812,
+      "finish_reason": "stop",
+      "id": "e5ed76ef98144f06843125daf1bccd35",
+      "reasoning_chars": 1245,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 10355.39925,
+      "ttft_ms": 512.2696249999999
+    },
+    {
+      "answer_chars": 2560,
+      "completion_tokens": 19152,
+      "finish_reason": "stop",
+      "id": "aime2025-10",
+      "reasoning_chars": 57364,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 275600.60375,
+      "ttft_ms": 697.710667
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recZWeueB7lSPR6wN",
+      "reasoning_chars": 103323,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 494772.644667,
+      "ttft_ms": 665.129917
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 2044,
+      "finish_reason": "stop",
+      "id": "fd7924876c4845cd83d95d61dfa0b236",
+      "reasoning_chars": 10320,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 30335.493541999997,
+      "ttft_ms": 750.047209
+    },
+    {
+      "answer_chars": 869,
+      "completion_tokens": 5090,
+      "finish_reason": "stop",
+      "id": "aime2025-25",
+      "reasoning_chars": 15604,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 74222.344125,
+      "ttft_ms": 466.266167
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recVvpD8miVjmmyfe",
+      "reasoning_chars": 70673,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 473515.62845799996,
+      "ttft_ms": 504.94404199999997
+    },
+    {
+      "answer_chars": 461,
+      "completion_tokens": 16689,
+      "finish_reason": "stop",
+      "id": "6bfe7d19299d4b3184636e1f51694306",
+      "reasoning_chars": 52354,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 234227.736792,
+      "ttft_ms": 1000.100333
+    },
+    {
+      "answer_chars": 1335,
+      "completion_tokens": 8724,
+      "finish_reason": "stop",
+      "id": "aime2025-26",
+      "reasoning_chars": 26649,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 128327.34074999999,
+      "ttft_ms": 604.3037089999999
+    },
+    {
+      "answer_chars": 825,
+      "completion_tokens": 13841,
+      "finish_reason": "stop",
+      "id": "recAAJoHMW45Lv5je",
+      "reasoning_chars": 43395,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 189924.378667,
+      "ttft_ms": 588.7228749999999
+    },
+    {
+      "answer_chars": 447,
+      "completion_tokens": 1097,
+      "finish_reason": "stop",
+      "id": "e1825d70c5844c22933087eafa89e39c",
+      "reasoning_chars": 1683,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 13734.76525,
+      "ttft_ms": 819.6647919999999
+    },
+    {
+      "answer_chars": 1223,
+      "completion_tokens": 14305,
+      "finish_reason": "stop",
+      "id": "aime2025-12",
+      "reasoning_chars": 37595,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 202018.749792,
+      "ttft_ms": 499.845667
+    },
+    {
+      "answer_chars": 684,
+      "completion_tokens": 14369,
+      "finish_reason": "stop",
+      "id": "reckEnrOPFT9Ru7tW",
+      "reasoning_chars": 46333,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 207876.502375,
+      "ttft_ms": 681.3592080000001
+    },
+    {
+      "answer_chars": 976,
+      "completion_tokens": 3547,
+      "finish_reason": "stop",
+      "id": "ab430ac3f18e4e02a2cb3f35498c6b30",
+      "reasoning_chars": 8732,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 49403.643709,
+      "ttft_ms": 680.7177089999999
+    },
+    {
+      "answer_chars": 1213,
+      "completion_tokens": 9093,
+      "finish_reason": "stop",
+      "id": "aime2025-27",
+      "reasoning_chars": 21496,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 125949.73950000001,
+      "ttft_ms": 602.090834
+    },
+    {
+      "answer_chars": 513,
+      "completion_tokens": 15188,
+      "finish_reason": "stop",
+      "id": "rec8nshandHARTkrg",
+      "reasoning_chars": 48381,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 215610.182875,
+      "ttft_ms": 817.330916
+    },
+    {
+      "answer_chars": 412,
+      "completion_tokens": 1497,
+      "finish_reason": "stop",
+      "id": "e8c5da5ca40647158b0c91dd695c6243",
+      "reasoning_chars": 3709,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 20659.387875,
+      "ttft_ms": 856.551041
+    },
+    {
+      "answer_chars": 2227,
+      "completion_tokens": 23105,
+      "finish_reason": "stop",
+      "id": "aime2025-13",
+      "reasoning_chars": 83457,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 343957.752417,
+      "ttft_ms": 505.31475000000006
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 10543,
+      "finish_reason": "stop",
+      "id": "recFaL6j8UMhutXrc",
+      "reasoning_chars": 31485,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 145068.49570899998,
+      "ttft_ms": 836.8546249999999
+    },
+    {
+      "answer_chars": 597,
+      "completion_tokens": 1423,
+      "finish_reason": "stop",
+      "id": "05efdc6fb2404ddc8dd5729bb68c74e5",
+      "reasoning_chars": 1877,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 17509.740167,
+      "ttft_ms": 626.016125
+    },
+    {
+      "answer_chars": 2191,
+      "completion_tokens": 31186,
+      "finish_reason": "stop",
+      "id": "aime2025-28",
+      "reasoning_chars": 57716,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 422570.07908399997,
+      "ttft_ms": 497.430084
+    },
+    {
+      "answer_chars": 240,
+      "completion_tokens": 7340,
+      "finish_reason": "stop",
+      "id": "reczQ4I0VpENdMtIj",
+      "reasoning_chars": 34384,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 109156.876083,
+      "ttft_ms": 804.101958
+    },
+    {
+      "answer_chars": 328,
+      "completion_tokens": 3299,
+      "finish_reason": "stop",
+      "id": "ba52e06cbe1a4310a77a7d12cd1db943",
+      "reasoning_chars": 9432,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 44636.754792,
+      "ttft_ms": 815.143458
+    },
+    {
+      "answer_chars": 2672,
+      "completion_tokens": 14345,
+      "finish_reason": "stop",
+      "id": "aime2025-29",
+      "reasoning_chars": 33002,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 195123.096583,
+      "ttft_ms": 555.538291
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recWxGU8Q4YReJ1tb",
+      "reasoning_chars": 98549,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 483260.69125,
+      "ttft_ms": 994.8692500000001
+    },
+    {
+      "answer_chars": 353,
+      "completion_tokens": 612,
+      "finish_reason": "stop",
+      "id": "591a77df21324272914be82ac6583399",
+      "reasoning_chars": 1139,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 8220.103834,
+      "ttft_ms": 952.042292
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-15",
+      "reasoning_chars": 75947,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 478804.75070900004,
+      "ttft_ms": 481.89016699999996
+    },
+    {
+      "answer_chars": 652,
+      "completion_tokens": 17445,
+      "finish_reason": "stop",
+      "id": "recMicVBcqy1xM1jq",
+      "reasoning_chars": 50090,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 249383.664958,
+      "ttft_ms": 825.2723749999999
+    },
+    {
+      "answer_chars": 311,
+      "completion_tokens": 2310,
+      "finish_reason": "stop",
+      "id": "e780f37a5baa4fe094cd9c157486664d",
+      "reasoning_chars": 9626,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 34629.941208,
+      "ttft_ms": 645.245625
+    },
+    {
+      "answer_chars": 1850,
+      "completion_tokens": 25912,
+      "finish_reason": "stop",
+      "id": "aime2025-30",
+      "reasoning_chars": 61789,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 371581.92183300003,
+      "ttft_ms": 509.720833
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 3922,
+      "finish_reason": "stop",
+      "id": "compsec-076",
+      "reasoning_chars": 15728,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 59311.450625,
+      "ttft_ms": 872.1859589999999
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "compsec-077",
+      "reasoning_chars": 47409,
+      "result": "exhausted",
+      "source": "COMPSEC",
+      "total_ms": 357547.668333,
+      "ttft_ms": 637.084917
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 473,
+      "finish_reason": "stop",
+      "id": "compsec-078",
+      "reasoning_chars": 1698,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 7329.838875,
+      "ttft_ms": 923.479834
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1946,
+      "finish_reason": "stop",
+      "id": "compsec-079",
+      "reasoning_chars": 8512,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 30315.196708,
+      "ttft_ms": 655.864875
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 3413,
+      "finish_reason": "stop",
+      "id": "compsec-080",
+      "reasoning_chars": 14966,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 54112.527458000004,
+      "ttft_ms": 690.819583
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 3935,
+      "finish_reason": "stop",
+      "id": "compsec-081",
+      "reasoning_chars": 16205,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 59342.275667,
+      "ttft_ms": 741.109458
+    },
+    {
+      "answer_chars": 12,
+      "completion_tokens": 12641,
+      "finish_reason": "stop",
+      "id": "compsec-082",
+      "reasoning_chars": 50188,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 196910.41691600002,
+      "ttft_ms": 617.597041
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 11531,
+      "finish_reason": "stop",
+      "id": "compsec-083",
+      "reasoning_chars": 35894,
+      "result": "fail",
+      "source": "COMPSEC",
+      "total_ms": 142400.100042,
+      "ttft_ms": 694.2423749999999
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 787,
+      "finish_reason": "stop",
+      "id": "compsec-084",
+      "reasoning_chars": 2890,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 11617.049165999999,
+      "ttft_ms": 725.943416
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 688,
+      "finish_reason": "stop",
+      "id": "compsec-085",
+      "reasoning_chars": 2518,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 11093.219208,
+      "ttft_ms": 500.896333
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1494,
+      "finish_reason": "stop",
+      "id": "compsec-086",
+      "reasoning_chars": 5944,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 23278.292959000002,
+      "ttft_ms": 614.809959
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 6682,
+      "finish_reason": "stop",
+      "id": "compsec-087",
+      "reasoning_chars": 23030,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 94928.751083,
+      "ttft_ms": 675.014458
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1191,
+      "finish_reason": "stop",
+      "id": "compsec-088",
+      "reasoning_chars": 4726,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 18610.182584000002,
+      "ttft_ms": 658.259
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 725,
+      "finish_reason": "stop",
+      "id": "compsec-089",
+      "reasoning_chars": 2721,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 11213.137959000002,
+      "ttft_ms": 579.160875
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1188,
+      "finish_reason": "stop",
+      "id": "compsec-090",
+      "reasoning_chars": 4129,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 16941.763667,
+      "ttft_ms": 605.751042
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 401,
+      "finish_reason": "stop",
+      "id": "compsec-091",
+      "reasoning_chars": 1508,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 6244.0455,
+      "ttft_ms": 523.230333
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 2410,
+      "finish_reason": "stop",
+      "id": "compsec-092",
+      "reasoning_chars": 9825,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 35765.083834000005,
+      "ttft_ms": 591.068959
+    }
+  ],
+  "config": {
+    "max_tokens": 32768,
+    "seed": 42,
+    "temperature": 0.0
+  },
+  "exhausted": 6,
+  "failed_answered": 5,
+  "model": "fit-128k",
+  "passed": 81
+}

bench/fit-17g/config-public.txt

@@ -0,0 +1,47 @@
+run:        fit-17g
+date:       2026-08-16T08:46:00+02:00
+driver:     bench-baselines-mtp.sh @ 016ed89
+bs4:        bs4 (our eval harness; not public)
+serve_cmd:  ~/llama.cpp/build-mtp/bin/llama-server -m ~/models/qwen38-27b-fit-256k-17g.gguf -md ~/models/Qwen3.8-27B-MTP-ONLY-Q8_0.gguf --spec-draft-n-max 3 -c 131072 -ctk q4_0 -ctv q4_0 -fa on -ngl 99 -ngld 99 --port 8083 --host 0.0.0.0
+--- desktop side ---
+llama.cpp:  22b8e31 (22b8e31)
+model:      9832877184 bytes  ~/models/qwen38-27b-fit-256k-17g.gguf
+draft:      4503398688 bytes
+gpu:        NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 250.00 W, 97887 MiB
+driver_ver: 610.57.04
+
+sampler:    temperature 0.0, seed 42, max_tokens 32768 (sent by the eval client in each request body; source: qa-eval-1786874555.json)
+
+--- server /props at run start ---
+{
+  "_note": "SERVER DEFAULTS ONLY — these are what llama-server starts with, NOT what the eval used. Every request overrides them per the 'sampler:' line above (temperature 0, fixed seed). model_ftype is llama.cpp's coarse single-label for the file; for a per-tensor mix it does not describe the quantization.",
+  "build_info": "b93-22b8e31",
+  "model_ftype": "IQ4_XS - 4.25 bpw",
+  "n_ctx": 131072,
+  "total_slots": 4,
+  "sampler_defaults": {
+    "temperature": 1.0,
+    "top_k": 20,
+    "top_p": 0.949999988079071,
+    "min_p": 0.05000000074505806,
+    "repeat_penalty": 1.0,
+    "samplers": [
+      "penalties",
+      "dry",
+      "top_n_sigma",
+      "top_k",
+      "typ_p",
+      "top_p",
+      "min_p",
+      "xtc",
+      "temperature"
+    ]
+  },
+  "chat_template": "<omitted: ~30KB jinja, no reproduction value>"
+}
+
+--- publication note ---
+Scrubbed for publication by sanitize-config.py: hostnames, internal
+repo paths and home paths removed, GPU index omitted (immaterial —
+one card, one power cap), chat template omitted for size. Everything
+that affects the measurement is kept verbatim.

bench/fit-17g/qa-eval-1786874555.csv

@@ -0,0 +1,93 @@
+id,source,domain,result,reasoning_chars,answer_chars,completion_tokens,finish_reason,ttft_ms,total_ms
+recNu3MXkvWUzHZr9,GPQA Diamond,Physics,fail,62223,541,24489,stop,360.3,247535.7
+001b51d76b4d422988f2c11f104a2c6c,SuperGPQA,Agronomy,pass,1813,9,426,stop,638.4,4853.1
+aime2025-01,AIME2025,Number Theory,pass,1640,340,848,stop,411.4,8227.2
+recoiTJPGUmzAkief,GPQA Diamond,Physics,pass,68013,592,15752,stop,527.5,171202.6
+b7e20eac98764fb0bf30e8366d951daa,SuperGPQA,Engineering,pass,21514,396,5113,stop,619.1,52793.8
+aime2025-16,AIME2025,Geometry,pass,3981,664,2088,stop,494.6,19982.0
+rec4UqStf9WUVif1f,GPQA Diamond,Physics,pass,3355,839,1887,stop,467.6,17703.4
+4a1d1780a93f4093b6fb7d3c314cbea8,SuperGPQA,Engineering,pass,15920,166,3468,stop,476.0,37815.0
+aime2025-02,AIME2025,Geometry,exhausted,82120,0,32768,length,761.1,343448.5
+recgI6tUQ7RLJRWGx,GPQA Diamond,Physics,pass,4767,680,1961,stop,787.8,20816.6
+6082513c8dba4ec68aa68f1bf5854d09,SuperGPQA,Law,fail,1333,299,344,stop,464.5,3479.7
+aime2025-03,AIME2025,Combinatorics,pass,9449,503,3818,stop,469.4,37285.5
+recDytVnNYZe2HuUU,GPQA Diamond (modified),Physics,pass,19476,442,8476,stop,467.1,85230.2
+bebf1ed45ae14ad7b4f205f3909cb58a,SuperGPQA,Literature and Arts,pass,1333,9,356,stop,520.0,4005.9
+aime2025-18,AIME2025,Combinatorics,pass,37955,1444,13467,stop,520.4,141875.6
+recNFJjE5PPTqVJGv,GPQA Diamond,Chemistry,pass,2116,488,1148,stop,559.7,11043.2
+7ca71b86327744b78e93185a45bc5cef,SuperGPQA,Medicine,fail,4857,9,1184,stop,442.1,12834.2
+aime2025-04,AIME2025,Algebra,pass,9372,453,4372,stop,439.9,42976.3
+rec2UlKqC6RFHdcro,GPQA Diamond,Physics,pass,18313,734,7509,stop,517.8,72992.9
+d44b94f7749345a39a65f6312bda8764,SuperGPQA,Engineering,pass,41556,385,11986,stop,567.7,132997.5
+aime2025-19,AIME2025,Algebra,pass,10038,1182,5797,stop,565.0,53367.4
+recv7GsQg3f0fvB1f,GPQA Diamond,Chemistry,pass,8434,371,1972,stop,526.5,21371.3
+febe406f44d74a40b50bb5b7c69d5dc1,SuperGPQA,Sociology,pass,8754,9,1950,stop,502.2,19193.1
+aime2025-05,AIME2025,"Combinatorics, Number Theory",pass,23646,1370,10999,stop,467.2,106966.0
+recrHBEJJoDTV05JR,GPQA Diamond,Chemistry,pass,39834,499,11540,stop,563.2,118407.3
+31950dc80ded400a9181f50626d1f75c,SuperGPQA,Science,pass,936,414,622,stop,594.5,6026.1
+aime2025-06,AIME2025,Geometry,pass,13027,778,5023,stop,420.4,51276.9
+recb80OwMgNnceA9t,GPQA Diamond,Chemistry,exhausted,98010,0,32768,length,494.0,332948.0
+0f14cd17be174618af6d60227e7dca9f,SuperGPQA,Engineering,pass,17181,527,6204,stop,1173.0,64284.7
+aime2025-21,AIME2025,"Algebra, Geometry",pass,6380,924,2962,stop,847.8,27891.4
+recA1i5ZAh0Uzclxp,GPQA Diamond,Chemistry,exhausted,106783,0,32768,length,591.9,360413.7
+cef9bcc087434cc2b4e354a9baef55eb,SuperGPQA,Engineering,pass,953,543,897,stop,913.6,8896.9
+aime2025-07,AIME2025,Combinatorics,pass,45715,1288,14365,stop,522.0,158779.9
+recqGD3fxPCI59vPQ,GPQA Diamond,Physics,pass,25730,591,8577,stop,590.9,90109.9
+9f93aa2cfdb547b5b3a4623f80f7fff6,SuperGPQA,Engineering,pass,1373,9,353,stop,518.9,4329.3
+aime2025-22,AIME2025,Number Theory,pass,10865,1035,4138,stop,510.2,42452.4
+rechKl68Uc6H7vU0N,GPQA Diamond,Chemistry,exhausted,97944,0,32768,length,584.3,347741.9
+97ad69dda7b2462c98638b79e78aea0b,SuperGPQA,Medicine,pass,22720,194,5546,stop,807.1,56474.9
+aime2025-08,AIME2025,Algebra,pass,13053,814,6724,stop,500.9,64382.0
+rec1zl5LvaatzGhFt,GPQA Diamond,Physics,pass,16426,608,5824,stop,620.4,58807.3
+e78e4e539d6f4e379ac140d923d7b1be,SuperGPQA,Military Science,pass,7362,220,1525,stop,494.3,16532.4
+aime2025-09,AIME2025,Algebra,exhausted,65782,0,32768,length,489.6,337796.7
+recTs7qzfJs6kfLUK,GPQA Diamond,Biology,exhausted,76995,0,32768,length,1766.6,301419.2
+8483667a25e74fdfa3188de4ea734f03,SuperGPQA,Science,fail,109574,203,30124,stop,761.4,315457.4
+aime2025-24,AIME2025,Algebra,pass,27527,1218,11456,stop,827.9,118657.9
+rec32C1ZEapBnCC0E,GPQA Diamond,Physics,pass,14949,690,5784,stop,643.3,60097.4
+e5ed76ef98144f06843125daf1bccd35,SuperGPQA,Science,pass,2051,9,912,stop,518.8,9200.9
+aime2025-10,AIME2025,Combinatorics,pass,67823,2204,24923,stop,583.0,259420.7
+recZWeueB7lSPR6wN,GPQA Diamond,Chemistry,exhausted,104167,0,32768,length,825.8,292873.6
+fd7924876c4845cd83d95d61dfa0b236,SuperGPQA,Sociology,fail,14000,102,2906,stop,761.2,30662.9
+aime2025-25,AIME2025,Combinatorics,exhausted,98598,0,32768,length,447.4,297958.6
+recVvpD8miVjmmyfe,GPQA Diamond,Biology,exhausted,75136,0,32768,length,755.8,309495.4
+6bfe7d19299d4b3184636e1f51694306,SuperGPQA,Agronomy,exhausted,101240,0,32768,length,876.0,311542.4
+aime2025-26,AIME2025,Combinatorics,exhausted,118446,0,32768,length,942.4,305537.4
+recAAJoHMW45Lv5je,GPQA Diamond,Chemistry,fail,77881,0,20513,stop,815.9,164054.8
+e1825d70c5844c22933087eafa89e39c,SuperGPQA,Engineering,pass,2259,408,1429,stop,862.3,13141.4
+aime2025-12,AIME2025,Algebra,pass,71388,1278,30274,stop,443.0,312026.7
+reckEnrOPFT9Ru7tW,GPQA Diamond,Chemistry,exhausted,105991,0,32768,length,795.1,333209.5
+ab430ac3f18e4e02a2cb3f35498c6b30,SuperGPQA,Engineering,pass,46633,1092,17871,stop,845.5,183831.3
+aime2025-27,AIME2025,Geometry,pass,39131,1244,16707,stop,699.1,165268.7
+rec8nshandHARTkrg,GPQA Diamond,Chemistry,exhausted,99198,0,32768,length,892.2,333792.6
+e8c5da5ca40647158b0c91dd695c6243,SuperGPQA,Science,pass,6529,548,2830,stop,1104.6,27616.3
+aime2025-13,AIME2025,Combinatorics,exhausted,84876,0,32768,length,474.6,289234.3
+recFaL6j8UMhutXrc,GPQA Diamond,Chemistry,exhausted,99094,0,32768,length,868.1,337121.2
+05efdc6fb2404ddc8dd5729bb68c74e5,SuperGPQA,Science,pass,4361,704,2472,stop,772.8,23082.8
+aime2025-28,AIME2025,Number Theory,exhausted,42341,0,32768,length,510.4,270782.2
+reczQ4I0VpENdMtIj,GPQA Diamond,Biology,pass,44165,326,9179,stop,759.5,93602.7
+ba52e06cbe1a4310a77a7d12cd1db943,SuperGPQA,Science,exhausted,86169,0,32768,length,918.4,321926.4
+aime2025-29,AIME2025,Geometry,exhausted,67348,0,32768,length,861.9,343280.4
+recWxGU8Q4YReJ1tb,GPQA Diamond,Chemistry,exhausted,95893,0,32768,length,1106.7,341060.4
+591a77df21324272914be82ac6583399,SuperGPQA,Science,fail,2786,480,1467,stop,828.2,14707.0
+aime2025-15,AIME2025,Number Theory,exhausted,82170,0,32768,length,426.3,362740.4
+recMicVBcqy1xM1jq,GPQA Diamond,Physics,fail,65037,508,19524,stop,846.4,202671.6
+e780f37a5baa4fe094cd9c157486664d,SuperGPQA,Sociology,fail,4675,236,1091,stop,645.7,12669.0
+aime2025-30,AIME2025,Algebra,exhausted,66429,0,32768,length,471.7,333789.2
+compsec-076,COMPSEC,FreeBSD / librpcsec_gss,pass,32031,10,8424,stop,948.3,93214.1
+compsec-077,COMPSEC,GNU Inetutils telnetd,pass,60787,10,16570,stop,656.3,178285.6
+compsec-078,COMPSEC,Botan,pass,2680,10,751,stop,772.1,8499.4
+compsec-079,COMPSEC,Botan,pass,30412,13,7326,stop,624.2,79250.8
+compsec-080,COMPSEC,Botan,pass,15506,11,3509,stop,545.5,39996.0
+compsec-081,COMPSEC,uds-c,pass,19363,10,4887,stop,655.3,53585.4
+compsec-082,COMPSEC,Firebird,pass,15663,10,3773,stop,580.2,42494.8
+compsec-083,COMPSEC,Firebird,pass,43080,9,11041,stop,585.7,121134.5
+compsec-084,COMPSEC,PHP,pass,11754,9,3287,stop,648.5,36503.9
+compsec-085,COMPSEC,PHP PDO Firebird,pass,9270,9,2373,stop,593.2,25341.4
+compsec-086,COMPSEC,libexpat,pass,11126,9,2830,stop,609.0,30856.7
+compsec-087,COMPSEC,Mbed TLS,pass,26016,9,7550,stop,617.6,79131.2
+compsec-088,COMPSEC,Mbed TLS,pass,9399,10,2441,stop,629.3,27086.3
+compsec-089,COMPSEC,Mbed TLS,pass,2803,10,725,stop,501.2,9049.0
+compsec-090,COMPSEC,Linux kernel,pass,9961,10,2806,stop,616.6,30156.2
+compsec-091,COMPSEC,Linux kernel USB gadget storage,pass,1816,9,440,stop,488.6,5379.4
+compsec-092,COMPSEC,Linux kernel AppArmor,pass,4451,10,1077,stop,517.3,11252.3

bench/fit-17g/qa-eval-1786874555.json

@@ -0,0 +1,1028 @@
+{
+  "accuracy": 0.6739130434782609,
+  "bench_version": "0.1.0",
+  "cases": 92,
+  "cases_detail": [
+    {
+      "answer_chars": 541,
+      "completion_tokens": 24489,
+      "finish_reason": "stop",
+      "id": "recNu3MXkvWUzHZr9",
+      "reasoning_chars": 62223,
+      "result": "fail",
+      "source": "GPQA Diamond",
+      "total_ms": 247535.712083,
+      "ttft_ms": 360.256958
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 426,
+      "finish_reason": "stop",
+      "id": "001b51d76b4d422988f2c11f104a2c6c",
+      "reasoning_chars": 1813,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 4853.057792,
+      "ttft_ms": 638.4332499999999
+    },
+    {
+      "answer_chars": 340,
+      "completion_tokens": 848,
+      "finish_reason": "stop",
+      "id": "aime2025-01",
+      "reasoning_chars": 1640,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 8227.198792,
+      "ttft_ms": 411.400292
+    },
+    {
+      "answer_chars": 592,
+      "completion_tokens": 15752,
+      "finish_reason": "stop",
+      "id": "recoiTJPGUmzAkief",
+      "reasoning_chars": 68013,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 171202.647791,
+      "ttft_ms": 527.483416
+    },
+    {
+      "answer_chars": 396,
+      "completion_tokens": 5113,
+      "finish_reason": "stop",
+      "id": "b7e20eac98764fb0bf30e8366d951daa",
+      "reasoning_chars": 21514,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 52793.800666,
+      "ttft_ms": 619.079791
+    },
+    {
+      "answer_chars": 664,
+      "completion_tokens": 2088,
+      "finish_reason": "stop",
+      "id": "aime2025-16",
+      "reasoning_chars": 3981,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 19982.015333,
+      "ttft_ms": 494.560333
+    },
+    {
+      "answer_chars": 839,
+      "completion_tokens": 1887,
+      "finish_reason": "stop",
+      "id": "rec4UqStf9WUVif1f",
+      "reasoning_chars": 3355,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 17703.403792,
+      "ttft_ms": 467.594875
+    },
+    {
+      "answer_chars": 166,
+      "completion_tokens": 3468,
+      "finish_reason": "stop",
+      "id": "4a1d1780a93f4093b6fb7d3c314cbea8",
+      "reasoning_chars": 15920,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 37814.997042,
+      "ttft_ms": 475.972625
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-02",
+      "reasoning_chars": 82120,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 343448.50033400004,
+      "ttft_ms": 761.119334
+    },
+    {
+      "answer_chars": 680,
+      "completion_tokens": 1961,
+      "finish_reason": "stop",
+      "id": "recgI6tUQ7RLJRWGx",
+      "reasoning_chars": 4767,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 20816.606625,
+      "ttft_ms": 787.8325000000001
+    },
+    {
+      "answer_chars": 299,
+      "completion_tokens": 344,
+      "finish_reason": "stop",
+      "id": "6082513c8dba4ec68aa68f1bf5854d09",
+      "reasoning_chars": 1333,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 3479.676333,
+      "ttft_ms": 464.45804200000003
+    },
+    {
+      "answer_chars": 503,
+      "completion_tokens": 3818,
+      "finish_reason": "stop",
+      "id": "aime2025-03",
+      "reasoning_chars": 9449,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 37285.532583,
+      "ttft_ms": 469.38625
+    },
+    {
+      "answer_chars": 442,
+      "completion_tokens": 8476,
+      "finish_reason": "stop",
+      "id": "recDytVnNYZe2HuUU",
+      "reasoning_chars": 19476,
+      "result": "pass",
+      "source": "GPQA Diamond (modified)",
+      "total_ms": 85230.243958,
+      "ttft_ms": 467.081
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 356,
+      "finish_reason": "stop",
+      "id": "bebf1ed45ae14ad7b4f205f3909cb58a",
+      "reasoning_chars": 1333,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 4005.8517910000005,
+      "ttft_ms": 520.005666
+    },
+    {
+      "answer_chars": 1444,
+      "completion_tokens": 13467,
+      "finish_reason": "stop",
+      "id": "aime2025-18",
+      "reasoning_chars": 37955,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 141875.615625,
+      "ttft_ms": 520.379
+    },
+    {
+      "answer_chars": 488,
+      "completion_tokens": 1148,
+      "finish_reason": "stop",
+      "id": "recNFJjE5PPTqVJGv",
+      "reasoning_chars": 2116,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 11043.201333,
+      "ttft_ms": 559.667958
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1184,
+      "finish_reason": "stop",
+      "id": "7ca71b86327744b78e93185a45bc5cef",
+      "reasoning_chars": 4857,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 12834.222833,
+      "ttft_ms": 442.140666
+    },
+    {
+      "answer_chars": 453,
+      "completion_tokens": 4372,
+      "finish_reason": "stop",
+      "id": "aime2025-04",
+      "reasoning_chars": 9372,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 42976.331708000005,
+      "ttft_ms": 439.932667
+    },
+    {
+      "answer_chars": 734,
+      "completion_tokens": 7509,
+      "finish_reason": "stop",
+      "id": "rec2UlKqC6RFHdcro",
+      "reasoning_chars": 18313,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 72992.86379100001,
+      "ttft_ms": 517.7518749999999
+    },
+    {
+      "answer_chars": 385,
+      "completion_tokens": 11986,
+      "finish_reason": "stop",
+      "id": "d44b94f7749345a39a65f6312bda8764",
+      "reasoning_chars": 41556,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 132997.479,
+      "ttft_ms": 567.692667
+    },
+    {
+      "answer_chars": 1182,
+      "completion_tokens": 5797,
+      "finish_reason": "stop",
+      "id": "aime2025-19",
+      "reasoning_chars": 10038,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 53367.353708,
+      "ttft_ms": 564.988833
+    },
+    {
+      "answer_chars": 371,
+      "completion_tokens": 1972,
+      "finish_reason": "stop",
+      "id": "recv7GsQg3f0fvB1f",
+      "reasoning_chars": 8434,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 21371.348792,
+      "ttft_ms": 526.5157499999999
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1950,
+      "finish_reason": "stop",
+      "id": "febe406f44d74a40b50bb5b7c69d5dc1",
+      "reasoning_chars": 8754,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 19193.075249999998,
+      "ttft_ms": 502.167708
+    },
+    {
+      "answer_chars": 1370,
+      "completion_tokens": 10999,
+      "finish_reason": "stop",
+      "id": "aime2025-05",
+      "reasoning_chars": 23646,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 106965.969167,
+      "ttft_ms": 467.2305
+    },
+    {
+      "answer_chars": 499,
+      "completion_tokens": 11540,
+      "finish_reason": "stop",
+      "id": "recrHBEJJoDTV05JR",
+      "reasoning_chars": 39834,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 118407.33570899999,
+      "ttft_ms": 563.183792
+    },
+    {
+      "answer_chars": 414,
+      "completion_tokens": 622,
+      "finish_reason": "stop",
+      "id": "31950dc80ded400a9181f50626d1f75c",
+      "reasoning_chars": 936,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 6026.053,
+      "ttft_ms": 594.520208
+    },
+    {
+      "answer_chars": 778,
+      "completion_tokens": 5023,
+      "finish_reason": "stop",
+      "id": "aime2025-06",
+      "reasoning_chars": 13027,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 51276.943208,
+      "ttft_ms": 420.40575
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recb80OwMgNnceA9t",
+      "reasoning_chars": 98010,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 332947.995833,
+      "ttft_ms": 494.03575
+    },
+    {
+      "answer_chars": 527,
+      "completion_tokens": 6204,
+      "finish_reason": "stop",
+      "id": "0f14cd17be174618af6d60227e7dca9f",
+      "reasoning_chars": 17181,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 64284.690958,
+      "ttft_ms": 1173.025583
+    },
+    {
+      "answer_chars": 924,
+      "completion_tokens": 2962,
+      "finish_reason": "stop",
+      "id": "aime2025-21",
+      "reasoning_chars": 6380,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 27891.423875,
+      "ttft_ms": 847.798666
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recA1i5ZAh0Uzclxp",
+      "reasoning_chars": 106783,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 360413.73812500003,
+      "ttft_ms": 591.943958
+    },
+    {
+      "answer_chars": 543,
+      "completion_tokens": 897,
+      "finish_reason": "stop",
+      "id": "cef9bcc087434cc2b4e354a9baef55eb",
+      "reasoning_chars": 953,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 8896.928209,
+      "ttft_ms": 913.552625
+    },
+    {
+      "answer_chars": 1288,
+      "completion_tokens": 14365,
+      "finish_reason": "stop",
+      "id": "aime2025-07",
+      "reasoning_chars": 45715,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 158779.87425,
+      "ttft_ms": 521.9512910000001
+    },
+    {
+      "answer_chars": 591,
+      "completion_tokens": 8577,
+      "finish_reason": "stop",
+      "id": "recqGD3fxPCI59vPQ",
+      "reasoning_chars": 25730,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 90109.873542,
+      "ttft_ms": 590.850209
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 353,
+      "finish_reason": "stop",
+      "id": "9f93aa2cfdb547b5b3a4623f80f7fff6",
+      "reasoning_chars": 1373,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 4329.323459,
+      "ttft_ms": 518.8629169999999
+    },
+    {
+      "answer_chars": 1035,
+      "completion_tokens": 4138,
+      "finish_reason": "stop",
+      "id": "aime2025-22",
+      "reasoning_chars": 10865,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 42452.4355,
+      "ttft_ms": 510.249291
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "rechKl68Uc6H7vU0N",
+      "reasoning_chars": 97944,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 347741.884875,
+      "ttft_ms": 584.3199169999999
+    },
+    {
+      "answer_chars": 194,
+      "completion_tokens": 5546,
+      "finish_reason": "stop",
+      "id": "97ad69dda7b2462c98638b79e78aea0b",
+      "reasoning_chars": 22720,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 56474.938292,
+      "ttft_ms": 807.078209
+    },
+    {
+      "answer_chars": 814,
+      "completion_tokens": 6724,
+      "finish_reason": "stop",
+      "id": "aime2025-08",
+      "reasoning_chars": 13053,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 64382.000542,
+      "ttft_ms": 500.859834
+    },
+    {
+      "answer_chars": 608,
+      "completion_tokens": 5824,
+      "finish_reason": "stop",
+      "id": "rec1zl5LvaatzGhFt",
+      "reasoning_chars": 16426,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 58807.323083,
+      "ttft_ms": 620.387167
+    },
+    {
+      "answer_chars": 220,
+      "completion_tokens": 1525,
+      "finish_reason": "stop",
+      "id": "e78e4e539d6f4e379ac140d923d7b1be",
+      "reasoning_chars": 7362,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 16532.44875,
+      "ttft_ms": 494.293875
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-09",
+      "reasoning_chars": 65782,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 337796.657791,
+      "ttft_ms": 489.595625
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recTs7qzfJs6kfLUK",
+      "reasoning_chars": 76995,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 301419.221292,
+      "ttft_ms": 1766.599833
+    },
+    {
+      "answer_chars": 203,
+      "completion_tokens": 30124,
+      "finish_reason": "stop",
+      "id": "8483667a25e74fdfa3188de4ea734f03",
+      "reasoning_chars": 109574,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 315457.419125,
+      "ttft_ms": 761.423709
+    },
+    {
+      "answer_chars": 1218,
+      "completion_tokens": 11456,
+      "finish_reason": "stop",
+      "id": "aime2025-24",
+      "reasoning_chars": 27527,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 118657.899375,
+      "ttft_ms": 827.893416
+    },
+    {
+      "answer_chars": 690,
+      "completion_tokens": 5784,
+      "finish_reason": "stop",
+      "id": "rec32C1ZEapBnCC0E",
+      "reasoning_chars": 14949,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 60097.438375000005,
+      "ttft_ms": 643.3076669999999
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 912,
+      "finish_reason": "stop",
+      "id": "e5ed76ef98144f06843125daf1bccd35",
+      "reasoning_chars": 2051,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 9200.858542,
+      "ttft_ms": 518.77325
+    },
+    {
+      "answer_chars": 2204,
+      "completion_tokens": 24923,
+      "finish_reason": "stop",
+      "id": "aime2025-10",
+      "reasoning_chars": 67823,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 259420.713375,
+      "ttft_ms": 582.958959
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recZWeueB7lSPR6wN",
+      "reasoning_chars": 104167,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 292873.60020800005,
+      "ttft_ms": 825.8457080000001
+    },
+    {
+      "answer_chars": 102,
+      "completion_tokens": 2906,
+      "finish_reason": "stop",
+      "id": "fd7924876c4845cd83d95d61dfa0b236",
+      "reasoning_chars": 14000,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 30662.914792,
+      "ttft_ms": 761.2434999999999
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-25",
+      "reasoning_chars": 98598,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 297958.59479199996,
+      "ttft_ms": 447.44125
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recVvpD8miVjmmyfe",
+      "reasoning_chars": 75136,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 309495.360875,
+      "ttft_ms": 755.84425
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "6bfe7d19299d4b3184636e1f51694306",
+      "reasoning_chars": 101240,
+      "result": "exhausted",
+      "source": "SuperGPQA",
+      "total_ms": 311542.38412500004,
+      "ttft_ms": 875.9735
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-26",
+      "reasoning_chars": 118446,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 305537.42925,
+      "ttft_ms": 942.4300420000001
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 20513,
+      "finish_reason": "stop",
+      "id": "recAAJoHMW45Lv5je",
+      "reasoning_chars": 77881,
+      "result": "fail",
+      "source": "GPQA Diamond",
+      "total_ms": 164054.83125000002,
+      "ttft_ms": 815.912084
+    },
+    {
+      "answer_chars": 408,
+      "completion_tokens": 1429,
+      "finish_reason": "stop",
+      "id": "e1825d70c5844c22933087eafa89e39c",
+      "reasoning_chars": 2259,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 13141.369209,
+      "ttft_ms": 862.3114589999999
+    },
+    {
+      "answer_chars": 1278,
+      "completion_tokens": 30274,
+      "finish_reason": "stop",
+      "id": "aime2025-12",
+      "reasoning_chars": 71388,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 312026.676958,
+      "ttft_ms": 443.024542
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "reckEnrOPFT9Ru7tW",
+      "reasoning_chars": 105991,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 333209.492708,
+      "ttft_ms": 795.115208
+    },
+    {
+      "answer_chars": 1092,
+      "completion_tokens": 17871,
+      "finish_reason": "stop",
+      "id": "ab430ac3f18e4e02a2cb3f35498c6b30",
+      "reasoning_chars": 46633,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 183831.28833399998,
+      "ttft_ms": 845.5382500000001
+    },
+    {
+      "answer_chars": 1244,
+      "completion_tokens": 16707,
+      "finish_reason": "stop",
+      "id": "aime2025-27",
+      "reasoning_chars": 39131,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 165268.677917,
+      "ttft_ms": 699.087375
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "rec8nshandHARTkrg",
+      "reasoning_chars": 99198,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 333792.648709,
+      "ttft_ms": 892.167917
+    },
+    {
+      "answer_chars": 548,
+      "completion_tokens": 2830,
+      "finish_reason": "stop",
+      "id": "e8c5da5ca40647158b0c91dd695c6243",
+      "reasoning_chars": 6529,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 27616.275458,
+      "ttft_ms": 1104.628416
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-13",
+      "reasoning_chars": 84876,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 289234.278959,
+      "ttft_ms": 474.554375
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recFaL6j8UMhutXrc",
+      "reasoning_chars": 99094,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 337121.159958,
+      "ttft_ms": 868.13275
+    },
+    {
+      "answer_chars": 704,
+      "completion_tokens": 2472,
+      "finish_reason": "stop",
+      "id": "05efdc6fb2404ddc8dd5729bb68c74e5",
+      "reasoning_chars": 4361,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 23082.823999999997,
+      "ttft_ms": 772.785792
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-28",
+      "reasoning_chars": 42341,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 270782.23350000003,
+      "ttft_ms": 510.42125000000004
+    },
+    {
+      "answer_chars": 326,
+      "completion_tokens": 9179,
+      "finish_reason": "stop",
+      "id": "reczQ4I0VpENdMtIj",
+      "reasoning_chars": 44165,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 93602.65783299999,
+      "ttft_ms": 759.521708
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "ba52e06cbe1a4310a77a7d12cd1db943",
+      "reasoning_chars": 86169,
+      "result": "exhausted",
+      "source": "SuperGPQA",
+      "total_ms": 321926.425583,
+      "ttft_ms": 918.427541
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-29",
+      "reasoning_chars": 67348,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 343280.43595899997,
+      "ttft_ms": 861.88575
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recWxGU8Q4YReJ1tb",
+      "reasoning_chars": 95893,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 341060.402542,
+      "ttft_ms": 1106.733875
+    },
+    {
+      "answer_chars": 480,
+      "completion_tokens": 1467,
+      "finish_reason": "stop",
+      "id": "591a77df21324272914be82ac6583399",
+      "reasoning_chars": 2786,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 14707.001333,
+      "ttft_ms": 828.2319580000001
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-15",
+      "reasoning_chars": 82170,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 362740.44533300004,
+      "ttft_ms": 426.33425
+    },
+    {
+      "answer_chars": 508,
+      "completion_tokens": 19524,
+      "finish_reason": "stop",
+      "id": "recMicVBcqy1xM1jq",
+      "reasoning_chars": 65037,
+      "result": "fail",
+      "source": "GPQA Diamond",
+      "total_ms": 202671.583833,
+      "ttft_ms": 846.4155
+    },
+    {
+      "answer_chars": 236,
+      "completion_tokens": 1091,
+      "finish_reason": "stop",
+      "id": "e780f37a5baa4fe094cd9c157486664d",
+      "reasoning_chars": 4675,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 12668.975250000001,
+      "ttft_ms": 645.7173750000001
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-30",
+      "reasoning_chars": 66429,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 333789.22612500004,
+      "ttft_ms": 471.712541
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 8424,
+      "finish_reason": "stop",
+      "id": "compsec-076",
+      "reasoning_chars": 32031,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 93214.069292,
+      "ttft_ms": 948.313584
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 16570,
+      "finish_reason": "stop",
+      "id": "compsec-077",
+      "reasoning_chars": 60787,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 178285.559792,
+      "ttft_ms": 656.340667
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 751,
+      "finish_reason": "stop",
+      "id": "compsec-078",
+      "reasoning_chars": 2680,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 8499.385917,
+      "ttft_ms": 772.1086670000001
+    },
+    {
+      "answer_chars": 13,
+      "completion_tokens": 7326,
+      "finish_reason": "stop",
+      "id": "compsec-079",
+      "reasoning_chars": 30412,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 79250.792709,
+      "ttft_ms": 624.2195419999999
+    },
+    {
+      "answer_chars": 11,
+      "completion_tokens": 3509,
+      "finish_reason": "stop",
+      "id": "compsec-080",
+      "reasoning_chars": 15506,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 39996.01675,
+      "ttft_ms": 545.5102499999999
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 4887,
+      "finish_reason": "stop",
+      "id": "compsec-081",
+      "reasoning_chars": 19363,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 53585.437834,
+      "ttft_ms": 655.252834
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 3773,
+      "finish_reason": "stop",
+      "id": "compsec-082",
+      "reasoning_chars": 15663,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 42494.792457999996,
+      "ttft_ms": 580.1704159999999
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 11041,
+      "finish_reason": "stop",
+      "id": "compsec-083",
+      "reasoning_chars": 43080,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 121134.461875,
+      "ttft_ms": 585.663542
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 3287,
+      "finish_reason": "stop",
+      "id": "compsec-084",
+      "reasoning_chars": 11754,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 36503.871708,
+      "ttft_ms": 648.515083
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 2373,
+      "finish_reason": "stop",
+      "id": "compsec-085",
+      "reasoning_chars": 9270,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 25341.3945,
+      "ttft_ms": 593.2052920000001
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 2830,
+      "finish_reason": "stop",
+      "id": "compsec-086",
+      "reasoning_chars": 11126,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 30856.708749999998,
+      "ttft_ms": 608.968
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 7550,
+      "finish_reason": "stop",
+      "id": "compsec-087",
+      "reasoning_chars": 26016,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 79131.196083,
+      "ttft_ms": 617.592083
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 2441,
+      "finish_reason": "stop",
+      "id": "compsec-088",
+      "reasoning_chars": 9399,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 27086.324167000002,
+      "ttft_ms": 629.295167
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 725,
+      "finish_reason": "stop",
+      "id": "compsec-089",
+      "reasoning_chars": 2803,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 9048.985375,
+      "ttft_ms": 501.207459
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 2806,
+      "finish_reason": "stop",
+      "id": "compsec-090",
+      "reasoning_chars": 9961,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 30156.173084,
+      "ttft_ms": 616.585875
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 440,
+      "finish_reason": "stop",
+      "id": "compsec-091",
+      "reasoning_chars": 1816,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 5379.381375,
+      "ttft_ms": 488.554167
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1077,
+      "finish_reason": "stop",
+      "id": "compsec-092",
+      "reasoning_chars": 4451,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 11252.273875,
+      "ttft_ms": 517.316959
+    }
+  ],
+  "config": {
+    "max_tokens": 32768,
+    "seed": 42,
+    "temperature": 0.0
+  },
+  "exhausted": 21,
+  "failed_answered": 9,
+  "model": "fit-17g",
+  "passed": 62
+}

bench/fit-192k/config-public.txt

@@ -0,0 +1,25 @@
+run:        fit-192k
+recorded:   2026-08-16T01:23:22+02:00 RETROACTIVELY — see caveat below
+source:     bin/bench-fits.sh (file last changed in ab44c65, i.e. unchanged
+            across all three fit legs) — the command below is
+            reconstructed from that script, which is version controlled and was
+            unchanged across all three fit legs. This is the authoritative record.
+            The serve log does NOT contain the config — verified 2026-08-16:
+            no type_k/type_v, no flash-attn, no build version at this verbosity.
+serve_cmd:  ~/llama.cpp/build-mtp/bin/llama-server -m ~/models/qwen38-27b-fit-192k-q4kv.gguf -md ~/models/Qwen3.8-27B-MTP-ONLY-Q8_0.gguf --spec-draft-n-max 3 -c 196608 -ctk q4_0 -ctv q4_0 -fa on -ngl 99 -ngld 99 --port 8083 --host 0.0.0.0
+client:     bs4 qa-eval --base-url http://<redacted>:8083/v1 --model fit-192k
+sampler:    temperature 0.0, seed 42, max_tokens 32768 (bs4, sent in request body)
+--- desktop side (captured 2026-08-16, not at run time) ---
+llama.cpp:  22b8e31
+model:      18825459584 bytes  ~/models/qwen38-27b-fit-192k-q4kv.gguf
+gpu:        NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 250.00 W
+driver_ver: 610.57.04
+--- caveat ---
+Written after the fact. Trustworthy because the driver script is in git and
+unchanged since the runs; NOT equivalent to a config captured before the run.
+
+--- publication note ---
+Scrubbed for publication by sanitize-config.py: hostnames, internal
+repo paths and home paths removed, GPU index omitted (immaterial —
+one card, one power cap), chat template omitted for size. Everything
+that affects the measurement is kept verbatim.

bench/fit-192k/qa-eval-1786830067.csv

@@ -0,0 +1,93 @@
+id,source,domain,result,reasoning_chars,answer_chars,completion_tokens,finish_reason,ttft_ms,total_ms
+recNu3MXkvWUzHZr9,GPQA Diamond,Physics,pass,4936,451,2443,stop,399.6,31295.6
+001b51d76b4d422988f2c11f104a2c6c,SuperGPQA,Agronomy,pass,2414,9,599,stop,454.5,8359.6
+aime2025-01,AIME2025,Number Theory,pass,1502,408,801,stop,444.6,10339.3
+recoiTJPGUmzAkief,GPQA Diamond,Physics,pass,7822,645,1927,stop,569.8,31185.6
+b7e20eac98764fb0bf30e8366d951daa,SuperGPQA,Engineering,pass,10843,538,2788,stop,538.1,37812.2
+aime2025-16,AIME2025,Geometry,pass,2108,631,1252,stop,488.4,15553.1
+rec4UqStf9WUVif1f,GPQA Diamond,Physics,pass,1783,474,1108,stop,525.8,13719.5
+4a1d1780a93f4093b6fb7d3c314cbea8,SuperGPQA,Engineering,fail,22951,9,4961,stop,469.1,76262.1
+aime2025-02,AIME2025,Geometry,pass,65592,1219,26079,stop,818.3,360042.0
+recgI6tUQ7RLJRWGx,GPQA Diamond,Physics,pass,5748,565,2278,stop,836.2,31435.5
+6082513c8dba4ec68aa68f1bf5854d09,SuperGPQA,Law,pass,9458,173,1967,stop,476.1,28106.5
+aime2025-03,AIME2025,Combinatorics,pass,1664,650,986,stop,514.6,12581.4
+recDytVnNYZe2HuUU,GPQA Diamond (modified),Physics,pass,10058,457,4591,stop,508.9,62373.0
+bebf1ed45ae14ad7b4f205f3909cb58a,SuperGPQA,Literature and Arts,pass,1436,9,384,stop,543.2,5540.1
+aime2025-18,AIME2025,Combinatorics,pass,16017,1115,5757,stop,740.7,80568.6
+recNFJjE5PPTqVJGv,GPQA Diamond,Chemistry,pass,1384,778,1018,stop,556.2,12578.8
+7ca71b86327744b78e93185a45bc5cef,SuperGPQA,Medicine,pass,3503,9,847,stop,494.4,12817.4
+aime2025-04,AIME2025,Algebra,pass,2798,433,1443,stop,472.5,18052.3
+rec2UlKqC6RFHdcro,GPQA Diamond,Physics,pass,2265,767,1100,stop,622.9,14286.1
+d44b94f7749345a39a65f6312bda8764,SuperGPQA,Engineering,pass,1793,298,593,stop,497.2,8716.5
+aime2025-19,AIME2025,Algebra,pass,4450,1155,2778,stop,524.3,33512.1
+recv7GsQg3f0fvB1f,GPQA Diamond,Chemistry,pass,40782,429,9787,stop,522.3,144361.4
+febe406f44d74a40b50bb5b7c69d5dc1,SuperGPQA,Sociology,pass,8705,9,1973,stop,624.0,26198.6
+aime2025-05,AIME2025,"Combinatorics, Number Theory",pass,7137,1032,2932,stop,505.3,39824.1
+recrHBEJJoDTV05JR,GPQA Diamond,Chemistry,pass,83031,595,26640,stop,519.5,370456.4
+31950dc80ded400a9181f50626d1f75c,SuperGPQA,Science,pass,699,475,573,stop,762.6,7321.9
+aime2025-06,AIME2025,Geometry,pass,3346,825,1577,stop,504.5,20951.7
+recb80OwMgNnceA9t,GPQA Diamond,Chemistry,pass,76394,518,27666,stop,514.8,404564.0
+0f14cd17be174618af6d60227e7dca9f,SuperGPQA,Engineering,pass,17039,777,6424,stop,1186.3,91093.0
+aime2025-21,AIME2025,"Algebra, Geometry",pass,7398,976,2912,stop,977.6,38763.8
+recA1i5ZAh0Uzclxp,GPQA Diamond,Chemistry,pass,51145,878,16280,stop,579.2,237960.6
+cef9bcc087434cc2b4e354a9baef55eb,SuperGPQA,Engineering,pass,1002,596,839,stop,843.4,10470.6
+aime2025-07,AIME2025,Combinatorics,pass,19796,1860,7081,stop,528.9,102988.0
+recqGD3fxPCI59vPQ,GPQA Diamond,Physics,pass,14964,714,4582,stop,624.0,67799.6
+9f93aa2cfdb547b5b3a4623f80f7fff6,SuperGPQA,Engineering,pass,1315,9,326,stop,626.8,4879.5
+aime2025-22,AIME2025,Number Theory,pass,5525,812,2260,stop,465.0,30156.7
+rechKl68Uc6H7vU0N,GPQA Diamond,Chemistry,pass,70331,253,23035,stop,597.7,327347.1
+97ad69dda7b2462c98638b79e78aea0b,SuperGPQA,Medicine,pass,2249,9,538,stop,719.9,7866.0
+aime2025-08,AIME2025,Algebra,pass,5717,812,2860,stop,503.2,36481.0
+rec1zl5LvaatzGhFt,GPQA Diamond,Physics,pass,2520,723,1156,stop,610.5,15197.6
+e78e4e539d6f4e379ac140d923d7b1be,SuperGPQA,Military Science,fail,6596,9,1337,stop,520.7,19068.0
+aime2025-09,AIME2025,Algebra,pass,41658,1540,21574,stop,513.9,281933.8
+recTs7qzfJs6kfLUK,GPQA Diamond,Biology,pass,67248,347,25692,stop,1037.9,341059.3
+8483667a25e74fdfa3188de4ea734f03,SuperGPQA,Science,pass,106994,885,30235,stop,742.2,446215.6
+aime2025-24,AIME2025,Algebra,pass,9090,1480,4274,stop,820.8,58242.9
+rec32C1ZEapBnCC0E,GPQA Diamond,Physics,pass,4657,511,2043,stop,540.4,27405.1
+e5ed76ef98144f06843125daf1bccd35,SuperGPQA,Science,pass,1291,431,825,stop,616.0,10582.4
+aime2025-10,AIME2025,Combinatorics,pass,57473,1865,19130,stop,591.3,269200.2
+recZWeueB7lSPR6wN,GPQA Diamond,Chemistry,exhausted,111206,0,32768,length,686.6,498221.2
+fd7924876c4845cd83d95d61dfa0b236,SuperGPQA,Sociology,pass,7252,199,1407,stop,852.1,20779.4
+aime2025-25,AIME2025,Combinatorics,pass,13132,1436,4790,stop,476.7,66163.7
+recVvpD8miVjmmyfe,GPQA Diamond,Biology,exhausted,68938,0,32768,length,518.5,460292.0
+6bfe7d19299d4b3184636e1f51694306,SuperGPQA,Agronomy,pass,28235,418,8694,stop,935.6,120118.4
+aime2025-26,AIME2025,Combinatorics,pass,17145,1076,6154,stop,522.0,85879.9
+recAAJoHMW45Lv5je,GPQA Diamond,Chemistry,pass,37041,685,11996,stop,530.2,164023.8
+e1825d70c5844c22933087eafa89e39c,SuperGPQA,Engineering,pass,2157,9,1115,stop,807.1,14093.8
+aime2025-12,AIME2025,Algebra,pass,22507,1299,10646,stop,535.2,139449.2
+reckEnrOPFT9Ru7tW,GPQA Diamond,Chemistry,pass,29103,598,9125,stop,642.1,130403.1
+ab430ac3f18e4e02a2cb3f35498c6b30,SuperGPQA,Engineering,pass,10539,579,4310,stop,621.6,59544.1
+aime2025-27,AIME2025,Geometry,pass,26066,1302,10602,stop,603.4,145547.0
+rec8nshandHARTkrg,GPQA Diamond,Chemistry,exhausted,109519,0,32768,length,873.6,470929.8
+e8c5da5ca40647158b0c91dd695c6243,SuperGPQA,Science,pass,13392,590,5390,stop,1591.7,75243.2
+aime2025-13,AIME2025,Combinatorics,pass,68901,2280,19526,stop,1806.1,292776.6
+recFaL6j8UMhutXrc,GPQA Diamond,Chemistry,pass,46311,695,15957,stop,2614.2,222713.2
+05efdc6fb2404ddc8dd5729bb68c74e5,SuperGPQA,Science,pass,1958,693,1446,stop,725.9,17601.0
+aime2025-28,AIME2025,Number Theory,exhausted,65433,0,32768,length,544.5,457339.0
+reczQ4I0VpENdMtIj,GPQA Diamond,Biology,pass,75489,274,16041,stop,915.5,235469.4
+ba52e06cbe1a4310a77a7d12cd1db943,SuperGPQA,Science,pass,11302,298,4082,stop,1028.5,53318.8
+aime2025-29,AIME2025,Geometry,pass,31429,1832,16128,stop,637.3,210239.5
+recWxGU8Q4YReJ1tb,GPQA Diamond,Chemistry,exhausted,96591,0,32768,length,1224.8,472367.2
+591a77df21324272914be82ac6583399,SuperGPQA,Science,fail,1397,353,689,stop,950.8,9232.1
+aime2025-15,AIME2025,Number Theory,exhausted,77683,0,32768,length,484.0,483552.3
+recMicVBcqy1xM1jq,GPQA Diamond,Physics,pass,39565,735,13522,stop,859.8,199109.0
+e780f37a5baa4fe094cd9c157486664d,SuperGPQA,Sociology,fail,12613,223,2852,stop,681.6,43924.2
+aime2025-30,AIME2025,Algebra,pass,54814,1510,24716,stop,556.4,355474.5
+compsec-076,COMPSEC,FreeBSD / librpcsec_gss,pass,19590,10,4877,stop,1007.9,74213.2
+compsec-077,COMPSEC,GNU Inetutils telnetd,exhausted,46873,0,32768,length,700.8,365861.9
+compsec-078,COMPSEC,Botan,pass,2226,10,615,stop,980.3,9750.7
+compsec-079,COMPSEC,Botan,pass,27818,10,6272,stop,713.4,100526.5
+compsec-080,COMPSEC,Botan,pass,10639,11,2466,stop,666.7,39808.0
+compsec-081,COMPSEC,uds-c,pass,36999,10,9487,stop,734.2,145307.8
+compsec-082,COMPSEC,Firebird,pass,52562,10,13064,stop,714.1,206453.3
+compsec-083,COMPSEC,Firebird,pass,81210,10,21859,stop,791.1,344479.0
+compsec-084,COMPSEC,PHP,pass,3958,9,1148,stop,792.9,18033.3
+compsec-085,COMPSEC,PHP PDO Firebird,pass,11133,9,2696,stop,607.4,42139.1
+compsec-086,COMPSEC,libexpat,pass,4932,9,1277,stop,648.2,19653.8
+compsec-087,COMPSEC,Mbed TLS,pass,34941,9,9872,stop,684.1,148723.9
+compsec-088,COMPSEC,Mbed TLS,pass,4654,10,1203,stop,717.5,18592.0
+compsec-089,COMPSEC,Mbed TLS,pass,3084,10,800,stop,560.0,12067.3
+compsec-090,COMPSEC,Linux kernel,pass,6440,10,1802,stop,632.2,25817.4
+compsec-091,COMPSEC,Linux kernel USB gadget storage,pass,1660,9,434,stop,546.4,6866.9
+compsec-092,COMPSEC,Linux kernel AppArmor,pass,18810,10,4591,stop,625.8,71831.0

bench/fit-192k/qa-eval-1786830067.json

@@ -0,0 +1,1028 @@
+{
+  "accuracy": 0.8804347826086957,
+  "bench_version": "0.1.0",
+  "cases": 92,
+  "cases_detail": [
+    {
+      "answer_chars": 451,
+      "completion_tokens": 2443,
+      "finish_reason": "stop",
+      "id": "recNu3MXkvWUzHZr9",
+      "reasoning_chars": 4936,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 31295.601708,
+      "ttft_ms": 399.6055
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 599,
+      "finish_reason": "stop",
+      "id": "001b51d76b4d422988f2c11f104a2c6c",
+      "reasoning_chars": 2414,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 8359.602083,
+      "ttft_ms": 454.54774999999995
+    },
+    {
+      "answer_chars": 408,
+      "completion_tokens": 801,
+      "finish_reason": "stop",
+      "id": "aime2025-01",
+      "reasoning_chars": 1502,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 10339.32525,
+      "ttft_ms": 444.571125
+    },
+    {
+      "answer_chars": 645,
+      "completion_tokens": 1927,
+      "finish_reason": "stop",
+      "id": "recoiTJPGUmzAkief",
+      "reasoning_chars": 7822,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 31185.61875,
+      "ttft_ms": 569.7921249999999
+    },
+    {
+      "answer_chars": 538,
+      "completion_tokens": 2788,
+      "finish_reason": "stop",
+      "id": "b7e20eac98764fb0bf30e8366d951daa",
+      "reasoning_chars": 10843,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 37812.201083,
+      "ttft_ms": 538.05125
+    },
+    {
+      "answer_chars": 631,
+      "completion_tokens": 1252,
+      "finish_reason": "stop",
+      "id": "aime2025-16",
+      "reasoning_chars": 2108,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 15553.081666,
+      "ttft_ms": 488.417125
+    },
+    {
+      "answer_chars": 474,
+      "completion_tokens": 1108,
+      "finish_reason": "stop",
+      "id": "rec4UqStf9WUVif1f",
+      "reasoning_chars": 1783,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 13719.513916,
+      "ttft_ms": 525.752875
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 4961,
+      "finish_reason": "stop",
+      "id": "4a1d1780a93f4093b6fb7d3c314cbea8",
+      "reasoning_chars": 22951,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 76262.11841600001,
+      "ttft_ms": 469.140375
+    },
+    {
+      "answer_chars": 1219,
+      "completion_tokens": 26079,
+      "finish_reason": "stop",
+      "id": "aime2025-02",
+      "reasoning_chars": 65592,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 360041.997875,
+      "ttft_ms": 818.316625
+    },
+    {
+      "answer_chars": 565,
+      "completion_tokens": 2278,
+      "finish_reason": "stop",
+      "id": "recgI6tUQ7RLJRWGx",
+      "reasoning_chars": 5748,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 31435.51425,
+      "ttft_ms": 836.2213750000001
+    },
+    {
+      "answer_chars": 173,
+      "completion_tokens": 1967,
+      "finish_reason": "stop",
+      "id": "6082513c8dba4ec68aa68f1bf5854d09",
+      "reasoning_chars": 9458,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 28106.493334,
+      "ttft_ms": 476.113875
+    },
+    {
+      "answer_chars": 650,
+      "completion_tokens": 986,
+      "finish_reason": "stop",
+      "id": "aime2025-03",
+      "reasoning_chars": 1664,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 12581.4355,
+      "ttft_ms": 514.6454590000001
+    },
+    {
+      "answer_chars": 457,
+      "completion_tokens": 4591,
+      "finish_reason": "stop",
+      "id": "recDytVnNYZe2HuUU",
+      "reasoning_chars": 10058,
+      "result": "pass",
+      "source": "GPQA Diamond (modified)",
+      "total_ms": 62373.030999999995,
+      "ttft_ms": 508.87266600000004
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 384,
+      "finish_reason": "stop",
+      "id": "bebf1ed45ae14ad7b4f205f3909cb58a",
+      "reasoning_chars": 1436,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 5540.072875,
+      "ttft_ms": 543.231792
+    },
+    {
+      "answer_chars": 1115,
+      "completion_tokens": 5757,
+      "finish_reason": "stop",
+      "id": "aime2025-18",
+      "reasoning_chars": 16017,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 80568.597541,
+      "ttft_ms": 740.695291
+    },
+    {
+      "answer_chars": 778,
+      "completion_tokens": 1018,
+      "finish_reason": "stop",
+      "id": "recNFJjE5PPTqVJGv",
+      "reasoning_chars": 1384,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 12578.768667,
+      "ttft_ms": 556.239792
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 847,
+      "finish_reason": "stop",
+      "id": "7ca71b86327744b78e93185a45bc5cef",
+      "reasoning_chars": 3503,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 12817.433040999998,
+      "ttft_ms": 494.44225
+    },
+    {
+      "answer_chars": 433,
+      "completion_tokens": 1443,
+      "finish_reason": "stop",
+      "id": "aime2025-04",
+      "reasoning_chars": 2798,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 18052.332667000002,
+      "ttft_ms": 472.539042
+    },
+    {
+      "answer_chars": 767,
+      "completion_tokens": 1100,
+      "finish_reason": "stop",
+      "id": "rec2UlKqC6RFHdcro",
+      "reasoning_chars": 2265,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 14286.128666999999,
+      "ttft_ms": 622.890375
+    },
+    {
+      "answer_chars": 298,
+      "completion_tokens": 593,
+      "finish_reason": "stop",
+      "id": "d44b94f7749345a39a65f6312bda8764",
+      "reasoning_chars": 1793,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 8716.484625,
+      "ttft_ms": 497.215667
+    },
+    {
+      "answer_chars": 1155,
+      "completion_tokens": 2778,
+      "finish_reason": "stop",
+      "id": "aime2025-19",
+      "reasoning_chars": 4450,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 33512.103542,
+      "ttft_ms": 524.2665840000001
+    },
+    {
+      "answer_chars": 429,
+      "completion_tokens": 9787,
+      "finish_reason": "stop",
+      "id": "recv7GsQg3f0fvB1f",
+      "reasoning_chars": 40782,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 144361.435666,
+      "ttft_ms": 522.277541
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1973,
+      "finish_reason": "stop",
+      "id": "febe406f44d74a40b50bb5b7c69d5dc1",
+      "reasoning_chars": 8705,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 26198.616958,
+      "ttft_ms": 623.9991249999999
+    },
+    {
+      "answer_chars": 1032,
+      "completion_tokens": 2932,
+      "finish_reason": "stop",
+      "id": "aime2025-05",
+      "reasoning_chars": 7137,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 39824.123374999996,
+      "ttft_ms": 505.3135
+    },
+    {
+      "answer_chars": 595,
+      "completion_tokens": 26640,
+      "finish_reason": "stop",
+      "id": "recrHBEJJoDTV05JR",
+      "reasoning_chars": 83031,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 370456.417042,
+      "ttft_ms": 519.5193340000001
+    },
+    {
+      "answer_chars": 475,
+      "completion_tokens": 573,
+      "finish_reason": "stop",
+      "id": "31950dc80ded400a9181f50626d1f75c",
+      "reasoning_chars": 699,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 7321.872708,
+      "ttft_ms": 762.591958
+    },
+    {
+      "answer_chars": 825,
+      "completion_tokens": 1577,
+      "finish_reason": "stop",
+      "id": "aime2025-06",
+      "reasoning_chars": 3346,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 20951.714375000003,
+      "ttft_ms": 504.47074999999995
+    },
+    {
+      "answer_chars": 518,
+      "completion_tokens": 27666,
+      "finish_reason": "stop",
+      "id": "recb80OwMgNnceA9t",
+      "reasoning_chars": 76394,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 404564.034291,
+      "ttft_ms": 514.815541
+    },
+    {
+      "answer_chars": 777,
+      "completion_tokens": 6424,
+      "finish_reason": "stop",
+      "id": "0f14cd17be174618af6d60227e7dca9f",
+      "reasoning_chars": 17039,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 91093.012875,
+      "ttft_ms": 1186.316458
+    },
+    {
+      "answer_chars": 976,
+      "completion_tokens": 2912,
+      "finish_reason": "stop",
+      "id": "aime2025-21",
+      "reasoning_chars": 7398,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 38763.822917,
+      "ttft_ms": 977.560833
+    },
+    {
+      "answer_chars": 878,
+      "completion_tokens": 16280,
+      "finish_reason": "stop",
+      "id": "recA1i5ZAh0Uzclxp",
+      "reasoning_chars": 51145,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 237960.646375,
+      "ttft_ms": 579.2425420000001
+    },
+    {
+      "answer_chars": 596,
+      "completion_tokens": 839,
+      "finish_reason": "stop",
+      "id": "cef9bcc087434cc2b4e354a9baef55eb",
+      "reasoning_chars": 1002,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 10470.621292,
+      "ttft_ms": 843.4355
+    },
+    {
+      "answer_chars": 1860,
+      "completion_tokens": 7081,
+      "finish_reason": "stop",
+      "id": "aime2025-07",
+      "reasoning_chars": 19796,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 102988.00825000001,
+      "ttft_ms": 528.925875
+    },
+    {
+      "answer_chars": 714,
+      "completion_tokens": 4582,
+      "finish_reason": "stop",
+      "id": "recqGD3fxPCI59vPQ",
+      "reasoning_chars": 14964,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 67799.600916,
+      "ttft_ms": 623.9945
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 326,
+      "finish_reason": "stop",
+      "id": "9f93aa2cfdb547b5b3a4623f80f7fff6",
+      "reasoning_chars": 1315,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 4879.490084,
+      "ttft_ms": 626.784417
+    },
+    {
+      "answer_chars": 812,
+      "completion_tokens": 2260,
+      "finish_reason": "stop",
+      "id": "aime2025-22",
+      "reasoning_chars": 5525,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 30156.678833,
+      "ttft_ms": 464.996125
+    },
+    {
+      "answer_chars": 253,
+      "completion_tokens": 23035,
+      "finish_reason": "stop",
+      "id": "rechKl68Uc6H7vU0N",
+      "reasoning_chars": 70331,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 327347.06475,
+      "ttft_ms": 597.701958
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 538,
+      "finish_reason": "stop",
+      "id": "97ad69dda7b2462c98638b79e78aea0b",
+      "reasoning_chars": 2249,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 7865.990041,
+      "ttft_ms": 719.940333
+    },
+    {
+      "answer_chars": 812,
+      "completion_tokens": 2860,
+      "finish_reason": "stop",
+      "id": "aime2025-08",
+      "reasoning_chars": 5717,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 36480.992625,
+      "ttft_ms": 503.21475000000004
+    },
+    {
+      "answer_chars": 723,
+      "completion_tokens": 1156,
+      "finish_reason": "stop",
+      "id": "rec1zl5LvaatzGhFt",
+      "reasoning_chars": 2520,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 15197.569749999999,
+      "ttft_ms": 610.507333
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1337,
+      "finish_reason": "stop",
+      "id": "e78e4e539d6f4e379ac140d923d7b1be",
+      "reasoning_chars": 6596,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 19067.997834,
+      "ttft_ms": 520.7455
+    },
+    {
+      "answer_chars": 1540,
+      "completion_tokens": 21574,
+      "finish_reason": "stop",
+      "id": "aime2025-09",
+      "reasoning_chars": 41658,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 281933.832875,
+      "ttft_ms": 513.85575
+    },
+    {
+      "answer_chars": 347,
+      "completion_tokens": 25692,
+      "finish_reason": "stop",
+      "id": "recTs7qzfJs6kfLUK",
+      "reasoning_chars": 67248,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 341059.269584,
+      "ttft_ms": 1037.865292
+    },
+    {
+      "answer_chars": 885,
+      "completion_tokens": 30235,
+      "finish_reason": "stop",
+      "id": "8483667a25e74fdfa3188de4ea734f03",
+      "reasoning_chars": 106994,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 446215.645708,
+      "ttft_ms": 742.24075
+    },
+    {
+      "answer_chars": 1480,
+      "completion_tokens": 4274,
+      "finish_reason": "stop",
+      "id": "aime2025-24",
+      "reasoning_chars": 9090,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 58242.886374999995,
+      "ttft_ms": 820.82575
+    },
+    {
+      "answer_chars": 511,
+      "completion_tokens": 2043,
+      "finish_reason": "stop",
+      "id": "rec32C1ZEapBnCC0E",
+      "reasoning_chars": 4657,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 27405.115917,
+      "ttft_ms": 540.3725000000001
+    },
+    {
+      "answer_chars": 431,
+      "completion_tokens": 825,
+      "finish_reason": "stop",
+      "id": "e5ed76ef98144f06843125daf1bccd35",
+      "reasoning_chars": 1291,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 10582.406459,
+      "ttft_ms": 615.954917
+    },
+    {
+      "answer_chars": 1865,
+      "completion_tokens": 19130,
+      "finish_reason": "stop",
+      "id": "aime2025-10",
+      "reasoning_chars": 57473,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 269200.155083,
+      "ttft_ms": 591.284041
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recZWeueB7lSPR6wN",
+      "reasoning_chars": 111206,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 498221.152792,
+      "ttft_ms": 686.6275840000001
+    },
+    {
+      "answer_chars": 199,
+      "completion_tokens": 1407,
+      "finish_reason": "stop",
+      "id": "fd7924876c4845cd83d95d61dfa0b236",
+      "reasoning_chars": 7252,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 20779.398833,
+      "ttft_ms": 852.102041
+    },
+    {
+      "answer_chars": 1436,
+      "completion_tokens": 4790,
+      "finish_reason": "stop",
+      "id": "aime2025-25",
+      "reasoning_chars": 13132,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 66163.68637499999,
+      "ttft_ms": 476.737709
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recVvpD8miVjmmyfe",
+      "reasoning_chars": 68938,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 460292.007958,
+      "ttft_ms": 518.500583
+    },
+    {
+      "answer_chars": 418,
+      "completion_tokens": 8694,
+      "finish_reason": "stop",
+      "id": "6bfe7d19299d4b3184636e1f51694306",
+      "reasoning_chars": 28235,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 120118.39775,
+      "ttft_ms": 935.6095829999999
+    },
+    {
+      "answer_chars": 1076,
+      "completion_tokens": 6154,
+      "finish_reason": "stop",
+      "id": "aime2025-26",
+      "reasoning_chars": 17145,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 85879.90304100001,
+      "ttft_ms": 521.997833
+    },
+    {
+      "answer_chars": 685,
+      "completion_tokens": 11996,
+      "finish_reason": "stop",
+      "id": "recAAJoHMW45Lv5je",
+      "reasoning_chars": 37041,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 164023.837542,
+      "ttft_ms": 530.193208
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1115,
+      "finish_reason": "stop",
+      "id": "e1825d70c5844c22933087eafa89e39c",
+      "reasoning_chars": 2157,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 14093.81475,
+      "ttft_ms": 807.07975
+    },
+    {
+      "answer_chars": 1299,
+      "completion_tokens": 10646,
+      "finish_reason": "stop",
+      "id": "aime2025-12",
+      "reasoning_chars": 22507,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 139449.189458,
+      "ttft_ms": 535.166833
+    },
+    {
+      "answer_chars": 598,
+      "completion_tokens": 9125,
+      "finish_reason": "stop",
+      "id": "reckEnrOPFT9Ru7tW",
+      "reasoning_chars": 29103,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 130403.11474999998,
+      "ttft_ms": 642.1070840000001
+    },
+    {
+      "answer_chars": 579,
+      "completion_tokens": 4310,
+      "finish_reason": "stop",
+      "id": "ab430ac3f18e4e02a2cb3f35498c6b30",
+      "reasoning_chars": 10539,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 59544.074334000004,
+      "ttft_ms": 621.629
+    },
+    {
+      "answer_chars": 1302,
+      "completion_tokens": 10602,
+      "finish_reason": "stop",
+      "id": "aime2025-27",
+      "reasoning_chars": 26066,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 145547.010667,
+      "ttft_ms": 603.377417
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "rec8nshandHARTkrg",
+      "reasoning_chars": 109519,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 470929.765041,
+      "ttft_ms": 873.600916
+    },
+    {
+      "answer_chars": 590,
+      "completion_tokens": 5390,
+      "finish_reason": "stop",
+      "id": "e8c5da5ca40647158b0c91dd695c6243",
+      "reasoning_chars": 13392,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 75243.150417,
+      "ttft_ms": 1591.7083750000002
+    },
+    {
+      "answer_chars": 2280,
+      "completion_tokens": 19526,
+      "finish_reason": "stop",
+      "id": "aime2025-13",
+      "reasoning_chars": 68901,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 292776.569333,
+      "ttft_ms": 1806.0629999999999
+    },
+    {
+      "answer_chars": 695,
+      "completion_tokens": 15957,
+      "finish_reason": "stop",
+      "id": "recFaL6j8UMhutXrc",
+      "reasoning_chars": 46311,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 222713.19020799999,
+      "ttft_ms": 2614.1707499999998
+    },
+    {
+      "answer_chars": 693,
+      "completion_tokens": 1446,
+      "finish_reason": "stop",
+      "id": "05efdc6fb2404ddc8dd5729bb68c74e5",
+      "reasoning_chars": 1958,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 17601.032709000003,
+      "ttft_ms": 725.869459
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-28",
+      "reasoning_chars": 65433,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 457338.952458,
+      "ttft_ms": 544.493583
+    },
+    {
+      "answer_chars": 274,
+      "completion_tokens": 16041,
+      "finish_reason": "stop",
+      "id": "reczQ4I0VpENdMtIj",
+      "reasoning_chars": 75489,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 235469.385292,
+      "ttft_ms": 915.453459
+    },
+    {
+      "answer_chars": 298,
+      "completion_tokens": 4082,
+      "finish_reason": "stop",
+      "id": "ba52e06cbe1a4310a77a7d12cd1db943",
+      "reasoning_chars": 11302,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 53318.797833,
+      "ttft_ms": 1028.459583
+    },
+    {
+      "answer_chars": 1832,
+      "completion_tokens": 16128,
+      "finish_reason": "stop",
+      "id": "aime2025-29",
+      "reasoning_chars": 31429,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 210239.475542,
+      "ttft_ms": 637.297667
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recWxGU8Q4YReJ1tb",
+      "reasoning_chars": 96591,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 472367.23208299995,
+      "ttft_ms": 1224.8100829999998
+    },
+    {
+      "answer_chars": 353,
+      "completion_tokens": 689,
+      "finish_reason": "stop",
+      "id": "591a77df21324272914be82ac6583399",
+      "reasoning_chars": 1397,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 9232.105709000001,
+      "ttft_ms": 950.81525
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-15",
+      "reasoning_chars": 77683,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 483552.26095799997,
+      "ttft_ms": 483.954208
+    },
+    {
+      "answer_chars": 735,
+      "completion_tokens": 13522,
+      "finish_reason": "stop",
+      "id": "recMicVBcqy1xM1jq",
+      "reasoning_chars": 39565,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 199108.983458,
+      "ttft_ms": 859.7615830000001
+    },
+    {
+      "answer_chars": 223,
+      "completion_tokens": 2852,
+      "finish_reason": "stop",
+      "id": "e780f37a5baa4fe094cd9c157486664d",
+      "reasoning_chars": 12613,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 43924.241208,
+      "ttft_ms": 681.594042
+    },
+    {
+      "answer_chars": 1510,
+      "completion_tokens": 24716,
+      "finish_reason": "stop",
+      "id": "aime2025-30",
+      "reasoning_chars": 54814,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 355474.51729199995,
+      "ttft_ms": 556.4345000000001
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 4877,
+      "finish_reason": "stop",
+      "id": "compsec-076",
+      "reasoning_chars": 19590,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 74213.201084,
+      "ttft_ms": 1007.85225
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "compsec-077",
+      "reasoning_chars": 46873,
+      "result": "exhausted",
+      "source": "COMPSEC",
+      "total_ms": 365861.869833,
+      "ttft_ms": 700.8217500000001
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 615,
+      "finish_reason": "stop",
+      "id": "compsec-078",
+      "reasoning_chars": 2226,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 9750.739208,
+      "ttft_ms": 980.335333
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 6272,
+      "finish_reason": "stop",
+      "id": "compsec-079",
+      "reasoning_chars": 27818,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 100526.52949999999,
+      "ttft_ms": 713.434625
+    },
+    {
+      "answer_chars": 11,
+      "completion_tokens": 2466,
+      "finish_reason": "stop",
+      "id": "compsec-080",
+      "reasoning_chars": 10639,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 39808.013708,
+      "ttft_ms": 666.6535409999999
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 9487,
+      "finish_reason": "stop",
+      "id": "compsec-081",
+      "reasoning_chars": 36999,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 145307.75808300002,
+      "ttft_ms": 734.2491249999999
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 13064,
+      "finish_reason": "stop",
+      "id": "compsec-082",
+      "reasoning_chars": 52562,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 206453.321625,
+      "ttft_ms": 714.126583
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 21859,
+      "finish_reason": "stop",
+      "id": "compsec-083",
+      "reasoning_chars": 81210,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 344479.036875,
+      "ttft_ms": 791.050583
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1148,
+      "finish_reason": "stop",
+      "id": "compsec-084",
+      "reasoning_chars": 3958,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 18033.309542000003,
+      "ttft_ms": 792.864208
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 2696,
+      "finish_reason": "stop",
+      "id": "compsec-085",
+      "reasoning_chars": 11133,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 42139.139916,
+      "ttft_ms": 607.402583
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1277,
+      "finish_reason": "stop",
+      "id": "compsec-086",
+      "reasoning_chars": 4932,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 19653.80075,
+      "ttft_ms": 648.174
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 9872,
+      "finish_reason": "stop",
+      "id": "compsec-087",
+      "reasoning_chars": 34941,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 148723.870834,
+      "ttft_ms": 684.0811669999999
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1203,
+      "finish_reason": "stop",
+      "id": "compsec-088",
+      "reasoning_chars": 4654,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 18591.996583,
+      "ttft_ms": 717.458541
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 800,
+      "finish_reason": "stop",
+      "id": "compsec-089",
+      "reasoning_chars": 3084,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 12067.343542,
+      "ttft_ms": 559.9655839999999
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1802,
+      "finish_reason": "stop",
+      "id": "compsec-090",
+      "reasoning_chars": 6440,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 25817.421792,
+      "ttft_ms": 632.1675839999999
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 434,
+      "finish_reason": "stop",
+      "id": "compsec-091",
+      "reasoning_chars": 1660,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 6866.947375,
+      "ttft_ms": 546.41725
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 4591,
+      "finish_reason": "stop",
+      "id": "compsec-092",
+      "reasoning_chars": 18810,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 71831.03016699999,
+      "ttft_ms": 625.790333
+    }
+  ],
+  "config": {
+    "max_tokens": 32768,
+    "seed": 42,
+    "temperature": 0.0
+  },
+  "exhausted": 7,
+  "failed_answered": 4,
+  "model": "fit-192k",
+  "passed": 81
+}

bench/fit-256k/config-public.txt

@@ -0,0 +1,28 @@
+run:        fit-256k
+recorded:   2026-08-16T03:15 CEST RETROACTIVELY — see caveat below
+source:     bin/bench-fits.sh (our tooling repo; file last changed in ab44c65, i.e.
+            unchanged across all three fit legs) — the command below is
+            reconstructed from that script, which is version controlled. This is
+            the authoritative record. The serve log does NOT contain the config —
+            verified 2026-08-16: no type_k/type_v, no flash-attn, no build
+            version at this verbosity.
+serve_cmd:  ~/llama.cpp/build-mtp/bin/llama-server -m ~/models/qwen38-27b-fit-256k-q4kv.gguf -md ~/models/Qwen3.8-27B-MTP-ONLY-Q8_0.gguf --spec-draft-n-max 3 -c 262144 -ctk q4_0 -ctv q4_0 -fa on -ngl 99 -ngld 99 --port 8083 --host 0.0.0.0
+client:     bs4 qa-eval --base-url http://<redacted>:8083/v1 --model fit-256k
+sampler:    temperature 0.0, seed 42, max_tokens 32768 (bs4, sent in request body)
+--- desktop side (captured 2026-08-16 03:15, not at run time) ---
+llama.cpp:  22b8e31
+model:      16812197504 bytes  ~/models/qwen38-27b-fit-256k-q4kv.gguf
+gpu:        NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 250.00 W
+driver_ver: 610.57.04
+--- caveat ---
+Written after the fact, same as the fit-128k / fit-192k records. Trustworthy
+because the driver script is in git and unchanged since the runs; NOT equivalent
+to a config captured before the run. Note the context differs per fit leg
+(131072 / 196608 / 262144) — that is the variable the fit legs exist to test,
+and it is NOT the same as the flat 131072 used by the 2026-08-16 baseline legs.
+
+--- publication note ---
+Scrubbed for publication by sanitize-config.py: hostnames, internal
+repo paths and home paths removed, GPU index omitted (immaterial —
+one card, one power cap), chat template omitted for size. Everything
+that affects the measurement is kept verbatim.

bench/fit-256k/qa-eval-1786841317.csv

@@ -0,0 +1,93 @@
+id,source,domain,result,reasoning_chars,answer_chars,completion_tokens,finish_reason,ttft_ms,total_ms
+recNu3MXkvWUzHZr9,GPQA Diamond,Physics,pass,5918,447,2584,stop,427.9,32783.8
+001b51d76b4d422988f2c11f104a2c6c,SuperGPQA,Agronomy,pass,2275,9,515,stop,433.1,7251.8
+aime2025-01,AIME2025,Number Theory,pass,1433,313,724,stop,450.2,9103.5
+recoiTJPGUmzAkief,GPQA Diamond,Physics,pass,25896,513,5866,stop,605.0,91368.0
+b7e20eac98764fb0bf30e8366d951daa,SuperGPQA,Engineering,pass,9569,564,2450,stop,546.2,34419.9
+aime2025-16,AIME2025,Geometry,pass,2322,11,1017,stop,479.4,13231.1
+rec4UqStf9WUVif1f,GPQA Diamond,Physics,pass,2958,236,1436,stop,526.2,18388.5
+4a1d1780a93f4093b6fb7d3c314cbea8,SuperGPQA,Engineering,pass,2686,121,582,stop,448.2,8713.2
+aime2025-02,AIME2025,Geometry,pass,59852,1521,25339,stop,804.7,344975.8
+recgI6tUQ7RLJRWGx,GPQA Diamond,Physics,pass,2735,447,1130,stop,728.3,15628.3
+6082513c8dba4ec68aa68f1bf5854d09,SuperGPQA,Law,pass,6467,9,1329,stop,517.8,19068.8
+aime2025-03,AIME2025,Combinatorics,pass,1973,550,1048,stop,460.2,12975.8
+recDytVnNYZe2HuUU,GPQA Diamond (modified),Physics,pass,3105,481,1638,stop,453.7,21416.3
+bebf1ed45ae14ad7b4f205f3909cb58a,SuperGPQA,Literature and Arts,pass,971,9,236,stop,517.5,3711.2
+aime2025-18,AIME2025,Combinatorics,pass,12226,1184,4410,stop,559.8,63913.5
+recNFJjE5PPTqVJGv,GPQA Diamond,Chemistry,pass,1789,497,999,stop,524.2,12477.9
+7ca71b86327744b78e93185a45bc5cef,SuperGPQA,Medicine,pass,19756,9,4813,stop,611.7,68755.9
+aime2025-04,AIME2025,Algebra,pass,2768,475,1437,stop,519.2,18191.2
+rec2UlKqC6RFHdcro,GPQA Diamond,Physics,pass,14222,554,5025,stop,523.9,70473.4
+d44b94f7749345a39a65f6312bda8764,SuperGPQA,Engineering,pass,9741,280,2815,stop,523.5,44109.1
+aime2025-19,AIME2025,Algebra,pass,5097,877,3115,stop,524.4,38191.3
+recv7GsQg3f0fvB1f,GPQA Diamond,Chemistry,pass,45028,449,10584,stop,557.4,157097.7
+febe406f44d74a40b50bb5b7c69d5dc1,SuperGPQA,Sociology,fail,19449,9,4315,stop,651.6,55830.9
+aime2025-05,AIME2025,"Combinatorics, Number Theory",pass,15671,1167,7106,stop,498.5,90739.2
+recrHBEJJoDTV05JR,GPQA Diamond,Chemistry,pass,62406,632,19556,stop,608.0,267669.5
+31950dc80ded400a9181f50626d1f75c,SuperGPQA,Science,pass,1502,9,473,stop,686.4,7179.4
+aime2025-06,AIME2025,Geometry,pass,4969,767,2042,stop,492.3,27497.9
+recb80OwMgNnceA9t,GPQA Diamond,Chemistry,exhausted,93049,0,32768,length,527.2,472650.7
+0f14cd17be174618af6d60227e7dca9f,SuperGPQA,Engineering,pass,7264,994,2809,stop,1145.9,40065.7
+aime2025-21,AIME2025,"Algebra, Geometry",pass,11917,1157,4462,stop,915.5,60278.8
+recA1i5ZAh0Uzclxp,GPQA Diamond,Chemistry,pass,61311,852,19614,stop,559.7,283812.4
+cef9bcc087434cc2b4e354a9baef55eb,SuperGPQA,Engineering,pass,2173,205,1051,stop,847.4,13570.0
+aime2025-07,AIME2025,Combinatorics,pass,27391,1508,9098,stop,519.7,131652.8
+recqGD3fxPCI59vPQ,GPQA Diamond,Physics,pass,21454,594,6658,stop,651.4,95350.8
+9f93aa2cfdb547b5b3a4623f80f7fff6,SuperGPQA,Engineering,pass,875,180,290,stop,577.6,4384.0
+aime2025-22,AIME2025,Number Theory,pass,6979,746,2827,stop,516.0,37019.6
+rechKl68Uc6H7vU0N,GPQA Diamond,Chemistry,exhausted,102204,0,32768,length,713.4,466687.1
+97ad69dda7b2462c98638b79e78aea0b,SuperGPQA,Medicine,pass,2135,9,521,stop,958.1,7781.6
+aime2025-08,AIME2025,Algebra,pass,7375,826,3384,stop,517.8,42264.3
+rec1zl5LvaatzGhFt,GPQA Diamond,Physics,pass,4743,406,1711,stop,707.9,23015.5
+e78e4e539d6f4e379ac140d923d7b1be,SuperGPQA,Military Science,fail,11068,9,2274,stop,505.4,32477.5
+aime2025-09,AIME2025,Algebra,pass,26676,1289,14365,stop,513.5,179593.6
+recTs7qzfJs6kfLUK,GPQA Diamond,Biology,pass,16698,9,5198,stop,928.8,74434.4
+8483667a25e74fdfa3188de4ea734f03,SuperGPQA,Science,fail,93624,628,26082,stop,568.7,379261.8
+aime2025-24,AIME2025,Algebra,pass,13629,1294,5755,stop,763.3,79076.7
+rec32C1ZEapBnCC0E,GPQA Diamond,Physics,pass,4316,261,1629,stop,568.4,22963.7
+e5ed76ef98144f06843125daf1bccd35,SuperGPQA,Science,pass,2555,9,1265,stop,583.9,15737.6
+aime2025-10,AIME2025,Combinatorics,pass,64156,1913,20887,stop,661.9,303869.7
+recZWeueB7lSPR6wN,GPQA Diamond,Chemistry,exhausted,112650,0,32768,length,783.8,493554.8
+fd7924876c4845cd83d95d61dfa0b236,SuperGPQA,Sociology,pass,11680,9,2329,stop,828.1,33429.7
+aime2025-25,AIME2025,Combinatorics,pass,8756,1516,3541,stop,527.5,46446.8
+recVvpD8miVjmmyfe,GPQA Diamond,Biology,exhausted,73787,0,32768,length,516.1,469471.1
+6bfe7d19299d4b3184636e1f51694306,SuperGPQA,Agronomy,pass,20763,555,6152,stop,1027.0,84060.7
+aime2025-26,AIME2025,Combinatorics,pass,23067,1152,7428,stop,533.0,105592.6
+recAAJoHMW45Lv5je,GPQA Diamond,Chemistry,pass,45070,1066,14420,stop,609.2,194508.6
+e1825d70c5844c22933087eafa89e39c,SuperGPQA,Engineering,pass,1670,462,1084,stop,802.4,13082.0
+aime2025-12,AIME2025,Algebra,pass,38449,1444,16627,stop,514.8,220773.4
+reckEnrOPFT9Ru7tW,GPQA Diamond,Chemistry,pass,40672,608,12530,stop,715.2,181802.5
+ab430ac3f18e4e02a2cb3f35498c6b30,SuperGPQA,Engineering,pass,10963,937,4505,stop,724.4,62796.4
+aime2025-27,AIME2025,Geometry,pass,24548,1192,10166,stop,626.0,137678.7
+rec8nshandHARTkrg,GPQA Diamond,Chemistry,pass,100630,9,30451,stop,922.7,426070.1
+e8c5da5ca40647158b0c91dd695c6243,SuperGPQA,Science,pass,5960,517,2358,stop,1053.6,30951.2
+aime2025-13,AIME2025,Combinatorics,pass,96625,1999,27528,stop,535.4,412246.3
+recFaL6j8UMhutXrc,GPQA Diamond,Chemistry,pass,58285,1129,18679,stop,1010.2,254496.0
+05efdc6fb2404ddc8dd5729bb68c74e5,SuperGPQA,Science,pass,1672,458,1204,stop,696.4,14794.5
+aime2025-28,AIME2025,Number Theory,exhausted,55067,0,32768,length,503.0,416457.2
+reczQ4I0VpENdMtIj,GPQA Diamond,Biology,pass,30415,270,6515,stop,845.9,94152.3
+ba52e06cbe1a4310a77a7d12cd1db943,SuperGPQA,Science,pass,15996,264,5853,stop,852.6,76672.6
+aime2025-29,AIME2025,Geometry,pass,54602,1986,24008,stop,604.8,320981.6
+recWxGU8Q4YReJ1tb,GPQA Diamond,Chemistry,exhausted,100990,0,32768,length,1155.0,469972.3
+591a77df21324272914be82ac6583399,SuperGPQA,Science,fail,1615,470,851,stop,1026.0,11110.7
+aime2025-15,AIME2025,Number Theory,exhausted,80520,0,32768,length,482.0,487762.2
+recMicVBcqy1xM1jq,GPQA Diamond,Physics,pass,19842,798,6382,stop,892.1,87630.8
+e780f37a5baa4fe094cd9c157486664d,SuperGPQA,Sociology,fail,20768,318,4862,stop,601.4,70783.0
+aime2025-30,AIME2025,Algebra,pass,76334,1912,28843,stop,634.2,408931.2
+compsec-076,COMPSEC,FreeBSD / librpcsec_gss,pass,12435,10,3143,stop,999.8,45112.4
+compsec-077,COMPSEC,GNU Inetutils telnetd,pass,18144,10,5013,stop,774.7,70577.0
+compsec-078,COMPSEC,Botan,pass,5276,10,1408,stop,770.0,20359.1
+compsec-079,COMPSEC,Botan,pass,42773,10,9990,stop,1039.9,148875.1
+compsec-080,COMPSEC,Botan,pass,15308,9,3439,stop,758.0,52848.9
+compsec-081,COMPSEC,uds-c,pass,37476,10,9295,stop,680.1,137047.8
+compsec-082,COMPSEC,Firebird,pass,54400,9,13510,stop,839.5,203265.3
+compsec-083,COMPSEC,Firebird,pass,43777,10,11329,stop,718.8,167394.5
+compsec-084,COMPSEC,PHP,pass,4067,9,1134,stop,746.1,16992.7
+compsec-085,COMPSEC,PHP PDO Firebird,pass,4972,9,1254,stop,637.9,18737.4
+compsec-086,COMPSEC,libexpat,pass,7287,9,1927,stop,650.5,29033.3
+compsec-087,COMPSEC,Mbed TLS,pass,36801,9,10374,stop,701.9,149365.5
+compsec-088,COMPSEC,Mbed TLS,pass,6651,10,1611,stop,712.0,23745.9
+compsec-089,COMPSEC,Mbed TLS,pass,2230,10,617,stop,559.1,8626.2
+compsec-090,COMPSEC,Linux kernel,pass,24514,10,7180,stop,605.9,96036.1
+compsec-091,COMPSEC,Linux kernel USB gadget storage,pass,3954,9,962,stop,630.9,15032.6
+compsec-092,COMPSEC,Linux kernel AppArmor,pass,17602,10,4390,stop,640.5,64935.5

bench/fit-256k/qa-eval-1786841317.json

@@ -0,0 +1,1028 @@
+{
+  "accuracy": 0.8695652173913043,
+  "bench_version": "0.1.0",
+  "cases": 92,
+  "cases_detail": [
+    {
+      "answer_chars": 447,
+      "completion_tokens": 2584,
+      "finish_reason": "stop",
+      "id": "recNu3MXkvWUzHZr9",
+      "reasoning_chars": 5918,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 32783.790083,
+      "ttft_ms": 427.85825
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 515,
+      "finish_reason": "stop",
+      "id": "001b51d76b4d422988f2c11f104a2c6c",
+      "reasoning_chars": 2275,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 7251.8349579999995,
+      "ttft_ms": 433.050375
+    },
+    {
+      "answer_chars": 313,
+      "completion_tokens": 724,
+      "finish_reason": "stop",
+      "id": "aime2025-01",
+      "reasoning_chars": 1433,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 9103.529083,
+      "ttft_ms": 450.168667
+    },
+    {
+      "answer_chars": 513,
+      "completion_tokens": 5866,
+      "finish_reason": "stop",
+      "id": "recoiTJPGUmzAkief",
+      "reasoning_chars": 25896,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 91367.957375,
+      "ttft_ms": 605.0009160000001
+    },
+    {
+      "answer_chars": 564,
+      "completion_tokens": 2450,
+      "finish_reason": "stop",
+      "id": "b7e20eac98764fb0bf30e8366d951daa",
+      "reasoning_chars": 9569,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 34419.906208,
+      "ttft_ms": 546.1648329999999
+    },
+    {
+      "answer_chars": 11,
+      "completion_tokens": 1017,
+      "finish_reason": "stop",
+      "id": "aime2025-16",
+      "reasoning_chars": 2322,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 13231.10475,
+      "ttft_ms": 479.37304200000005
+    },
+    {
+      "answer_chars": 236,
+      "completion_tokens": 1436,
+      "finish_reason": "stop",
+      "id": "rec4UqStf9WUVif1f",
+      "reasoning_chars": 2958,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 18388.531208,
+      "ttft_ms": 526.239166
+    },
+    {
+      "answer_chars": 121,
+      "completion_tokens": 582,
+      "finish_reason": "stop",
+      "id": "4a1d1780a93f4093b6fb7d3c314cbea8",
+      "reasoning_chars": 2686,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 8713.160249999999,
+      "ttft_ms": 448.203584
+    },
+    {
+      "answer_chars": 1521,
+      "completion_tokens": 25339,
+      "finish_reason": "stop",
+      "id": "aime2025-02",
+      "reasoning_chars": 59852,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 344975.794875,
+      "ttft_ms": 804.663625
+    },
+    {
+      "answer_chars": 447,
+      "completion_tokens": 1130,
+      "finish_reason": "stop",
+      "id": "recgI6tUQ7RLJRWGx",
+      "reasoning_chars": 2735,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 15628.318458000002,
+      "ttft_ms": 728.2525410000001
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1329,
+      "finish_reason": "stop",
+      "id": "6082513c8dba4ec68aa68f1bf5854d09",
+      "reasoning_chars": 6467,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 19068.834666000002,
+      "ttft_ms": 517.797583
+    },
+    {
+      "answer_chars": 550,
+      "completion_tokens": 1048,
+      "finish_reason": "stop",
+      "id": "aime2025-03",
+      "reasoning_chars": 1973,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 12975.841292000001,
+      "ttft_ms": 460.232459
+    },
+    {
+      "answer_chars": 481,
+      "completion_tokens": 1638,
+      "finish_reason": "stop",
+      "id": "recDytVnNYZe2HuUU",
+      "reasoning_chars": 3105,
+      "result": "pass",
+      "source": "GPQA Diamond (modified)",
+      "total_ms": 21416.313124999997,
+      "ttft_ms": 453.71579099999997
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 236,
+      "finish_reason": "stop",
+      "id": "bebf1ed45ae14ad7b4f205f3909cb58a",
+      "reasoning_chars": 971,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 3711.179917,
+      "ttft_ms": 517.5102499999999
+    },
+    {
+      "answer_chars": 1184,
+      "completion_tokens": 4410,
+      "finish_reason": "stop",
+      "id": "aime2025-18",
+      "reasoning_chars": 12226,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 63913.514084,
+      "ttft_ms": 559.792334
+    },
+    {
+      "answer_chars": 497,
+      "completion_tokens": 999,
+      "finish_reason": "stop",
+      "id": "recNFJjE5PPTqVJGv",
+      "reasoning_chars": 1789,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 12477.851333,
+      "ttft_ms": 524.238458
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 4813,
+      "finish_reason": "stop",
+      "id": "7ca71b86327744b78e93185a45bc5cef",
+      "reasoning_chars": 19756,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 68755.92287499999,
+      "ttft_ms": 611.6530409999999
+    },
+    {
+      "answer_chars": 475,
+      "completion_tokens": 1437,
+      "finish_reason": "stop",
+      "id": "aime2025-04",
+      "reasoning_chars": 2768,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 18191.219209,
+      "ttft_ms": 519.2407919999999
+    },
+    {
+      "answer_chars": 554,
+      "completion_tokens": 5025,
+      "finish_reason": "stop",
+      "id": "rec2UlKqC6RFHdcro",
+      "reasoning_chars": 14222,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 70473.393167,
+      "ttft_ms": 523.883333
+    },
+    {
+      "answer_chars": 280,
+      "completion_tokens": 2815,
+      "finish_reason": "stop",
+      "id": "d44b94f7749345a39a65f6312bda8764",
+      "reasoning_chars": 9741,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 44109.07804099999,
+      "ttft_ms": 523.50125
+    },
+    {
+      "answer_chars": 877,
+      "completion_tokens": 3115,
+      "finish_reason": "stop",
+      "id": "aime2025-19",
+      "reasoning_chars": 5097,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 38191.326833,
+      "ttft_ms": 524.416458
+    },
+    {
+      "answer_chars": 449,
+      "completion_tokens": 10584,
+      "finish_reason": "stop",
+      "id": "recv7GsQg3f0fvB1f",
+      "reasoning_chars": 45028,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 157097.680584,
+      "ttft_ms": 557.36275
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 4315,
+      "finish_reason": "stop",
+      "id": "febe406f44d74a40b50bb5b7c69d5dc1",
+      "reasoning_chars": 19449,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 55830.868084,
+      "ttft_ms": 651.646292
+    },
+    {
+      "answer_chars": 1167,
+      "completion_tokens": 7106,
+      "finish_reason": "stop",
+      "id": "aime2025-05",
+      "reasoning_chars": 15671,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 90739.24625,
+      "ttft_ms": 498.489958
+    },
+    {
+      "answer_chars": 632,
+      "completion_tokens": 19556,
+      "finish_reason": "stop",
+      "id": "recrHBEJJoDTV05JR",
+      "reasoning_chars": 62406,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 267669.512625,
+      "ttft_ms": 607.960791
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 473,
+      "finish_reason": "stop",
+      "id": "31950dc80ded400a9181f50626d1f75c",
+      "reasoning_chars": 1502,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 7179.391125,
+      "ttft_ms": 686.416917
+    },
+    {
+      "answer_chars": 767,
+      "completion_tokens": 2042,
+      "finish_reason": "stop",
+      "id": "aime2025-06",
+      "reasoning_chars": 4969,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 27497.851167,
+      "ttft_ms": 492.310542
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recb80OwMgNnceA9t",
+      "reasoning_chars": 93049,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 472650.679125,
+      "ttft_ms": 527.1583330000001
+    },
+    {
+      "answer_chars": 994,
+      "completion_tokens": 2809,
+      "finish_reason": "stop",
+      "id": "0f14cd17be174618af6d60227e7dca9f",
+      "reasoning_chars": 7264,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 40065.691583,
+      "ttft_ms": 1145.8605420000001
+    },
+    {
+      "answer_chars": 1157,
+      "completion_tokens": 4462,
+      "finish_reason": "stop",
+      "id": "aime2025-21",
+      "reasoning_chars": 11917,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 60278.815917,
+      "ttft_ms": 915.503875
+    },
+    {
+      "answer_chars": 852,
+      "completion_tokens": 19614,
+      "finish_reason": "stop",
+      "id": "recA1i5ZAh0Uzclxp",
+      "reasoning_chars": 61311,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 283812.398125,
+      "ttft_ms": 559.6838750000001
+    },
+    {
+      "answer_chars": 205,
+      "completion_tokens": 1051,
+      "finish_reason": "stop",
+      "id": "cef9bcc087434cc2b4e354a9baef55eb",
+      "reasoning_chars": 2173,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 13569.998249999999,
+      "ttft_ms": 847.419667
+    },
+    {
+      "answer_chars": 1508,
+      "completion_tokens": 9098,
+      "finish_reason": "stop",
+      "id": "aime2025-07",
+      "reasoning_chars": 27391,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 131652.772792,
+      "ttft_ms": 519.6575839999999
+    },
+    {
+      "answer_chars": 594,
+      "completion_tokens": 6658,
+      "finish_reason": "stop",
+      "id": "recqGD3fxPCI59vPQ",
+      "reasoning_chars": 21454,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 95350.777208,
+      "ttft_ms": 651.443
+    },
+    {
+      "answer_chars": 180,
+      "completion_tokens": 290,
+      "finish_reason": "stop",
+      "id": "9f93aa2cfdb547b5b3a4623f80f7fff6",
+      "reasoning_chars": 875,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 4384.0284169999995,
+      "ttft_ms": 577.5555
+    },
+    {
+      "answer_chars": 746,
+      "completion_tokens": 2827,
+      "finish_reason": "stop",
+      "id": "aime2025-22",
+      "reasoning_chars": 6979,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 37019.557042,
+      "ttft_ms": 515.9891250000001
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "rechKl68Uc6H7vU0N",
+      "reasoning_chars": 102204,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 466687.117834,
+      "ttft_ms": 713.413334
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 521,
+      "finish_reason": "stop",
+      "id": "97ad69dda7b2462c98638b79e78aea0b",
+      "reasoning_chars": 2135,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 7781.613375,
+      "ttft_ms": 958.0845
+    },
+    {
+      "answer_chars": 826,
+      "completion_tokens": 3384,
+      "finish_reason": "stop",
+      "id": "aime2025-08",
+      "reasoning_chars": 7375,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 42264.326166,
+      "ttft_ms": 517.81475
+    },
+    {
+      "answer_chars": 406,
+      "completion_tokens": 1711,
+      "finish_reason": "stop",
+      "id": "rec1zl5LvaatzGhFt",
+      "reasoning_chars": 4743,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 23015.452041,
+      "ttft_ms": 707.882291
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 2274,
+      "finish_reason": "stop",
+      "id": "e78e4e539d6f4e379ac140d923d7b1be",
+      "reasoning_chars": 11068,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 32477.529124999997,
+      "ttft_ms": 505.41775
+    },
+    {
+      "answer_chars": 1289,
+      "completion_tokens": 14365,
+      "finish_reason": "stop",
+      "id": "aime2025-09",
+      "reasoning_chars": 26676,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 179593.557417,
+      "ttft_ms": 513.5193340000001
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 5198,
+      "finish_reason": "stop",
+      "id": "recTs7qzfJs6kfLUK",
+      "reasoning_chars": 16698,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 74434.430042,
+      "ttft_ms": 928.802959
+    },
+    {
+      "answer_chars": 628,
+      "completion_tokens": 26082,
+      "finish_reason": "stop",
+      "id": "8483667a25e74fdfa3188de4ea734f03",
+      "reasoning_chars": 93624,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 379261.83650000003,
+      "ttft_ms": 568.733958
+    },
+    {
+      "answer_chars": 1294,
+      "completion_tokens": 5755,
+      "finish_reason": "stop",
+      "id": "aime2025-24",
+      "reasoning_chars": 13629,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 79076.662666,
+      "ttft_ms": 763.337083
+    },
+    {
+      "answer_chars": 261,
+      "completion_tokens": 1629,
+      "finish_reason": "stop",
+      "id": "rec32C1ZEapBnCC0E",
+      "reasoning_chars": 4316,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 22963.713708,
+      "ttft_ms": 568.427875
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1265,
+      "finish_reason": "stop",
+      "id": "e5ed76ef98144f06843125daf1bccd35",
+      "reasoning_chars": 2555,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 15737.619083,
+      "ttft_ms": 583.867791
+    },
+    {
+      "answer_chars": 1913,
+      "completion_tokens": 20887,
+      "finish_reason": "stop",
+      "id": "aime2025-10",
+      "reasoning_chars": 64156,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 303869.707375,
+      "ttft_ms": 661.884917
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recZWeueB7lSPR6wN",
+      "reasoning_chars": 112650,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 493554.82691600005,
+      "ttft_ms": 783.787
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 2329,
+      "finish_reason": "stop",
+      "id": "fd7924876c4845cd83d95d61dfa0b236",
+      "reasoning_chars": 11680,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 33429.654125,
+      "ttft_ms": 828.107083
+    },
+    {
+      "answer_chars": 1516,
+      "completion_tokens": 3541,
+      "finish_reason": "stop",
+      "id": "aime2025-25",
+      "reasoning_chars": 8756,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 46446.801375,
+      "ttft_ms": 527.522417
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recVvpD8miVjmmyfe",
+      "reasoning_chars": 73787,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 469471.107458,
+      "ttft_ms": 516.121083
+    },
+    {
+      "answer_chars": 555,
+      "completion_tokens": 6152,
+      "finish_reason": "stop",
+      "id": "6bfe7d19299d4b3184636e1f51694306",
+      "reasoning_chars": 20763,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 84060.745792,
+      "ttft_ms": 1027.043084
+    },
+    {
+      "answer_chars": 1152,
+      "completion_tokens": 7428,
+      "finish_reason": "stop",
+      "id": "aime2025-26",
+      "reasoning_chars": 23067,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 105592.622875,
+      "ttft_ms": 533.0126670000001
+    },
+    {
+      "answer_chars": 1066,
+      "completion_tokens": 14420,
+      "finish_reason": "stop",
+      "id": "recAAJoHMW45Lv5je",
+      "reasoning_chars": 45070,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 194508.552917,
+      "ttft_ms": 609.175417
+    },
+    {
+      "answer_chars": 462,
+      "completion_tokens": 1084,
+      "finish_reason": "stop",
+      "id": "e1825d70c5844c22933087eafa89e39c",
+      "reasoning_chars": 1670,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 13081.987125,
+      "ttft_ms": 802.438625
+    },
+    {
+      "answer_chars": 1444,
+      "completion_tokens": 16627,
+      "finish_reason": "stop",
+      "id": "aime2025-12",
+      "reasoning_chars": 38449,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 220773.386458,
+      "ttft_ms": 514.76325
+    },
+    {
+      "answer_chars": 608,
+      "completion_tokens": 12530,
+      "finish_reason": "stop",
+      "id": "reckEnrOPFT9Ru7tW",
+      "reasoning_chars": 40672,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 181802.454208,
+      "ttft_ms": 715.188833
+    },
+    {
+      "answer_chars": 937,
+      "completion_tokens": 4505,
+      "finish_reason": "stop",
+      "id": "ab430ac3f18e4e02a2cb3f35498c6b30",
+      "reasoning_chars": 10963,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 62796.388667,
+      "ttft_ms": 724.35925
+    },
+    {
+      "answer_chars": 1192,
+      "completion_tokens": 10166,
+      "finish_reason": "stop",
+      "id": "aime2025-27",
+      "reasoning_chars": 24548,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 137678.69475,
+      "ttft_ms": 625.979125
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 30451,
+      "finish_reason": "stop",
+      "id": "rec8nshandHARTkrg",
+      "reasoning_chars": 100630,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 426070.122167,
+      "ttft_ms": 922.731542
+    },
+    {
+      "answer_chars": 517,
+      "completion_tokens": 2358,
+      "finish_reason": "stop",
+      "id": "e8c5da5ca40647158b0c91dd695c6243",
+      "reasoning_chars": 5960,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 30951.200209,
+      "ttft_ms": 1053.5945
+    },
+    {
+      "answer_chars": 1999,
+      "completion_tokens": 27528,
+      "finish_reason": "stop",
+      "id": "aime2025-13",
+      "reasoning_chars": 96625,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 412246.286416,
+      "ttft_ms": 535.400333
+    },
+    {
+      "answer_chars": 1129,
+      "completion_tokens": 18679,
+      "finish_reason": "stop",
+      "id": "recFaL6j8UMhutXrc",
+      "reasoning_chars": 58285,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 254495.964166,
+      "ttft_ms": 1010.184625
+    },
+    {
+      "answer_chars": 458,
+      "completion_tokens": 1204,
+      "finish_reason": "stop",
+      "id": "05efdc6fb2404ddc8dd5729bb68c74e5",
+      "reasoning_chars": 1672,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 14794.488667,
+      "ttft_ms": 696.417292
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-28",
+      "reasoning_chars": 55067,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 416457.180208,
+      "ttft_ms": 502.9515
+    },
+    {
+      "answer_chars": 270,
+      "completion_tokens": 6515,
+      "finish_reason": "stop",
+      "id": "reczQ4I0VpENdMtIj",
+      "reasoning_chars": 30415,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 94152.348459,
+      "ttft_ms": 845.909375
+    },
+    {
+      "answer_chars": 264,
+      "completion_tokens": 5853,
+      "finish_reason": "stop",
+      "id": "ba52e06cbe1a4310a77a7d12cd1db943",
+      "reasoning_chars": 15996,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 76672.621541,
+      "ttft_ms": 852.608916
+    },
+    {
+      "answer_chars": 1986,
+      "completion_tokens": 24008,
+      "finish_reason": "stop",
+      "id": "aime2025-29",
+      "reasoning_chars": 54602,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 320981.63262499997,
+      "ttft_ms": 604.779541
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recWxGU8Q4YReJ1tb",
+      "reasoning_chars": 100990,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 469972.320334,
+      "ttft_ms": 1155.049334
+    },
+    {
+      "answer_chars": 470,
+      "completion_tokens": 851,
+      "finish_reason": "stop",
+      "id": "591a77df21324272914be82ac6583399",
+      "reasoning_chars": 1615,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 11110.698917,
+      "ttft_ms": 1025.992
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-15",
+      "reasoning_chars": 80520,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 487762.17254199996,
+      "ttft_ms": 481.98458300000004
+    },
+    {
+      "answer_chars": 798,
+      "completion_tokens": 6382,
+      "finish_reason": "stop",
+      "id": "recMicVBcqy1xM1jq",
+      "reasoning_chars": 19842,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 87630.761458,
+      "ttft_ms": 892.07225
+    },
+    {
+      "answer_chars": 318,
+      "completion_tokens": 4862,
+      "finish_reason": "stop",
+      "id": "e780f37a5baa4fe094cd9c157486664d",
+      "reasoning_chars": 20768,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 70782.98479199999,
+      "ttft_ms": 601.408
+    },
+    {
+      "answer_chars": 1912,
+      "completion_tokens": 28843,
+      "finish_reason": "stop",
+      "id": "aime2025-30",
+      "reasoning_chars": 76334,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 408931.17208299995,
+      "ttft_ms": 634.181917
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 3143,
+      "finish_reason": "stop",
+      "id": "compsec-076",
+      "reasoning_chars": 12435,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 45112.374041,
+      "ttft_ms": 999.834458
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 5013,
+      "finish_reason": "stop",
+      "id": "compsec-077",
+      "reasoning_chars": 18144,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 70576.978583,
+      "ttft_ms": 774.697792
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1408,
+      "finish_reason": "stop",
+      "id": "compsec-078",
+      "reasoning_chars": 5276,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 20359.128458,
+      "ttft_ms": 770.0350000000001
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 9990,
+      "finish_reason": "stop",
+      "id": "compsec-079",
+      "reasoning_chars": 42773,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 148875.145583,
+      "ttft_ms": 1039.946458
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 3439,
+      "finish_reason": "stop",
+      "id": "compsec-080",
+      "reasoning_chars": 15308,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 52848.938209,
+      "ttft_ms": 758.004584
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 9295,
+      "finish_reason": "stop",
+      "id": "compsec-081",
+      "reasoning_chars": 37476,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 137047.765708,
+      "ttft_ms": 680.141166
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 13510,
+      "finish_reason": "stop",
+      "id": "compsec-082",
+      "reasoning_chars": 54400,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 203265.347791,
+      "ttft_ms": 839.5028329999999
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 11329,
+      "finish_reason": "stop",
+      "id": "compsec-083",
+      "reasoning_chars": 43777,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 167394.472916,
+      "ttft_ms": 718.835458
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1134,
+      "finish_reason": "stop",
+      "id": "compsec-084",
+      "reasoning_chars": 4067,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 16992.731667,
+      "ttft_ms": 746.074417
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1254,
+      "finish_reason": "stop",
+      "id": "compsec-085",
+      "reasoning_chars": 4972,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 18737.374708,
+      "ttft_ms": 637.940667
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1927,
+      "finish_reason": "stop",
+      "id": "compsec-086",
+      "reasoning_chars": 7287,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 29033.311333,
+      "ttft_ms": 650.522333
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 10374,
+      "finish_reason": "stop",
+      "id": "compsec-087",
+      "reasoning_chars": 36801,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 149365.527542,
+      "ttft_ms": 701.871792
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1611,
+      "finish_reason": "stop",
+      "id": "compsec-088",
+      "reasoning_chars": 6651,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 23745.865458,
+      "ttft_ms": 712.036125
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 617,
+      "finish_reason": "stop",
+      "id": "compsec-089",
+      "reasoning_chars": 2230,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 8626.226792000001,
+      "ttft_ms": 559.131834
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 7180,
+      "finish_reason": "stop",
+      "id": "compsec-090",
+      "reasoning_chars": 24514,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 96036.09141699999,
+      "ttft_ms": 605.856667
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 962,
+      "finish_reason": "stop",
+      "id": "compsec-091",
+      "reasoning_chars": 3954,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 15032.648833,
+      "ttft_ms": 630.936291
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 4390,
+      "finish_reason": "stop",
+      "id": "compsec-092",
+      "reasoning_chars": 17602,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 64935.494417,
+      "ttft_ms": 640.4516249999999
+    }
+  ],
+  "config": {
+    "max_tokens": 32768,
+    "seed": 42,
+    "temperature": 0.0
+  },
+  "exhausted": 7,
+  "failed_answered": 5,
+  "model": "fit-256k",
+  "passed": 80
+}

bench/iq4/config-public.txt

@@ -0,0 +1,47 @@
+run:        iq4-mtp
+date:       2026-08-16T06:04:08+02:00
+driver:     bench-baselines-mtp.sh @ 2073ef7
+bs4:        bs4 (our eval harness; not public)
+serve_cmd:  ~/llama.cpp/build-mtp/bin/llama-server -m ~/models/Qwen3.8-27B-IQ4_XS.gguf -md ~/models/Qwen3.8-27B-MTP-ONLY-Q8_0.gguf --spec-draft-n-max 3 -c 131072 -ctk q4_0 -ctv q4_0 -fa on -ngl 99 -ngld 99 --port 8083 --host 0.0.0.0
+--- desktop side ---
+llama.cpp:  22b8e31 (22b8e31)
+model:      15705860064 bytes  ~/models/Qwen3.8-27B-IQ4_XS.gguf
+draft:      4503398688 bytes
+gpu:        NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 250.00 W, 97887 MiB
+driver_ver: 610.57.04
+
+sampler:    temperature 0.0, seed 42, max_tokens 32768 (sent by the eval client in each request body; source: qa-eval-1786862759.json)
+
+--- server /props at run start ---
+{
+  "_note": "SERVER DEFAULTS ONLY — these are what llama-server starts with, NOT what the eval used. Every request overrides them per the 'sampler:' line above (temperature 0, fixed seed). model_ftype is llama.cpp's coarse single-label for the file; for a per-tensor mix it does not describe the quantization.",
+  "build_info": "b93-22b8e31",
+  "model_ftype": "IQ4_XS - 4.25 bpw",
+  "n_ctx": 131072,
+  "total_slots": 4,
+  "sampler_defaults": {
+    "temperature": 1.0,
+    "top_k": 20,
+    "top_p": 0.949999988079071,
+    "min_p": 0.05000000074505806,
+    "repeat_penalty": 1.0,
+    "samplers": [
+      "penalties",
+      "dry",
+      "top_n_sigma",
+      "top_k",
+      "typ_p",
+      "top_p",
+      "min_p",
+      "xtc",
+      "temperature"
+    ]
+  },
+  "chat_template": "<omitted: ~30KB jinja, no reproduction value>"
+}
+
+--- publication note ---
+Scrubbed for publication by sanitize-config.py: hostnames, internal
+repo paths and home paths removed, GPU index omitted (immaterial —
+one card, one power cap), chat template omitted for size. Everything
+that affects the measurement is kept verbatim.

bench/iq4/qa-eval-1786862759.csv

@@ -0,0 +1,93 @@
+id,source,domain,result,reasoning_chars,answer_chars,completion_tokens,finish_reason,ttft_ms,total_ms
+recNu3MXkvWUzHZr9,GPQA Diamond,Physics,pass,4957,516,2256,stop,393.0,24132.8
+001b51d76b4d422988f2c11f104a2c6c,SuperGPQA,Agronomy,pass,2137,9,503,stop,407.6,5462.6
+aime2025-01,AIME2025,Number Theory,pass,1028,418,685,stop,415.3,7030.3
+recoiTJPGUmzAkief,GPQA Diamond,Physics,pass,9843,625,2407,stop,493.0,30989.6
+b7e20eac98764fb0bf30e8366d951daa,SuperGPQA,Engineering,pass,24172,754,6870,stop,588.7,75237.4
+aime2025-16,AIME2025,Geometry,pass,2413,703,1491,stop,484.5,15082.4
+rec4UqStf9WUVif1f,GPQA Diamond,Physics,pass,1498,435,1011,stop,449.6,10234.5
+4a1d1780a93f4093b6fb7d3c314cbea8,SuperGPQA,Engineering,fail,27294,9,5940,stop,427.6,73045.9
+aime2025-02,AIME2025,Geometry,exhausted,81510,0,32768,length,817.1,378020.4
+recgI6tUQ7RLJRWGx,GPQA Diamond,Physics,pass,3664,510,1460,stop,759.0,16712.4
+6082513c8dba4ec68aa68f1bf5854d09,SuperGPQA,Law,pass,6005,9,1268,stop,435.1,14860.8
+aime2025-03,AIME2025,Combinatorics,pass,1693,495,881,stop,436.7,9080.4
+recDytVnNYZe2HuUU,GPQA Diamond (modified),Physics,pass,5964,372,2983,stop,444.4,32639.1
+bebf1ed45ae14ad7b4f205f3909cb58a,SuperGPQA,Literature and Arts,pass,671,9,221,stop,470.4,2749.0
+aime2025-18,AIME2025,Combinatorics,pass,11367,1365,4077,stop,498.0,46770.9
+recNFJjE5PPTqVJGv,GPQA Diamond,Chemistry,pass,1295,802,958,stop,472.2,9654.1
+7ca71b86327744b78e93185a45bc5cef,SuperGPQA,Medicine,pass,16963,9,4017,stop,452.5,47203.3
+aime2025-04,AIME2025,Algebra,pass,2886,446,1430,stop,457.1,15442.7
+rec2UlKqC6RFHdcro,GPQA Diamond,Physics,pass,3315,456,1388,stop,463.0,15559.5
+d44b94f7749345a39a65f6312bda8764,SuperGPQA,Engineering,pass,2597,477,875,stop,467.9,11410.3
+aime2025-19,AIME2025,Algebra,pass,3438,1136,2293,stop,496.2,23392.6
+recv7GsQg3f0fvB1f,GPQA Diamond,Chemistry,pass,29017,452,6919,stop,548.6,84313.7
+febe406f44d74a40b50bb5b7c69d5dc1,SuperGPQA,Sociology,pass,9216,9,2071,stop,535.5,23664.9
+aime2025-05,AIME2025,"Combinatorics, Number Theory",pass,5638,1555,2738,stop,453.5,28895.1
+recrHBEJJoDTV05JR,GPQA Diamond,Chemistry,pass,52703,561,16217,stop,524.0,184967.1
+31950dc80ded400a9181f50626d1f75c,SuperGPQA,Science,pass,1132,309,558,stop,606.8,6029.9
+aime2025-06,AIME2025,Geometry,pass,2065,1107,1186,stop,432.6,13148.9
+recb80OwMgNnceA9t,GPQA Diamond,Chemistry,exhausted,89248,0,32768,length,456.4,392565.1
+0f14cd17be174618af6d60227e7dca9f,SuperGPQA,Engineering,pass,21150,515,7332,stop,1079.8,86715.0
+aime2025-21,AIME2025,"Algebra, Geometry",pass,10531,1344,4022,stop,859.3,44826.2
+recA1i5ZAh0Uzclxp,GPQA Diamond,Chemistry,pass,81860,498,25758,stop,540.1,311519.3
+cef9bcc087434cc2b4e354a9baef55eb,SuperGPQA,Engineering,pass,1271,511,963,stop,864.7,10116.9
+aime2025-07,AIME2025,Combinatorics,pass,18603,1784,6471,stop,451.0,77894.8
+recqGD3fxPCI59vPQ,GPQA Diamond,Physics,pass,4876,566,1749,stop,617.5,19604.5
+9f93aa2cfdb547b5b3a4623f80f7fff6,SuperGPQA,Engineering,pass,893,143,277,stop,434.1,3316.8
+aime2025-22,AIME2025,Number Theory,pass,4136,1001,1834,stop,446.5,19451.6
+rechKl68Uc6H7vU0N,GPQA Diamond,Chemistry,exhausted,100408,0,32768,length,516.4,387256.3
+97ad69dda7b2462c98638b79e78aea0b,SuperGPQA,Medicine,pass,12824,9,3196,stop,782.8,34916.9
+aime2025-08,AIME2025,Algebra,pass,5916,877,2936,stop,456.2,30667.1
+rec1zl5LvaatzGhFt,GPQA Diamond,Physics,pass,3983,378,1633,stop,592.1,17666.5
+e78e4e539d6f4e379ac140d923d7b1be,SuperGPQA,Military Science,fail,10468,9,2148,stop,454.9,25005.6
+aime2025-09,AIME2025,Algebra,pass,48410,1936,21490,stop,475.8,241600.9
+recTs7qzfJs6kfLUK,GPQA Diamond,Biology,pass,31295,9,10379,stop,893.6,118889.9
+8483667a25e74fdfa3188de4ea734f03,SuperGPQA,Science,fail,67959,860,19649,stop,594.3,236243.3
+aime2025-24,AIME2025,Algebra,pass,21164,937,8856,stop,600.5,100464.6
+rec32C1ZEapBnCC0E,GPQA Diamond,Physics,pass,5929,383,2173,stop,567.8,24801.6
+e5ed76ef98144f06843125daf1bccd35,SuperGPQA,Science,pass,2202,9,992,stop,477.3,10312.1
+aime2025-10,AIME2025,Combinatorics,pass,95696,1602,30868,stop,643.4,370198.4
+recZWeueB7lSPR6wN,GPQA Diamond,Chemistry,exhausted,109416,0,32768,length,795.2,411463.3
+fd7924876c4845cd83d95d61dfa0b236,SuperGPQA,Sociology,pass,8388,9,1594,stop,771.0,19853.9
+aime2025-25,AIME2025,Combinatorics,pass,16903,1536,5938,stop,449.6,68168.5
+recVvpD8miVjmmyfe,GPQA Diamond,Biology,exhausted,72883,0,32768,length,462.6,384184.0
+6bfe7d19299d4b3184636e1f51694306,SuperGPQA,Agronomy,pass,36420,461,11118,stop,906.8,127574.6
+aime2025-26,AIME2025,Combinatorics,pass,14244,822,4804,stop,539.2,57414.4
+recAAJoHMW45Lv5je,GPQA Diamond,Chemistry,pass,64264,846,20175,stop,571.4,231093.5
+e1825d70c5844c22933087eafa89e39c,SuperGPQA,Engineering,pass,1788,455,1242,stop,915.3,12656.2
+aime2025-12,AIME2025,Algebra,pass,16030,1336,6755,stop,499.8,75467.9
+reckEnrOPFT9Ru7tW,GPQA Diamond,Chemistry,pass,73378,666,22636,stop,528.5,273153.8
+ab430ac3f18e4e02a2cb3f35498c6b30,SuperGPQA,Engineering,pass,8989,1167,3751,stop,719.8,43082.8
+aime2025-27,AIME2025,Geometry,pass,30128,1215,11718,stop,573.6,134166.7
+rec8nshandHARTkrg,GPQA Diamond,Chemistry,exhausted,106637,0,32768,length,812.0,388554.8
+e8c5da5ca40647158b0c91dd695c6243,SuperGPQA,Science,pass,6663,408,2704,stop,1001.0,30449.6
+aime2025-13,AIME2025,Combinatorics,pass,92853,1893,25789,stop,495.1,320011.8
+recFaL6j8UMhutXrc,GPQA Diamond,Chemistry,pass,42251,856,14372,stop,861.8,162496.3
+05efdc6fb2404ddc8dd5729bb68c74e5,SuperGPQA,Science,pass,1437,572,1077,stop,633.2,10957.5
+aime2025-28,AIME2025,Number Theory,exhausted,63844,0,32768,length,474.7,374460.2
+reczQ4I0VpENdMtIj,GPQA Diamond,Biology,pass,35113,9,7554,stop,814.7,89780.7
+ba52e06cbe1a4310a77a7d12cd1db943,SuperGPQA,Science,pass,10077,9,3691,stop,749.3,39057.6
+aime2025-29,AIME2025,Geometry,pass,37424,2107,20322,stop,543.7,215322.8
+recWxGU8Q4YReJ1tb,GPQA Diamond,Chemistry,exhausted,93893,0,32768,length,969.6,396274.6
+591a77df21324272914be82ac6583399,SuperGPQA,Science,fail,2183,427,1149,stop,866.8,12314.4
+aime2025-15,AIME2025,Number Theory,exhausted,76942,0,32768,length,464.4,396845.8
+recMicVBcqy1xM1jq,GPQA Diamond,Physics,pass,23534,921,8865,stop,876.7,100289.9
+e780f37a5baa4fe094cd9c157486664d,SuperGPQA,Sociology,fail,14690,182,3321,stop,539.2,39935.4
+aime2025-30,AIME2025,Algebra,pass,70913,1460,28421,stop,483.0,333739.9
+compsec-076,COMPSEC,FreeBSD / librpcsec_gss,pass,7182,10,1786,stop,857.5,22333.5
+compsec-077,COMPSEC,GNU Inetutils telnetd,pass,29427,10,8356,stop,565.5,97578.7
+compsec-078,COMPSEC,Botan,pass,3729,10,1016,stop,645.0,11978.9
+compsec-079,COMPSEC,Botan,pass,25159,10,5771,stop,650.8,72865.6
+compsec-080,COMPSEC,Botan,pass,24033,9,5416,stop,553.4,68310.6
+compsec-081,COMPSEC,uds-c,pass,64895,10,16465,stop,666.2,201030.2
+compsec-082,COMPSEC,Firebird,pass,48679,9,12261,stop,830.3,153512.6
+compsec-083,COMPSEC,Firebird,pass,65547,10,17304,stop,697.8,217749.6
+compsec-084,COMPSEC,PHP,pass,2434,9,693,stop,717.0,8961.6
+compsec-085,COMPSEC,PHP PDO Firebird,pass,24580,9,6334,stop,501.2,79052.3
+compsec-086,COMPSEC,libexpat,pass,4644,9,1200,stop,662.6,14616.3
+compsec-087,COMPSEC,Mbed TLS,pass,43808,9,12658,stop,601.6,152057.4
+compsec-088,COMPSEC,Mbed TLS,pass,4418,10,1151,stop,634.3,14193.8
+compsec-089,COMPSEC,Mbed TLS,pass,2212,10,649,stop,511.9,7471.2
+compsec-090,COMPSEC,Linux kernel,pass,2576,10,780,stop,604.6,8964.2
+compsec-091,COMPSEC,Linux kernel USB gadget storage,pass,2311,9,622,stop,499.0,7727.5
+compsec-092,COMPSEC,Linux kernel AppArmor,pass,9552,10,2318,stop,538.3,28152.6

bench/iq4/qa-eval-1786862759.json

@@ -0,0 +1,1028 @@
+{
+  "accuracy": 0.8478260869565217,
+  "bench_version": "0.1.0",
+  "cases": 92,
+  "cases_detail": [
+    {
+      "answer_chars": 516,
+      "completion_tokens": 2256,
+      "finish_reason": "stop",
+      "id": "recNu3MXkvWUzHZr9",
+      "reasoning_chars": 4957,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 24132.795375,
+      "ttft_ms": 392.962333
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 503,
+      "finish_reason": "stop",
+      "id": "001b51d76b4d422988f2c11f104a2c6c",
+      "reasoning_chars": 2137,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 5462.6095829999995,
+      "ttft_ms": 407.617083
+    },
+    {
+      "answer_chars": 418,
+      "completion_tokens": 685,
+      "finish_reason": "stop",
+      "id": "aime2025-01",
+      "reasoning_chars": 1028,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 7030.3345,
+      "ttft_ms": 415.26629199999996
+    },
+    {
+      "answer_chars": 625,
+      "completion_tokens": 2407,
+      "finish_reason": "stop",
+      "id": "recoiTJPGUmzAkief",
+      "reasoning_chars": 9843,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 30989.649375,
+      "ttft_ms": 492.995792
+    },
+    {
+      "answer_chars": 754,
+      "completion_tokens": 6870,
+      "finish_reason": "stop",
+      "id": "b7e20eac98764fb0bf30e8366d951daa",
+      "reasoning_chars": 24172,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 75237.439625,
+      "ttft_ms": 588.689208
+    },
+    {
+      "answer_chars": 703,
+      "completion_tokens": 1491,
+      "finish_reason": "stop",
+      "id": "aime2025-16",
+      "reasoning_chars": 2413,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 15082.376083,
+      "ttft_ms": 484.47216599999996
+    },
+    {
+      "answer_chars": 435,
+      "completion_tokens": 1011,
+      "finish_reason": "stop",
+      "id": "rec4UqStf9WUVif1f",
+      "reasoning_chars": 1498,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 10234.532584,
+      "ttft_ms": 449.586
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 5940,
+      "finish_reason": "stop",
+      "id": "4a1d1780a93f4093b6fb7d3c314cbea8",
+      "reasoning_chars": 27294,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 73045.92358300001,
+      "ttft_ms": 427.620041
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-02",
+      "reasoning_chars": 81510,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 378020.357958,
+      "ttft_ms": 817.065292
+    },
+    {
+      "answer_chars": 510,
+      "completion_tokens": 1460,
+      "finish_reason": "stop",
+      "id": "recgI6tUQ7RLJRWGx",
+      "reasoning_chars": 3664,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 16712.409417,
+      "ttft_ms": 759.0152919999999
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1268,
+      "finish_reason": "stop",
+      "id": "6082513c8dba4ec68aa68f1bf5854d09",
+      "reasoning_chars": 6005,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 14860.7845,
+      "ttft_ms": 435.130042
+    },
+    {
+      "answer_chars": 495,
+      "completion_tokens": 881,
+      "finish_reason": "stop",
+      "id": "aime2025-03",
+      "reasoning_chars": 1693,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 9080.410625,
+      "ttft_ms": 436.679416
+    },
+    {
+      "answer_chars": 372,
+      "completion_tokens": 2983,
+      "finish_reason": "stop",
+      "id": "recDytVnNYZe2HuUU",
+      "reasoning_chars": 5964,
+      "result": "pass",
+      "source": "GPQA Diamond (modified)",
+      "total_ms": 32639.136042,
+      "ttft_ms": 444.372042
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 221,
+      "finish_reason": "stop",
+      "id": "bebf1ed45ae14ad7b4f205f3909cb58a",
+      "reasoning_chars": 671,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 2749.034125,
+      "ttft_ms": 470.37320900000003
+    },
+    {
+      "answer_chars": 1365,
+      "completion_tokens": 4077,
+      "finish_reason": "stop",
+      "id": "aime2025-18",
+      "reasoning_chars": 11367,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 46770.856791,
+      "ttft_ms": 497.998541
+    },
+    {
+      "answer_chars": 802,
+      "completion_tokens": 958,
+      "finish_reason": "stop",
+      "id": "recNFJjE5PPTqVJGv",
+      "reasoning_chars": 1295,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 9654.135042,
+      "ttft_ms": 472.195209
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 4017,
+      "finish_reason": "stop",
+      "id": "7ca71b86327744b78e93185a45bc5cef",
+      "reasoning_chars": 16963,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 47203.322917,
+      "ttft_ms": 452.497209
+    },
+    {
+      "answer_chars": 446,
+      "completion_tokens": 1430,
+      "finish_reason": "stop",
+      "id": "aime2025-04",
+      "reasoning_chars": 2886,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 15442.723167,
+      "ttft_ms": 457.081167
+    },
+    {
+      "answer_chars": 456,
+      "completion_tokens": 1388,
+      "finish_reason": "stop",
+      "id": "rec2UlKqC6RFHdcro",
+      "reasoning_chars": 3315,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 15559.484958000001,
+      "ttft_ms": 463.016042
+    },
+    {
+      "answer_chars": 477,
+      "completion_tokens": 875,
+      "finish_reason": "stop",
+      "id": "d44b94f7749345a39a65f6312bda8764",
+      "reasoning_chars": 2597,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 11410.305333,
+      "ttft_ms": 467.892791
+    },
+    {
+      "answer_chars": 1136,
+      "completion_tokens": 2293,
+      "finish_reason": "stop",
+      "id": "aime2025-19",
+      "reasoning_chars": 3438,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 23392.591167,
+      "ttft_ms": 496.226208
+    },
+    {
+      "answer_chars": 452,
+      "completion_tokens": 6919,
+      "finish_reason": "stop",
+      "id": "recv7GsQg3f0fvB1f",
+      "reasoning_chars": 29017,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 84313.740916,
+      "ttft_ms": 548.5755
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 2071,
+      "finish_reason": "stop",
+      "id": "febe406f44d74a40b50bb5b7c69d5dc1",
+      "reasoning_chars": 9216,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 23664.936333,
+      "ttft_ms": 535.487458
+    },
+    {
+      "answer_chars": 1555,
+      "completion_tokens": 2738,
+      "finish_reason": "stop",
+      "id": "aime2025-05",
+      "reasoning_chars": 5638,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 28895.086958,
+      "ttft_ms": 453.45441600000004
+    },
+    {
+      "answer_chars": 561,
+      "completion_tokens": 16217,
+      "finish_reason": "stop",
+      "id": "recrHBEJJoDTV05JR",
+      "reasoning_chars": 52703,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 184967.125917,
+      "ttft_ms": 523.957125
+    },
+    {
+      "answer_chars": 309,
+      "completion_tokens": 558,
+      "finish_reason": "stop",
+      "id": "31950dc80ded400a9181f50626d1f75c",
+      "reasoning_chars": 1132,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 6029.928166999999,
+      "ttft_ms": 606.811
+    },
+    {
+      "answer_chars": 1107,
+      "completion_tokens": 1186,
+      "finish_reason": "stop",
+      "id": "aime2025-06",
+      "reasoning_chars": 2065,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 13148.948459,
+      "ttft_ms": 432.645667
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recb80OwMgNnceA9t",
+      "reasoning_chars": 89248,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 392565.071625,
+      "ttft_ms": 456.39854099999997
+    },
+    {
+      "answer_chars": 515,
+      "completion_tokens": 7332,
+      "finish_reason": "stop",
+      "id": "0f14cd17be174618af6d60227e7dca9f",
+      "reasoning_chars": 21150,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 86714.99533399999,
+      "ttft_ms": 1079.799
+    },
+    {
+      "answer_chars": 1344,
+      "completion_tokens": 4022,
+      "finish_reason": "stop",
+      "id": "aime2025-21",
+      "reasoning_chars": 10531,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 44826.154,
+      "ttft_ms": 859.2632500000001
+    },
+    {
+      "answer_chars": 498,
+      "completion_tokens": 25758,
+      "finish_reason": "stop",
+      "id": "recA1i5ZAh0Uzclxp",
+      "reasoning_chars": 81860,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 311519.27575000003,
+      "ttft_ms": 540.1415830000001
+    },
+    {
+      "answer_chars": 511,
+      "completion_tokens": 963,
+      "finish_reason": "stop",
+      "id": "cef9bcc087434cc2b4e354a9baef55eb",
+      "reasoning_chars": 1271,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 10116.862958,
+      "ttft_ms": 864.6772080000001
+    },
+    {
+      "answer_chars": 1784,
+      "completion_tokens": 6471,
+      "finish_reason": "stop",
+      "id": "aime2025-07",
+      "reasoning_chars": 18603,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 77894.772,
+      "ttft_ms": 451.046
+    },
+    {
+      "answer_chars": 566,
+      "completion_tokens": 1749,
+      "finish_reason": "stop",
+      "id": "recqGD3fxPCI59vPQ",
+      "reasoning_chars": 4876,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 19604.495,
+      "ttft_ms": 617.5035
+    },
+    {
+      "answer_chars": 143,
+      "completion_tokens": 277,
+      "finish_reason": "stop",
+      "id": "9f93aa2cfdb547b5b3a4623f80f7fff6",
+      "reasoning_chars": 893,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 3316.814,
+      "ttft_ms": 434.074875
+    },
+    {
+      "answer_chars": 1001,
+      "completion_tokens": 1834,
+      "finish_reason": "stop",
+      "id": "aime2025-22",
+      "reasoning_chars": 4136,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 19451.551959,
+      "ttft_ms": 446.52854199999996
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "rechKl68Uc6H7vU0N",
+      "reasoning_chars": 100408,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 387256.25325,
+      "ttft_ms": 516.356917
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 3196,
+      "finish_reason": "stop",
+      "id": "97ad69dda7b2462c98638b79e78aea0b",
+      "reasoning_chars": 12824,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 34916.925708,
+      "ttft_ms": 782.8485830000001
+    },
+    {
+      "answer_chars": 877,
+      "completion_tokens": 2936,
+      "finish_reason": "stop",
+      "id": "aime2025-08",
+      "reasoning_chars": 5916,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 30667.065749999998,
+      "ttft_ms": 456.21937499999996
+    },
+    {
+      "answer_chars": 378,
+      "completion_tokens": 1633,
+      "finish_reason": "stop",
+      "id": "rec1zl5LvaatzGhFt",
+      "reasoning_chars": 3983,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 17666.456292,
+      "ttft_ms": 592.073625
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 2148,
+      "finish_reason": "stop",
+      "id": "e78e4e539d6f4e379ac140d923d7b1be",
+      "reasoning_chars": 10468,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 25005.553290999997,
+      "ttft_ms": 454.879333
+    },
+    {
+      "answer_chars": 1936,
+      "completion_tokens": 21490,
+      "finish_reason": "stop",
+      "id": "aime2025-09",
+      "reasoning_chars": 48410,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 241600.916375,
+      "ttft_ms": 475.80062499999997
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 10379,
+      "finish_reason": "stop",
+      "id": "recTs7qzfJs6kfLUK",
+      "reasoning_chars": 31295,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 118889.904167,
+      "ttft_ms": 893.556292
+    },
+    {
+      "answer_chars": 860,
+      "completion_tokens": 19649,
+      "finish_reason": "stop",
+      "id": "8483667a25e74fdfa3188de4ea734f03",
+      "reasoning_chars": 67959,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 236243.349833,
+      "ttft_ms": 594.271791
+    },
+    {
+      "answer_chars": 937,
+      "completion_tokens": 8856,
+      "finish_reason": "stop",
+      "id": "aime2025-24",
+      "reasoning_chars": 21164,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 100464.641709,
+      "ttft_ms": 600.457792
+    },
+    {
+      "answer_chars": 383,
+      "completion_tokens": 2173,
+      "finish_reason": "stop",
+      "id": "rec32C1ZEapBnCC0E",
+      "reasoning_chars": 5929,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 24801.57075,
+      "ttft_ms": 567.779417
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 992,
+      "finish_reason": "stop",
+      "id": "e5ed76ef98144f06843125daf1bccd35",
+      "reasoning_chars": 2202,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 10312.147333,
+      "ttft_ms": 477.321958
+    },
+    {
+      "answer_chars": 1602,
+      "completion_tokens": 30868,
+      "finish_reason": "stop",
+      "id": "aime2025-10",
+      "reasoning_chars": 95696,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 370198.370458,
+      "ttft_ms": 643.408833
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recZWeueB7lSPR6wN",
+      "reasoning_chars": 109416,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 411463.27120799996,
+      "ttft_ms": 795.192208
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1594,
+      "finish_reason": "stop",
+      "id": "fd7924876c4845cd83d95d61dfa0b236",
+      "reasoning_chars": 8388,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 19853.921125,
+      "ttft_ms": 771.021292
+    },
+    {
+      "answer_chars": 1536,
+      "completion_tokens": 5938,
+      "finish_reason": "stop",
+      "id": "aime2025-25",
+      "reasoning_chars": 16903,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 68168.48787499999,
+      "ttft_ms": 449.57216700000004
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recVvpD8miVjmmyfe",
+      "reasoning_chars": 72883,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 384184.031,
+      "ttft_ms": 462.625042
+    },
+    {
+      "answer_chars": 461,
+      "completion_tokens": 11118,
+      "finish_reason": "stop",
+      "id": "6bfe7d19299d4b3184636e1f51694306",
+      "reasoning_chars": 36420,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 127574.636375,
+      "ttft_ms": 906.764208
+    },
+    {
+      "answer_chars": 822,
+      "completion_tokens": 4804,
+      "finish_reason": "stop",
+      "id": "aime2025-26",
+      "reasoning_chars": 14244,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 57414.397625,
+      "ttft_ms": 539.1835000000001
+    },
+    {
+      "answer_chars": 846,
+      "completion_tokens": 20175,
+      "finish_reason": "stop",
+      "id": "recAAJoHMW45Lv5je",
+      "reasoning_chars": 64264,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 231093.521209,
+      "ttft_ms": 571.3779169999999
+    },
+    {
+      "answer_chars": 455,
+      "completion_tokens": 1242,
+      "finish_reason": "stop",
+      "id": "e1825d70c5844c22933087eafa89e39c",
+      "reasoning_chars": 1788,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 12656.155209,
+      "ttft_ms": 915.341375
+    },
+    {
+      "answer_chars": 1336,
+      "completion_tokens": 6755,
+      "finish_reason": "stop",
+      "id": "aime2025-12",
+      "reasoning_chars": 16030,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 75467.93075,
+      "ttft_ms": 499.806792
+    },
+    {
+      "answer_chars": 666,
+      "completion_tokens": 22636,
+      "finish_reason": "stop",
+      "id": "reckEnrOPFT9Ru7tW",
+      "reasoning_chars": 73378,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 273153.801333,
+      "ttft_ms": 528.4896669999999
+    },
+    {
+      "answer_chars": 1167,
+      "completion_tokens": 3751,
+      "finish_reason": "stop",
+      "id": "ab430ac3f18e4e02a2cb3f35498c6b30",
+      "reasoning_chars": 8989,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 43082.825584,
+      "ttft_ms": 719.753375
+    },
+    {
+      "answer_chars": 1215,
+      "completion_tokens": 11718,
+      "finish_reason": "stop",
+      "id": "aime2025-27",
+      "reasoning_chars": 30128,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 134166.749541,
+      "ttft_ms": 573.593291
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "rec8nshandHARTkrg",
+      "reasoning_chars": 106637,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 388554.750916,
+      "ttft_ms": 811.9523330000001
+    },
+    {
+      "answer_chars": 408,
+      "completion_tokens": 2704,
+      "finish_reason": "stop",
+      "id": "e8c5da5ca40647158b0c91dd695c6243",
+      "reasoning_chars": 6663,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 30449.635791999997,
+      "ttft_ms": 1000.963875
+    },
+    {
+      "answer_chars": 1893,
+      "completion_tokens": 25789,
+      "finish_reason": "stop",
+      "id": "aime2025-13",
+      "reasoning_chars": 92853,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 320011.844458,
+      "ttft_ms": 495.081
+    },
+    {
+      "answer_chars": 856,
+      "completion_tokens": 14372,
+      "finish_reason": "stop",
+      "id": "recFaL6j8UMhutXrc",
+      "reasoning_chars": 42251,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 162496.277167,
+      "ttft_ms": 861.797375
+    },
+    {
+      "answer_chars": 572,
+      "completion_tokens": 1077,
+      "finish_reason": "stop",
+      "id": "05efdc6fb2404ddc8dd5729bb68c74e5",
+      "reasoning_chars": 1437,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 10957.472834,
+      "ttft_ms": 633.23125
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-28",
+      "reasoning_chars": 63844,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 374460.237125,
+      "ttft_ms": 474.684459
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 7554,
+      "finish_reason": "stop",
+      "id": "reczQ4I0VpENdMtIj",
+      "reasoning_chars": 35113,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 89780.66775000001,
+      "ttft_ms": 814.702333
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 3691,
+      "finish_reason": "stop",
+      "id": "ba52e06cbe1a4310a77a7d12cd1db943",
+      "reasoning_chars": 10077,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 39057.569,
+      "ttft_ms": 749.285625
+    },
+    {
+      "answer_chars": 2107,
+      "completion_tokens": 20322,
+      "finish_reason": "stop",
+      "id": "aime2025-29",
+      "reasoning_chars": 37424,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 215322.795167,
+      "ttft_ms": 543.663833
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recWxGU8Q4YReJ1tb",
+      "reasoning_chars": 93893,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 396274.570042,
+      "ttft_ms": 969.61275
+    },
+    {
+      "answer_chars": 427,
+      "completion_tokens": 1149,
+      "finish_reason": "stop",
+      "id": "591a77df21324272914be82ac6583399",
+      "reasoning_chars": 2183,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 12314.380625,
+      "ttft_ms": 866.8087089999999
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-15",
+      "reasoning_chars": 76942,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 396845.81741699995,
+      "ttft_ms": 464.445417
+    },
+    {
+      "answer_chars": 921,
+      "completion_tokens": 8865,
+      "finish_reason": "stop",
+      "id": "recMicVBcqy1xM1jq",
+      "reasoning_chars": 23534,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 100289.881333,
+      "ttft_ms": 876.686458
+    },
+    {
+      "answer_chars": 182,
+      "completion_tokens": 3321,
+      "finish_reason": "stop",
+      "id": "e780f37a5baa4fe094cd9c157486664d",
+      "reasoning_chars": 14690,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 39935.412625,
+      "ttft_ms": 539.244791
+    },
+    {
+      "answer_chars": 1460,
+      "completion_tokens": 28421,
+      "finish_reason": "stop",
+      "id": "aime2025-30",
+      "reasoning_chars": 70913,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 333739.869917,
+      "ttft_ms": 483.025875
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1786,
+      "finish_reason": "stop",
+      "id": "compsec-076",
+      "reasoning_chars": 7182,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 22333.537041,
+      "ttft_ms": 857.504375
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 8356,
+      "finish_reason": "stop",
+      "id": "compsec-077",
+      "reasoning_chars": 29427,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 97578.713708,
+      "ttft_ms": 565.4876250000001
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1016,
+      "finish_reason": "stop",
+      "id": "compsec-078",
+      "reasoning_chars": 3729,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 11978.9155,
+      "ttft_ms": 645.03025
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 5771,
+      "finish_reason": "stop",
+      "id": "compsec-079",
+      "reasoning_chars": 25159,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 72865.583541,
+      "ttft_ms": 650.810625
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 5416,
+      "finish_reason": "stop",
+      "id": "compsec-080",
+      "reasoning_chars": 24033,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 68310.592584,
+      "ttft_ms": 553.357459
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 16465,
+      "finish_reason": "stop",
+      "id": "compsec-081",
+      "reasoning_chars": 64895,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 201030.182166,
+      "ttft_ms": 666.184166
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 12261,
+      "finish_reason": "stop",
+      "id": "compsec-082",
+      "reasoning_chars": 48679,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 153512.559875,
+      "ttft_ms": 830.349083
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 17304,
+      "finish_reason": "stop",
+      "id": "compsec-083",
+      "reasoning_chars": 65547,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 217749.60100000002,
+      "ttft_ms": 697.829041
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 693,
+      "finish_reason": "stop",
+      "id": "compsec-084",
+      "reasoning_chars": 2434,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 8961.636875,
+      "ttft_ms": 717.026917
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 6334,
+      "finish_reason": "stop",
+      "id": "compsec-085",
+      "reasoning_chars": 24580,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 79052.252583,
+      "ttft_ms": 501.243542
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1200,
+      "finish_reason": "stop",
+      "id": "compsec-086",
+      "reasoning_chars": 4644,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 14616.335791000001,
+      "ttft_ms": 662.555333
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 12658,
+      "finish_reason": "stop",
+      "id": "compsec-087",
+      "reasoning_chars": 43808,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 152057.35204199998,
+      "ttft_ms": 601.607334
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1151,
+      "finish_reason": "stop",
+      "id": "compsec-088",
+      "reasoning_chars": 4418,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 14193.834041,
+      "ttft_ms": 634.264666
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 649,
+      "finish_reason": "stop",
+      "id": "compsec-089",
+      "reasoning_chars": 2212,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 7471.202291,
+      "ttft_ms": 511.90250000000003
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 780,
+      "finish_reason": "stop",
+      "id": "compsec-090",
+      "reasoning_chars": 2576,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 8964.221792,
+      "ttft_ms": 604.6085
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 622,
+      "finish_reason": "stop",
+      "id": "compsec-091",
+      "reasoning_chars": 2311,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 7727.497917,
+      "ttft_ms": 499.007833
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 2318,
+      "finish_reason": "stop",
+      "id": "compsec-092",
+      "reasoning_chars": 9552,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 28152.645,
+      "ttft_ms": 538.3396660000001
+    }
+  ],
+  "config": {
+    "max_tokens": 32768,
+    "seed": 42,
+    "temperature": 0.0
+  },
+  "exhausted": 9,
+  "failed_answered": 5,
+  "model": "iq4-mtp",
+  "passed": 78
+}

bench/q8/config-public.txt

@@ -0,0 +1,47 @@
+run:        q8-mtp
+date:       2026-08-16T02:52:51+02:00
+driver:     bench-baselines-mtp.sh @ 4d81187
+bs4:        bs4 (our eval harness; not public)
+serve_cmd:  ~/llama.cpp/build-mtp/bin/llama-server -m ~/models/Qwen3.8-27B-Q8_0.gguf -md ~/models/Qwen3.8-27B-MTP-ONLY-Q8_0.gguf --spec-draft-n-max 3 -c 131072 -ctk q4_0 -ctv q4_0 -fa on -ngl 99 -ngld 99 --port 8083 --host 0.0.0.0
+--- desktop side ---
+llama.cpp:  22b8e31 (22b8e31)
+model:      29047085024 bytes  ~/models/Qwen3.8-27B-Q8_0.gguf
+draft:      4503398688 bytes
+gpu:        NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 250.00 W, 97887 MiB
+driver_ver: 610.57.04
+
+sampler:    temperature 0.0, seed 42, max_tokens 32768 (sent by the eval client in each request body; source: qa-eval-1786853047.json)
+
+--- server /props at run start ---
+{
+  "_note": "SERVER DEFAULTS ONLY — these are what llama-server starts with, NOT what the eval used. Every request overrides them per the 'sampler:' line above (temperature 0, fixed seed). model_ftype is llama.cpp's coarse single-label for the file; for a per-tensor mix it does not describe the quantization.",
+  "build_info": "b93-22b8e31",
+  "model_ftype": "Q8_0",
+  "n_ctx": 131072,
+  "total_slots": 4,
+  "sampler_defaults": {
+    "temperature": 1.0,
+    "top_k": 20,
+    "top_p": 0.949999988079071,
+    "min_p": 0.05000000074505806,
+    "repeat_penalty": 1.0,
+    "samplers": [
+      "penalties",
+      "dry",
+      "top_n_sigma",
+      "top_k",
+      "typ_p",
+      "top_p",
+      "min_p",
+      "xtc",
+      "temperature"
+    ]
+  },
+  "chat_template": "<omitted: ~30KB jinja, no reproduction value>"
+}
+
+--- publication note ---
+Scrubbed for publication by sanitize-config.py: hostnames, internal
+repo paths and home paths removed, GPU index omitted (immaterial —
+one card, one power cap), chat template omitted for size. Everything
+that affects the measurement is kept verbatim.

bench/q8/qa-eval-1786853047.csv

@@ -0,0 +1,93 @@
+id,source,domain,result,reasoning_chars,answer_chars,completion_tokens,finish_reason,ttft_ms,total_ms
+recNu3MXkvWUzHZr9,GPQA Diamond,Physics,pass,4701,575,2116,stop,359.1,27456.6
+001b51d76b4d422988f2c11f104a2c6c,SuperGPQA,Agronomy,pass,1958,9,473,stop,441.6,6207.8
+aime2025-01,AIME2025,Number Theory,pass,1234,414,731,stop,451.2,9283.4
+recoiTJPGUmzAkief,GPQA Diamond,Physics,pass,13843,363,3178,stop,554.4,51646.3
+b7e20eac98764fb0bf30e8366d951daa,SuperGPQA,Engineering,fail,51804,422,14665,stop,578.1,198819.5
+aime2025-16,AIME2025,Geometry,pass,3571,731,1979,stop,582.0,26516.6
+rec4UqStf9WUVif1f,GPQA Diamond,Physics,pass,1470,418,972,stop,497.7,12552.1
+4a1d1780a93f4093b6fb7d3c314cbea8,SuperGPQA,Engineering,fail,16691,159,3560,stop,458.6,56200.7
+aime2025-02,AIME2025,Geometry,pass,40824,1421,16615,stop,801.5,238811.1
+recgI6tUQ7RLJRWGx,GPQA Diamond,Physics,pass,4160,618,1764,stop,666.9,25740.6
+6082513c8dba4ec68aa68f1bf5854d09,SuperGPQA,Law,pass,4232,9,857,stop,473.7,12780.1
+aime2025-03,AIME2025,Combinatorics,pass,1870,612,1006,stop,470.3,13878.7
+recDytVnNYZe2HuUU,GPQA Diamond (modified),Physics,pass,5351,517,2426,stop,470.2,34159.0
+bebf1ed45ae14ad7b4f205f3909cb58a,SuperGPQA,Literature and Arts,pass,1604,9,435,stop,511.8,6477.3
+aime2025-18,AIME2025,Combinatorics,pass,16032,1430,5814,stop,552.6,84963.1
+recNFJjE5PPTqVJGv,GPQA Diamond,Chemistry,pass,2829,772,1507,stop,539.0,20557.4
+7ca71b86327744b78e93185a45bc5cef,SuperGPQA,Medicine,pass,12862,9,3097,stop,494.5,45193.0
+aime2025-04,AIME2025,Algebra,pass,1826,657,1117,stop,463.3,14533.8
+rec2UlKqC6RFHdcro,GPQA Diamond,Physics,pass,4151,606,1818,stop,482.6,26101.1
+d44b94f7749345a39a65f6312bda8764,SuperGPQA,Engineering,pass,4443,325,1401,stop,488.7,22851.7
+aime2025-19,AIME2025,Algebra,pass,3650,1060,2353,stop,506.2,30637.1
+recv7GsQg3f0fvB1f,GPQA Diamond,Chemistry,pass,56548,672,13521,stop,622.7,207777.7
+febe406f44d74a40b50bb5b7c69d5dc1,SuperGPQA,Sociology,pass,15522,127,3565,stop,634.0,49053.7
+aime2025-05,AIME2025,"Combinatorics, Number Theory",pass,9752,1313,4515,stop,477.3,63086.6
+recrHBEJJoDTV05JR,GPQA Diamond,Chemistry,pass,47078,962,14733,stop,531.7,218094.1
+31950dc80ded400a9181f50626d1f75c,SuperGPQA,Science,pass,909,325,540,stop,611.9,7376.5
+aime2025-06,AIME2025,Geometry,pass,4389,873,1956,stop,452.1,27535.9
+recb80OwMgNnceA9t,GPQA Diamond,Chemistry,exhausted,91845,0,32768,length,510.6,502161.4
+0f14cd17be174618af6d60227e7dca9f,SuperGPQA,Engineering,pass,6986,915,2978,stop,1168.0,43433.6
+aime2025-21,AIME2025,"Algebra, Geometry",pass,11657,1321,4517,stop,870.1,62776.6
+recA1i5ZAh0Uzclxp,GPQA Diamond,Chemistry,pass,45301,989,14450,stop,575.5,214332.1
+cef9bcc087434cc2b4e354a9baef55eb,SuperGPQA,Engineering,pass,1056,485,857,stop,795.8,10955.6
+aime2025-07,AIME2025,Combinatorics,pass,19495,1795,7143,stop,510.8,107042.8
+recqGD3fxPCI59vPQ,GPQA Diamond,Physics,pass,24812,727,7297,stop,597.3,109647.1
+9f93aa2cfdb547b5b3a4623f80f7fff6,SuperGPQA,Engineering,pass,2195,191,690,stop,552.2,10307.9
+aime2025-22,AIME2025,Number Theory,fail,4202,1060,2030,stop,476.5,26262.3
+rechKl68Uc6H7vU0N,GPQA Diamond,Chemistry,exhausted,101889,0,32768,length,589.7,486320.0
+97ad69dda7b2462c98638b79e78aea0b,SuperGPQA,Medicine,pass,5808,122,1448,stop,817.8,20874.6
+aime2025-08,AIME2025,Algebra,pass,5323,1096,2767,stop,471.5,35525.3
+rec1zl5LvaatzGhFt,GPQA Diamond,Physics,pass,2285,401,835,stop,709.0,11320.5
+e78e4e539d6f4e379ac140d923d7b1be,SuperGPQA,Military Science,fail,16665,9,3390,stop,473.8,50566.4
+aime2025-09,AIME2025,Algebra,pass,46935,1669,20456,stop,511.9,280544.5
+recTs7qzfJs6kfLUK,GPQA Diamond,Biology,pass,22405,339,7863,stop,946.2,114526.4
+8483667a25e74fdfa3188de4ea734f03,SuperGPQA,Science,fail,102234,953,29267,stop,602.8,449802.7
+aime2025-24,AIME2025,Algebra,pass,15429,1226,6515,stop,745.3,90301.1
+rec32C1ZEapBnCC0E,GPQA Diamond,Physics,pass,5631,535,2513,stop,543.3,35348.5
+e5ed76ef98144f06843125daf1bccd35,SuperGPQA,Science,pass,1871,335,1130,stop,501.8,14369.6
+aime2025-10,AIME2025,Combinatorics,pass,41121,2097,14430,stop,662.9,207504.8
+recZWeueB7lSPR6wN,GPQA Diamond,Chemistry,exhausted,111809,0,32768,length,622.6,506089.8
+fd7924876c4845cd83d95d61dfa0b236,SuperGPQA,Sociology,pass,9416,9,1889,stop,788.7,27349.8
+aime2025-25,AIME2025,Combinatorics,pass,33469,1381,11406,stop,467.4,163280.6
+recVvpD8miVjmmyfe,GPQA Diamond,Biology,exhausted,71858,0,32768,length,563.9,480198.1
+6bfe7d19299d4b3184636e1f51694306,SuperGPQA,Agronomy,pass,31232,597,9044,stop,917.5,123836.9
+aime2025-26,AIME2025,Combinatorics,pass,20058,1038,6974,stop,564.9,101663.8
+recAAJoHMW45Lv5je,GPQA Diamond,Chemistry,pass,56011,663,16947,stop,534.8,236405.8
+e1825d70c5844c22933087eafa89e39c,SuperGPQA,Engineering,pass,1498,414,1013,stop,808.0,12681.6
+aime2025-12,AIME2025,Algebra,pass,24641,1690,13523,stop,484.1,172934.2
+reckEnrOPFT9Ru7tW,GPQA Diamond,Chemistry,pass,22555,770,7029,stop,690.7,102551.2
+ab430ac3f18e4e02a2cb3f35498c6b30,SuperGPQA,Engineering,pass,8942,840,3568,stop,574.6,51126.2
+aime2025-27,AIME2025,Geometry,pass,36498,1239,14283,stop,571.2,200309.2
+rec8nshandHARTkrg,GPQA Diamond,Chemistry,pass,73328,498,22955,stop,878.6,326324.8
+e8c5da5ca40647158b0c91dd695c6243,SuperGPQA,Science,pass,6470,705,2750,stop,901.4,37356.8
+aime2025-13,AIME2025,Combinatorics,exhausted,113865,0,32768,length,526.0,469657.6
+recFaL6j8UMhutXrc,GPQA Diamond,Chemistry,pass,86504,1575,28289,stop,944.9,400071.8
+05efdc6fb2404ddc8dd5729bb68c74e5,SuperGPQA,Science,pass,3248,655,2132,stop,791.7,26802.9
+aime2025-28,AIME2025,Number Theory,exhausted,64329,0,32768,length,493.0,455399.0
+reczQ4I0VpENdMtIj,GPQA Diamond,Biology,pass,29362,335,6207,stop,786.8,91141.2
+ba52e06cbe1a4310a77a7d12cd1db943,SuperGPQA,Science,pass,27339,292,9504,stop,771.0,125482.9
+aime2025-29,AIME2025,Geometry,pass,63742,1669,27648,stop,594.1,381561.2
+recWxGU8Q4YReJ1tb,GPQA Diamond,Chemistry,exhausted,98325,0,32768,length,1088.5,477832.9
+591a77df21324272914be82ac6583399,SuperGPQA,Science,fail,2190,330,1124,stop,872.8,15071.5
+aime2025-15,AIME2025,Number Theory,exhausted,77388,0,32768,length,438.5,488279.0
+recMicVBcqy1xM1jq,GPQA Diamond,Physics,pass,38300,610,14108,stop,823.6,201131.7
+e780f37a5baa4fe094cd9c157486664d,SuperGPQA,Sociology,fail,30028,345,6801,stop,635.0,100085.7
+aime2025-30,AIME2025,Algebra,pass,54238,1827,23088,stop,500.9,328146.3
+compsec-076,COMPSEC,FreeBSD / librpcsec_gss,pass,6439,10,1671,stop,896.4,24692.0
+compsec-077,COMPSEC,GNU Inetutils telnetd,pass,24880,10,6698,stop,601.0,100004.4
+compsec-078,COMPSEC,Botan,pass,2685,10,749,stop,684.5,11191.5
+compsec-079,COMPSEC,Botan,pass,32135,10,7426,stop,600.3,118136.7
+compsec-080,COMPSEC,Botan,pass,12486,9,2731,stop,632.1,43336.2
+compsec-081,COMPSEC,uds-c,pass,27125,10,6657,stop,608.0,99869.3
+compsec-082,COMPSEC,Firebird,pass,31725,10,7949,stop,685.8,122373.6
+compsec-083,COMPSEC,Firebird,pass,43646,10,11357,stop,645.8,176298.5
+compsec-084,COMPSEC,PHP,pass,3979,9,1046,stop,640.8,15888.1
+compsec-085,COMPSEC,PHP PDO Firebird,pass,7012,9,1764,stop,483.2,27259.2
+compsec-086,COMPSEC,libexpat,pass,5129,9,1352,stop,699.8,20433.3
+compsec-087,COMPSEC,Mbed TLS,pass,10591,9,2933,stop,602.9,43862.8
+compsec-088,COMPSEC,Mbed TLS,pass,9526,10,2381,stop,595.7,36932.5
+compsec-089,COMPSEC,Mbed TLS,pass,2845,10,772,stop,522.1,11385.1
+compsec-090,COMPSEC,Linux kernel,pass,4157,10,1157,stop,562.5,16899.3
+compsec-091,COMPSEC,Linux kernel USB gadget storage,pass,2443,9,643,stop,596.1,9724.6
+compsec-092,COMPSEC,Linux kernel AppArmor,pass,9002,10,2142,stop,538.2,32139.2

bench/q8/qa-eval-1786853047.json

@@ -0,0 +1,1028 @@
+{
+  "accuracy": 0.8369565217391305,
+  "bench_version": "0.1.0",
+  "cases": 92,
+  "cases_detail": [
+    {
+      "answer_chars": 575,
+      "completion_tokens": 2116,
+      "finish_reason": "stop",
+      "id": "recNu3MXkvWUzHZr9",
+      "reasoning_chars": 4701,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 27456.625834000002,
+      "ttft_ms": 359.130959
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 473,
+      "finish_reason": "stop",
+      "id": "001b51d76b4d422988f2c11f104a2c6c",
+      "reasoning_chars": 1958,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 6207.820583000001,
+      "ttft_ms": 441.594125
+    },
+    {
+      "answer_chars": 414,
+      "completion_tokens": 731,
+      "finish_reason": "stop",
+      "id": "aime2025-01",
+      "reasoning_chars": 1234,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 9283.4445,
+      "ttft_ms": 451.192625
+    },
+    {
+      "answer_chars": 363,
+      "completion_tokens": 3178,
+      "finish_reason": "stop",
+      "id": "recoiTJPGUmzAkief",
+      "reasoning_chars": 13843,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 51646.299917000004,
+      "ttft_ms": 554.41025
+    },
+    {
+      "answer_chars": 422,
+      "completion_tokens": 14665,
+      "finish_reason": "stop",
+      "id": "b7e20eac98764fb0bf30e8366d951daa",
+      "reasoning_chars": 51804,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 198819.493416,
+      "ttft_ms": 578.119458
+    },
+    {
+      "answer_chars": 731,
+      "completion_tokens": 1979,
+      "finish_reason": "stop",
+      "id": "aime2025-16",
+      "reasoning_chars": 3571,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 26516.596374999997,
+      "ttft_ms": 581.993291
+    },
+    {
+      "answer_chars": 418,
+      "completion_tokens": 972,
+      "finish_reason": "stop",
+      "id": "rec4UqStf9WUVif1f",
+      "reasoning_chars": 1470,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 12552.091125,
+      "ttft_ms": 497.682875
+    },
+    {
+      "answer_chars": 159,
+      "completion_tokens": 3560,
+      "finish_reason": "stop",
+      "id": "4a1d1780a93f4093b6fb7d3c314cbea8",
+      "reasoning_chars": 16691,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 56200.715375,
+      "ttft_ms": 458.59191599999997
+    },
+    {
+      "answer_chars": 1421,
+      "completion_tokens": 16615,
+      "finish_reason": "stop",
+      "id": "aime2025-02",
+      "reasoning_chars": 40824,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 238811.061625,
+      "ttft_ms": 801.4995
+    },
+    {
+      "answer_chars": 618,
+      "completion_tokens": 1764,
+      "finish_reason": "stop",
+      "id": "recgI6tUQ7RLJRWGx",
+      "reasoning_chars": 4160,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 25740.642417,
+      "ttft_ms": 666.9044170000001
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 857,
+      "finish_reason": "stop",
+      "id": "6082513c8dba4ec68aa68f1bf5854d09",
+      "reasoning_chars": 4232,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 12780.08675,
+      "ttft_ms": 473.662417
+    },
+    {
+      "answer_chars": 612,
+      "completion_tokens": 1006,
+      "finish_reason": "stop",
+      "id": "aime2025-03",
+      "reasoning_chars": 1870,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 13878.669249999999,
+      "ttft_ms": 470.332417
+    },
+    {
+      "answer_chars": 517,
+      "completion_tokens": 2426,
+      "finish_reason": "stop",
+      "id": "recDytVnNYZe2HuUU",
+      "reasoning_chars": 5351,
+      "result": "pass",
+      "source": "GPQA Diamond (modified)",
+      "total_ms": 34158.975875000004,
+      "ttft_ms": 470.16366700000003
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 435,
+      "finish_reason": "stop",
+      "id": "bebf1ed45ae14ad7b4f205f3909cb58a",
+      "reasoning_chars": 1604,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 6477.3483750000005,
+      "ttft_ms": 511.84637499999997
+    },
+    {
+      "answer_chars": 1430,
+      "completion_tokens": 5814,
+      "finish_reason": "stop",
+      "id": "aime2025-18",
+      "reasoning_chars": 16032,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 84963.127792,
+      "ttft_ms": 552.57425
+    },
+    {
+      "answer_chars": 772,
+      "completion_tokens": 1507,
+      "finish_reason": "stop",
+      "id": "recNFJjE5PPTqVJGv",
+      "reasoning_chars": 2829,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 20557.352708,
+      "ttft_ms": 539.001583
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 3097,
+      "finish_reason": "stop",
+      "id": "7ca71b86327744b78e93185a45bc5cef",
+      "reasoning_chars": 12862,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 45193.0045,
+      "ttft_ms": 494.537625
+    },
+    {
+      "answer_chars": 657,
+      "completion_tokens": 1117,
+      "finish_reason": "stop",
+      "id": "aime2025-04",
+      "reasoning_chars": 1826,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 14533.831750000001,
+      "ttft_ms": 463.292792
+    },
+    {
+      "answer_chars": 606,
+      "completion_tokens": 1818,
+      "finish_reason": "stop",
+      "id": "rec2UlKqC6RFHdcro",
+      "reasoning_chars": 4151,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 26101.106375,
+      "ttft_ms": 482.605875
+    },
+    {
+      "answer_chars": 325,
+      "completion_tokens": 1401,
+      "finish_reason": "stop",
+      "id": "d44b94f7749345a39a65f6312bda8764",
+      "reasoning_chars": 4443,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 22851.748375,
+      "ttft_ms": 488.66375
+    },
+    {
+      "answer_chars": 1060,
+      "completion_tokens": 2353,
+      "finish_reason": "stop",
+      "id": "aime2025-19",
+      "reasoning_chars": 3650,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 30637.0505,
+      "ttft_ms": 506.202625
+    },
+    {
+      "answer_chars": 672,
+      "completion_tokens": 13521,
+      "finish_reason": "stop",
+      "id": "recv7GsQg3f0fvB1f",
+      "reasoning_chars": 56548,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 207777.664042,
+      "ttft_ms": 622.6594170000001
+    },
+    {
+      "answer_chars": 127,
+      "completion_tokens": 3565,
+      "finish_reason": "stop",
+      "id": "febe406f44d74a40b50bb5b7c69d5dc1",
+      "reasoning_chars": 15522,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 49053.660875,
+      "ttft_ms": 633.976583
+    },
+    {
+      "answer_chars": 1313,
+      "completion_tokens": 4515,
+      "finish_reason": "stop",
+      "id": "aime2025-05",
+      "reasoning_chars": 9752,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 63086.649083,
+      "ttft_ms": 477.33354099999997
+    },
+    {
+      "answer_chars": 962,
+      "completion_tokens": 14733,
+      "finish_reason": "stop",
+      "id": "recrHBEJJoDTV05JR",
+      "reasoning_chars": 47078,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 218094.104166,
+      "ttft_ms": 531.713791
+    },
+    {
+      "answer_chars": 325,
+      "completion_tokens": 540,
+      "finish_reason": "stop",
+      "id": "31950dc80ded400a9181f50626d1f75c",
+      "reasoning_chars": 909,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 7376.520375,
+      "ttft_ms": 611.90075
+    },
+    {
+      "answer_chars": 873,
+      "completion_tokens": 1956,
+      "finish_reason": "stop",
+      "id": "aime2025-06",
+      "reasoning_chars": 4389,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 27535.874750000003,
+      "ttft_ms": 452.08575
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recb80OwMgNnceA9t",
+      "reasoning_chars": 91845,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 502161.387417,
+      "ttft_ms": 510.573417
+    },
+    {
+      "answer_chars": 915,
+      "completion_tokens": 2978,
+      "finish_reason": "stop",
+      "id": "0f14cd17be174618af6d60227e7dca9f",
+      "reasoning_chars": 6986,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 43433.608958,
+      "ttft_ms": 1168.0147080000002
+    },
+    {
+      "answer_chars": 1321,
+      "completion_tokens": 4517,
+      "finish_reason": "stop",
+      "id": "aime2025-21",
+      "reasoning_chars": 11657,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 62776.589708,
+      "ttft_ms": 870.065083
+    },
+    {
+      "answer_chars": 989,
+      "completion_tokens": 14450,
+      "finish_reason": "stop",
+      "id": "recA1i5ZAh0Uzclxp",
+      "reasoning_chars": 45301,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 214332.063125,
+      "ttft_ms": 575.4913750000001
+    },
+    {
+      "answer_chars": 485,
+      "completion_tokens": 857,
+      "finish_reason": "stop",
+      "id": "cef9bcc087434cc2b4e354a9baef55eb",
+      "reasoning_chars": 1056,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 10955.635209,
+      "ttft_ms": 795.782584
+    },
+    {
+      "answer_chars": 1795,
+      "completion_tokens": 7143,
+      "finish_reason": "stop",
+      "id": "aime2025-07",
+      "reasoning_chars": 19495,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 107042.78525,
+      "ttft_ms": 510.82020900000003
+    },
+    {
+      "answer_chars": 727,
+      "completion_tokens": 7297,
+      "finish_reason": "stop",
+      "id": "recqGD3fxPCI59vPQ",
+      "reasoning_chars": 24812,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 109647.099334,
+      "ttft_ms": 597.345667
+    },
+    {
+      "answer_chars": 191,
+      "completion_tokens": 690,
+      "finish_reason": "stop",
+      "id": "9f93aa2cfdb547b5b3a4623f80f7fff6",
+      "reasoning_chars": 2195,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 10307.940125,
+      "ttft_ms": 552.197167
+    },
+    {
+      "answer_chars": 1060,
+      "completion_tokens": 2030,
+      "finish_reason": "stop",
+      "id": "aime2025-22",
+      "reasoning_chars": 4202,
+      "result": "fail",
+      "source": "AIME2025",
+      "total_ms": 26262.289125,
+      "ttft_ms": 476.522167
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "rechKl68Uc6H7vU0N",
+      "reasoning_chars": 101889,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 486320.004375,
+      "ttft_ms": 589.710666
+    },
+    {
+      "answer_chars": 122,
+      "completion_tokens": 1448,
+      "finish_reason": "stop",
+      "id": "97ad69dda7b2462c98638b79e78aea0b",
+      "reasoning_chars": 5808,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 20874.60025,
+      "ttft_ms": 817.808208
+    },
+    {
+      "answer_chars": 1096,
+      "completion_tokens": 2767,
+      "finish_reason": "stop",
+      "id": "aime2025-08",
+      "reasoning_chars": 5323,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 35525.251958,
+      "ttft_ms": 471.541625
+    },
+    {
+      "answer_chars": 401,
+      "completion_tokens": 835,
+      "finish_reason": "stop",
+      "id": "rec1zl5LvaatzGhFt",
+      "reasoning_chars": 2285,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 11320.547542,
+      "ttft_ms": 708.979375
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 3390,
+      "finish_reason": "stop",
+      "id": "e78e4e539d6f4e379ac140d923d7b1be",
+      "reasoning_chars": 16665,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 50566.380125,
+      "ttft_ms": 473.783833
+    },
+    {
+      "answer_chars": 1669,
+      "completion_tokens": 20456,
+      "finish_reason": "stop",
+      "id": "aime2025-09",
+      "reasoning_chars": 46935,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 280544.515,
+      "ttft_ms": 511.88204199999996
+    },
+    {
+      "answer_chars": 339,
+      "completion_tokens": 7863,
+      "finish_reason": "stop",
+      "id": "recTs7qzfJs6kfLUK",
+      "reasoning_chars": 22405,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 114526.443333,
+      "ttft_ms": 946.1744170000001
+    },
+    {
+      "answer_chars": 953,
+      "completion_tokens": 29267,
+      "finish_reason": "stop",
+      "id": "8483667a25e74fdfa3188de4ea734f03",
+      "reasoning_chars": 102234,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 449802.710541,
+      "ttft_ms": 602.766875
+    },
+    {
+      "answer_chars": 1226,
+      "completion_tokens": 6515,
+      "finish_reason": "stop",
+      "id": "aime2025-24",
+      "reasoning_chars": 15429,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 90301.084666,
+      "ttft_ms": 745.3406249999999
+    },
+    {
+      "answer_chars": 535,
+      "completion_tokens": 2513,
+      "finish_reason": "stop",
+      "id": "rec32C1ZEapBnCC0E",
+      "reasoning_chars": 5631,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 35348.47425,
+      "ttft_ms": 543.251667
+    },
+    {
+      "answer_chars": 335,
+      "completion_tokens": 1130,
+      "finish_reason": "stop",
+      "id": "e5ed76ef98144f06843125daf1bccd35",
+      "reasoning_chars": 1871,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 14369.618208,
+      "ttft_ms": 501.82775000000004
+    },
+    {
+      "answer_chars": 2097,
+      "completion_tokens": 14430,
+      "finish_reason": "stop",
+      "id": "aime2025-10",
+      "reasoning_chars": 41121,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 207504.819334,
+      "ttft_ms": 662.8592920000001
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recZWeueB7lSPR6wN",
+      "reasoning_chars": 111809,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 506089.753583,
+      "ttft_ms": 622.592708
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1889,
+      "finish_reason": "stop",
+      "id": "fd7924876c4845cd83d95d61dfa0b236",
+      "reasoning_chars": 9416,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 27349.834583,
+      "ttft_ms": 788.725
+    },
+    {
+      "answer_chars": 1381,
+      "completion_tokens": 11406,
+      "finish_reason": "stop",
+      "id": "aime2025-25",
+      "reasoning_chars": 33469,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 163280.632333,
+      "ttft_ms": 467.42170799999997
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recVvpD8miVjmmyfe",
+      "reasoning_chars": 71858,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 480198.054417,
+      "ttft_ms": 563.9400420000001
+    },
+    {
+      "answer_chars": 597,
+      "completion_tokens": 9044,
+      "finish_reason": "stop",
+      "id": "6bfe7d19299d4b3184636e1f51694306",
+      "reasoning_chars": 31232,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 123836.93679200001,
+      "ttft_ms": 917.491833
+    },
+    {
+      "answer_chars": 1038,
+      "completion_tokens": 6974,
+      "finish_reason": "stop",
+      "id": "aime2025-26",
+      "reasoning_chars": 20058,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 101663.767416,
+      "ttft_ms": 564.897875
+    },
+    {
+      "answer_chars": 663,
+      "completion_tokens": 16947,
+      "finish_reason": "stop",
+      "id": "recAAJoHMW45Lv5je",
+      "reasoning_chars": 56011,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 236405.806666,
+      "ttft_ms": 534.812458
+    },
+    {
+      "answer_chars": 414,
+      "completion_tokens": 1013,
+      "finish_reason": "stop",
+      "id": "e1825d70c5844c22933087eafa89e39c",
+      "reasoning_chars": 1498,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 12681.578958999999,
+      "ttft_ms": 807.969375
+    },
+    {
+      "answer_chars": 1690,
+      "completion_tokens": 13523,
+      "finish_reason": "stop",
+      "id": "aime2025-12",
+      "reasoning_chars": 24641,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 172934.19700000001,
+      "ttft_ms": 484.131542
+    },
+    {
+      "answer_chars": 770,
+      "completion_tokens": 7029,
+      "finish_reason": "stop",
+      "id": "reckEnrOPFT9Ru7tW",
+      "reasoning_chars": 22555,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 102551.2195,
+      "ttft_ms": 690.736
+    },
+    {
+      "answer_chars": 840,
+      "completion_tokens": 3568,
+      "finish_reason": "stop",
+      "id": "ab430ac3f18e4e02a2cb3f35498c6b30",
+      "reasoning_chars": 8942,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 51126.206375,
+      "ttft_ms": 574.6495
+    },
+    {
+      "answer_chars": 1239,
+      "completion_tokens": 14283,
+      "finish_reason": "stop",
+      "id": "aime2025-27",
+      "reasoning_chars": 36498,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 200309.19775000002,
+      "ttft_ms": 571.1750840000001
+    },
+    {
+      "answer_chars": 498,
+      "completion_tokens": 22955,
+      "finish_reason": "stop",
+      "id": "rec8nshandHARTkrg",
+      "reasoning_chars": 73328,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 326324.774541,
+      "ttft_ms": 878.552416
+    },
+    {
+      "answer_chars": 705,
+      "completion_tokens": 2750,
+      "finish_reason": "stop",
+      "id": "e8c5da5ca40647158b0c91dd695c6243",
+      "reasoning_chars": 6470,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 37356.805541999995,
+      "ttft_ms": 901.4087499999999
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-13",
+      "reasoning_chars": 113865,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 469657.62200000003,
+      "ttft_ms": 525.9973329999999
+    },
+    {
+      "answer_chars": 1575,
+      "completion_tokens": 28289,
+      "finish_reason": "stop",
+      "id": "recFaL6j8UMhutXrc",
+      "reasoning_chars": 86504,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 400071.826917,
+      "ttft_ms": 944.872958
+    },
+    {
+      "answer_chars": 655,
+      "completion_tokens": 2132,
+      "finish_reason": "stop",
+      "id": "05efdc6fb2404ddc8dd5729bb68c74e5",
+      "reasoning_chars": 3248,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 26802.890292,
+      "ttft_ms": 791.721584
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-28",
+      "reasoning_chars": 64329,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 455399.040875,
+      "ttft_ms": 492.990125
+    },
+    {
+      "answer_chars": 335,
+      "completion_tokens": 6207,
+      "finish_reason": "stop",
+      "id": "reczQ4I0VpENdMtIj",
+      "reasoning_chars": 29362,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 91141.16124999999,
+      "ttft_ms": 786.774583
+    },
+    {
+      "answer_chars": 292,
+      "completion_tokens": 9504,
+      "finish_reason": "stop",
+      "id": "ba52e06cbe1a4310a77a7d12cd1db943",
+      "reasoning_chars": 27339,
+      "result": "pass",
+      "source": "SuperGPQA",
+      "total_ms": 125482.8795,
+      "ttft_ms": 770.972209
+    },
+    {
+      "answer_chars": 1669,
+      "completion_tokens": 27648,
+      "finish_reason": "stop",
+      "id": "aime2025-29",
+      "reasoning_chars": 63742,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 381561.177042,
+      "ttft_ms": 594.065709
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "recWxGU8Q4YReJ1tb",
+      "reasoning_chars": 98325,
+      "result": "exhausted",
+      "source": "GPQA Diamond",
+      "total_ms": 477832.87579099997,
+      "ttft_ms": 1088.534791
+    },
+    {
+      "answer_chars": 330,
+      "completion_tokens": 1124,
+      "finish_reason": "stop",
+      "id": "591a77df21324272914be82ac6583399",
+      "reasoning_chars": 2190,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 15071.496667,
+      "ttft_ms": 872.8029590000001
+    },
+    {
+      "answer_chars": 0,
+      "completion_tokens": 32768,
+      "finish_reason": "length",
+      "id": "aime2025-15",
+      "reasoning_chars": 77388,
+      "result": "exhausted",
+      "source": "AIME2025",
+      "total_ms": 488278.975292,
+      "ttft_ms": 438.50212500000004
+    },
+    {
+      "answer_chars": 610,
+      "completion_tokens": 14108,
+      "finish_reason": "stop",
+      "id": "recMicVBcqy1xM1jq",
+      "reasoning_chars": 38300,
+      "result": "pass",
+      "source": "GPQA Diamond",
+      "total_ms": 201131.696708,
+      "ttft_ms": 823.588542
+    },
+    {
+      "answer_chars": 345,
+      "completion_tokens": 6801,
+      "finish_reason": "stop",
+      "id": "e780f37a5baa4fe094cd9c157486664d",
+      "reasoning_chars": 30028,
+      "result": "fail",
+      "source": "SuperGPQA",
+      "total_ms": 100085.698041,
+      "ttft_ms": 635.0276660000001
+    },
+    {
+      "answer_chars": 1827,
+      "completion_tokens": 23088,
+      "finish_reason": "stop",
+      "id": "aime2025-30",
+      "reasoning_chars": 54238,
+      "result": "pass",
+      "source": "AIME2025",
+      "total_ms": 328146.294042,
+      "ttft_ms": 500.85208299999994
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1671,
+      "finish_reason": "stop",
+      "id": "compsec-076",
+      "reasoning_chars": 6439,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 24691.996709000003,
+      "ttft_ms": 896.443125
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 6698,
+      "finish_reason": "stop",
+      "id": "compsec-077",
+      "reasoning_chars": 24880,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 100004.369167,
+      "ttft_ms": 600.965792
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 749,
+      "finish_reason": "stop",
+      "id": "compsec-078",
+      "reasoning_chars": 2685,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 11191.506334,
+      "ttft_ms": 684.492709
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 7426,
+      "finish_reason": "stop",
+      "id": "compsec-079",
+      "reasoning_chars": 32135,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 118136.657833,
+      "ttft_ms": 600.282291
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 2731,
+      "finish_reason": "stop",
+      "id": "compsec-080",
+      "reasoning_chars": 12486,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 43336.239665999994,
+      "ttft_ms": 632.071375
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 6657,
+      "finish_reason": "stop",
+      "id": "compsec-081",
+      "reasoning_chars": 27125,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 99869.314375,
+      "ttft_ms": 608.006042
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 7949,
+      "finish_reason": "stop",
+      "id": "compsec-082",
+      "reasoning_chars": 31725,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 122373.649583,
+      "ttft_ms": 685.7641669999999
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 11357,
+      "finish_reason": "stop",
+      "id": "compsec-083",
+      "reasoning_chars": 43646,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 176298.53866700002,
+      "ttft_ms": 645.836958
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1046,
+      "finish_reason": "stop",
+      "id": "compsec-084",
+      "reasoning_chars": 3979,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 15888.129125000001,
+      "ttft_ms": 640.836167
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1764,
+      "finish_reason": "stop",
+      "id": "compsec-085",
+      "reasoning_chars": 7012,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 27259.208708,
+      "ttft_ms": 483.22645800000004
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 1352,
+      "finish_reason": "stop",
+      "id": "compsec-086",
+      "reasoning_chars": 5129,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 20433.278416,
+      "ttft_ms": 699.7719999999999
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 2933,
+      "finish_reason": "stop",
+      "id": "compsec-087",
+      "reasoning_chars": 10591,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 43862.756541,
+      "ttft_ms": 602.864083
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 2381,
+      "finish_reason": "stop",
+      "id": "compsec-088",
+      "reasoning_chars": 9526,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 36932.5335,
+      "ttft_ms": 595.733166
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 772,
+      "finish_reason": "stop",
+      "id": "compsec-089",
+      "reasoning_chars": 2845,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 11385.085542,
+      "ttft_ms": 522.12125
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 1157,
+      "finish_reason": "stop",
+      "id": "compsec-090",
+      "reasoning_chars": 4157,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 16899.257292000002,
+      "ttft_ms": 562.517583
+    },
+    {
+      "answer_chars": 9,
+      "completion_tokens": 643,
+      "finish_reason": "stop",
+      "id": "compsec-091",
+      "reasoning_chars": 2443,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 9724.619584,
+      "ttft_ms": 596.148292
+    },
+    {
+      "answer_chars": 10,
+      "completion_tokens": 2142,
+      "finish_reason": "stop",
+      "id": "compsec-092",
+      "reasoning_chars": 9002,
+      "result": "pass",
+      "source": "COMPSEC",
+      "total_ms": 32139.205208,
+      "ttft_ms": 538.2330410000001
+    }
+  ],
+  "config": {
+    "max_tokens": 32768,
+    "seed": 42,
+    "temperature": 0.0
+  },
+  "exhausted": 8,
+  "failed_answered": 7,
+  "model": "q8-mtp",
+  "passed": 77
+}