# Evaluation protocol

The release compares Grug v2 with `ProCreations/grug-v1.1-qwen-3.8-27b` and its foundation, `Qwen/Qwen3.8-27B`. Each result identifies the exact checkpoint revision, runtime, reasoning effort, and sampling settings. These are independent measurements from this build, not the upstream model cards' scores.

Model execution, training, merging, quantization, and generation checks run on Hugging Face Jobs. GPU jobs use an NVIDIA H200. Local work consists of orchestration, source editing, metadata checks, and analysis of saved outputs.

The primary inference runtime is vLLM 0.29.0 with PyTorch 2.13.0+cu130 and Transformers 5.16.1. The core evaluation uses a 24,576-token context, temperature 0.6, top-p 0.95, top-k 20, and seed 42. Results are one sample per problem. The release repetition penalty is 1.05, with matching reruns of v1.1 and Qwen. Earlier repetition-penalty-1.0 runs remain separate development comparisons. Repetition stress tests and serving smoke tests also include greedy decoding.

| Evaluation | Cases | Scoring |
|---|---:|---|
| HumanEval | 132 | Official executable tests; dataset indices 32–163 |
| MBPP sanitized test | 225 | All supplied executable tests; indices 32–256 |
| GSM8K test | 200 | Final numeric answer |
| MATH-500 test | 150 | Symbolic final-answer verification with math-verify 0.9.0 |
| BFCL v4 subset | 200 | Official AST checker; 50 cases each from simple Python, multiple, irrelevance, and live multiple |
| Authored repository repairs | 12 | Actual edits, test execution, and additional held-out assertions |
| SWE-bench Verified repository subset | 12 | Original issues and repositories, official test patches, and fail-to-pass/pass-to-pass grading |

HumanEval and MBPP indices 0–31 were reserved for development and are excluded from the reported release coding evaluations. Core output limits are 8,192 tokens for HumanEval, 6,144 for MBPP, 4,096 for GSM8K, and 6,144 for MATH. A separately identified xhigh evaluation doubles each limit for all compared models. Those results remain separate from the original-limit results. Truncation rates accompany accuracy because additional reasoning can exhaust a fixed output allowance.

Coding accuracy requires working code in the visible final answer. Code found only in unfinished hidden reasoning does not earn final-answer credit. A second completed-answer metric also requires an end-of-sequence stop. Historical raw harness scores are retained separately where they used a more permissive fallback. GSM8K results likewise exclude passes found only in hidden reasoning. Math accuracy is rescored from the final answer, including boxed answers; adherence to the requested `ANSWER:` prefix is a separate formatting metric.

BFCL uses upstream revision `6ea57973c7a6097fd7c5915698c54c17c5b1b6c8`. Positive and negative controls check the AST grader before evaluation. This is a fixed 200-case subset, not a full BFCL leaderboard submission.

The SWE-bench subset uses dataset revision `c104f840cc67f8b6eec6f759ebc8b2693d585d4a` and official harness revision `726c5461e2ef52d83cf1ea2107870a8bb3328d57`. Six Django and six SymPy cases were chosen deterministically within the supported version groups before evaluating the models. Every case passed an oracle check: the unfixed repository failed the specified target tests, and the gold fix passed the required fail-to-pass and pass-to-pass tests. Target test coverage was checked explicitly.

These SWE-bench runs use a shared pinned Python 3.9 environment rather than the official per-instance Docker images. They are a repository-repair subset diagnostic, not the full SWE-bench Verified benchmark or a leaderboard score. The case IDs, environment provenance, and per-case results accompany the release metrics.

The agent gets a persistent repository, a bash tool, and at most 35 tool turns. Low and medium allow 2,048 output tokens per turn; xhigh allows 4,096. Tool output is bounded, commands have a 90-second limit, and old complete assistant/tool turns are removed when needed to leave room for generation. The corrected harness counts explicit token IDs and reserves an additional 256 tokens. Earlier runs with incorrect context accounting are development records and are excluded from final repository comparisons.

Only production changes are evaluated against a fresh repository. Editing tests or runners cannot earn credit. The evaluator applies the official held-out test patch after the agent ends. Reports include both patches that resolve the tests and patches that resolve them with a completed final response, plus step-limit and context-eviction counts. Agent thought text is not retained in subsequent tool-call history, consistently across compared models. A separate matched ablation retains reasoning in `reasoning_content` for all three models using the same original budgets. A further xhigh comparison retains reasoning and raises the context to 65,536 and per-turn output allowance to 16,384 for every model. These runs remain separate tables; the larger-budget comparison was added because 8/12 of the base model’s xhigh original-budget episodes ended with unfinished reasoning. Scores under the small caps must not be interpreted as unconstrained base-model capability.

OpenCode regression checks contain 30 actual session-title prompts and 10 additional requests where tools should not be called. Completed single-line title quality uses the actual title prompts’ 50-character limit; general questions and greetings are not graded as title-writing tasks. Legacy agent replay, recovery, and repetition suites are development diagnostics. Valid tool syntax in a replay is not equivalent to solving an agentic task.

Repetition inspection considers completion, visible answers, repeated nontrivial lines, and repeated 16-word shingles. The earlier detector checked only three phrase lengths and missed some real loops. Heuristic flags are inspected before being described as pathological repetition. No finite evaluation establishes that a language model can never loop. Grug style statistics are descriptive heuristics and do not replace correctness checks.

The new training material uses training splits and separately authored title examples. Task groups are split before effort variants are expanded. Exact instance identifiers and normalized issue text from the 12 SWE-bench cases were checked against inherited replay and produced no matches. This does not rule out paraphrased overlap or exposure during the foundation model's pretraining. Raw inherited replay remains private.

MTP training freezes the entire main model and optimizes only the 15 native draft-head tensors. Held-out draft/verifier token agreement is distinct from real speculative acceptance and end-to-end speed. All original backbone shard hashes are checked after the head is integrated. Runtime results measure MTP disabled and enabled on the same model and record the actual sampling settings and speculative counters.

GGUF conversion and checks use llama.cpp revision `2a3005c23f60cb38dab70b8ea2ddbd969bcf3e87`. Each text GGUF contains its own MTP head; MTP matrix tensors retain Q8_0 precision. Quantization checks cover generation, native title and tool parsing, embedded-template compatibility, speculative decoding, and a fixed 16-case executable coding smoke test. That smoke test is not a full quantized-model benchmark. The vision encoder is preserved from the foundation checkpoint; this build does not establish vision benchmark performance.

The Job ledger includes preparation, successful runs, failed attempts, and canceled runs. Cost estimates use the published hardware rate and a conservative duration allowance. They are not invoices. The submission helper reserves each active Job's entire timeout and refuses a submission that would take the build above the authorized $170 limit.

The GGUF API capability probe records the `reasoning` alias as unsupported in the pinned llama-server input parser. Its canonical `reasoning_content` and inline-history routes are tested separately. This recorded capability failure is retained in the raw results and is not rewritten as a pass. Client-side field normalization is documented in `usage.md`.

HumanEval grading preserves imports and helper functions supplied in the original prompt before executing the generated full target function and official tests. An earlier full-function adapter omitted that prelude, causing false failures such as the missing `encode_cyclic` helper in HumanEval/38. All saved release-model, baseline, MTP, and GGUF HumanEval outputs are consistently rescored on HF CPU compute; the old raw verdicts remain available. Positive and negative controls verify the corrected grader. This follows the [official evaluator’s inclusion of the prompt](https://github.com/openai/human-eval/blob/master/human_eval/execution.py).

All HumanEval and MBPP final answers, including baselines, are replayed through the corrected coding grader on HF CPU Jobs. The extractor deterministically selects the last code block defining the requested function, so a later demonstration or REPL example cannot replace an otherwise valid implementation. It does not try alternatives until tests pass. One-code-block format compliance is recorded separately from functional accuracy. A positive control with a valid implementation followed by an invalid-to-execute REPL demo verifies this distinction. Raw generation and original verdicts are retained in the private build record.

BFCL reporting retains the official AST verdict and additionally requires a visible final decision. An unfinished reasoning-only response is not credited as a successful decision to avoid tools. The saved-text audit excludes one such Qwen xhigh pass at penalty 1.05 and three at penalty 1.0; it changes no Grug v1.1, v2, or MTP score. Completed-decision accuracy additionally requires EOS and is provided in the machine-readable records. This audit inspects already-generated text and stored AST verdicts, without running a model or generated code locally.

The final coding replay uses the same HF CPU environment for every checkpoint: Python 3.11, `datasets==5.0.1`, eight grading workers, and the published `verify_code.py` limits of four CPU seconds, six wall-clock seconds, and 768 MiB per program. External NLP models/data are not provisioned for generated NLTK or spaCy imports. Ten such failures all also failed the original grader. One previously passing visible v2 low HumanEval/75 implementation, a cubic prime search, exits with signal 9 under the CPU replay limits and is counted as a failure in the final table. Runtime limits and environment can affect executable-test scores; old and corrected verdicts are preserved consistently.

A static audit inspected 6,060 saved shell commands across all 252 matched repetition-penalty-1.05 SWE episodes. Three flags were reviewed: two cloned a local fixture checkout, and one contained an issue URL in a test comment. No explicit external solution fetch was found. This checks command text; it is not network traffic capture or proof against hidden dynamic retrieval or pretraining overlap. Command-list hashes and reviewed flags are included in `results/swe_command_audit.json`.