# Running Grug 27B v2

The checkpoint contains the main model and its Grug-tuned native MTP head. There is no separate draft model to download. A runtime still needs speculative decoding enabled to use the head.

## Safetensors with vLLM

The release was tested with vLLM 0.29.0, Transformers 5.16.1, and an H200. Use the Qwen reasoning and tool parsers:

```bash
vllm serve ProCreations/grug-27b-v2 \
  --served-model-name grug-v2 \
  --max-model-len 24576 \
  --max-num-seqs 8 --gpu-memory-utilization 0.85 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
```

The repository's generation defaults are temperature **0.6**, top-p **0.95**, top-k **20**, and repetition penalty **1.05**. Set them explicitly if a client or server overrides repository defaults. The default effort is **medium**. Choose `low`, `medium`, or `xhigh` through `chat_template_kwargs`:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "grug-v2",
    "messages": [{"role": "user", "content": "Implement a cycle detector for a directed graph. Include tests."}],
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "repetition_penalty": 1.05,
    "max_tokens": 16384,
    "chat_template_kwargs": {"reasoning_effort": "xhigh"}
  }'
```

Use the model's chat template for tool conversations. Supply tools through the API's `tools` field; do not manually insert tool-call wrappers into a user message. For session titles, make the title instruction the system message and ask for the title only. The release tests cover title requests with tools present as well as ordinary tool use.

Thinking is enabled by default. A request can disable it with `chat_template_kwargs: {"enable_thinking": false}`. Higher effort permits more deliberation, but also needs enough output tokens. Accuracy is not guaranteed to rise on every problem or fixed-budget benchmark; consult the measured effort tables.

The compatibility template accepts historical assistant reasoning in `reasoning_content`, the `reasoning` alias, or inline `<think>...</think>` content, and normalizes it once. Use the representation supported by your client. In particular, the tested llama-server HTTP parser accepts `reasoning_content` and discards the `reasoning` alias before template rendering. Rename an incoming `reasoning` string to `reasoning_content` in your client when needed (the included `normalize_messages.py` helper does this without mutating the input history); inline history and canonical `reasoning_content` both passed the GGUF checks. This is a server input limitation, so changing the model template cannot recover a discarded field. See the [pinned llama.cpp message parser](https://github.com/ggml-org/llama.cpp/blob/2a3005c23f60cb38dab70b8ea2ddbd969bcf3e87/common/chat.cpp). Avoid duplicating the same thought text into multiple fields. The vLLM 0.29.0 request schema normalizes incoming `reasoning_content` to `reasoning`, then provides both names to the template; the helper remains compatible with that route. This compatibility follows the pinned [vLLM request normalizer](https://github.com/vllm-project/vllm/blob/v0.29.0/vllm/entrypoints/openai/chat_completion/protocol.py) and [message parser](https://github.com/vllm-project/vllm/blob/v0.29.0/vllm/entrypoints/chat_utils.py).

For multi-turn tool work, retain each assistant message's reasoning alongside its text and tool calls when your client supports it. The matched repository ablation improved v2's low/medium/xhigh results to 9/12, 10/12, and 11/12 when prior reasoning was retained. This is a 12-issue diagnostic, not a guarantee for every client. Preserve the tool-call IDs when adding tool results.

The helper can normalize a saved JSON message list before forwarding it between runtimes:

```bash
python normalize_messages.py < history.json > normalized-history.json
```

To run without speculative decoding, omit `--speculative-config`; the head remains packaged in the checkpoint.

## GGUF with llama.cpp

The release GGUF files include their MTP head. Use a llama.cpp build containing commit `2a3005c23f60cb38dab70b8ea2ddbd969bcf3e87` or compatible later support for Qwen3.5-family native MTP. The release checks use that exact commit.

```bash
llama-server \
  -hf ProCreations/grug-27b-v2-gguf:Q4_K_M \
  -ngl 99 -c 12288 --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  --repeat-penalty 1.05 --repeat-last-n 12288 \
  --spec-type draft-mtp --spec-draft-n-max 2 -ngld 99
```

With a downloaded file, replace `-hf ...` with `-m /path/to/grug-27b-v2-Q4_K_M.gguf`. The server's OpenAI-compatible chat endpoint accepts `reasoning_effort` as `low`, `medium`, or `xhigh`. The embedded Jinja template is needed for consistent reasoning and tool formatting.

GGUF clients may use their own sampling defaults. Apply the settings above; the filename alone does not select a repetition penalty. The validation context is 12,288 tokens for GGUF and 24,576 for the main evaluation. The foundation's larger configured context is preserved, but this release does not establish quality throughout that range.

The optional `mmproj-grug-27b-v2-F16.gguf` is the vision projector. It is not a draft model. Vision weights are preserved from Qwen; this build's quality measurements focus on text, coding, and tools.