usage.md

Running Grug 27B v2

The checkpoint contains the main model and its Grug-tuned native MTP head. There is no separate draft model to download. A runtime still needs speculative decoding enabled to use the head.

Safetensors with vLLM

The release was tested with vLLM 0.29.0, Transformers 5.16.1, and an H200. Use the Qwen reasoning and tool parsers:

vllm serve ProCreations/grug-27b-v2 \
  --served-model-name grug-v2 \
  --max-model-len 24576 \
  --max-num-seqs 8 --gpu-memory-utilization 0.85 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

The repository's generation defaults are temperature 0.6, top-p 0.95, top-k 20, and repetition penalty 1.05. Set them explicitly if a client or server overrides repository defaults. The default effort is medium. Choose low, medium, or xhigh through chat_template_kwargs:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "grug-v2",
    "messages": [{"role": "user", "content": "Implement a cycle detector for a directed graph. Include tests."}],
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "repetition_penalty": 1.05,
    "max_tokens": 16384,
    "chat_template_kwargs": {"reasoning_effort": "xhigh"}
  }'

Use the model's chat template for tool conversations. Supply tools through the API's tools field; do not manually insert tool-call wrappers into a user message. For session titles, make the title instruction the system message and ask for the title only. The release tests cover title requests with tools present as well as ordinary tool use.

Thinking is enabled by default. A request can disable it with chat_template_kwargs: {"enable_thinking": false}. Higher effort permits more deliberation, but also needs enough output tokens. Accuracy is not guaranteed to rise on every problem or fixed-budget benchmark; consult the measured effort tables.

The compatibility template accepts historical assistant reasoning in reasoning_content, the reasoning alias, or inline <think>...</think> content, and normalizes it once. Use the representation supported by your client. In particular, the tested llama-server HTTP parser accepts reasoning_content and discards the reasoning alias before template rendering. Rename an incoming reasoning string to reasoning_content in your client when needed (the included normalize_messages.py helper does this without mutating the input history); inline history and canonical reasoning_content both passed the GGUF checks. This is a server input limitation, so changing the model template cannot recover a discarded field. See the pinned llama.cpp message parser. Avoid duplicating the same thought text into multiple fields. The vLLM 0.29.0 request schema normalizes incoming reasoning_content to reasoning, then provides both names to the template; the helper remains compatible with that route. This compatibility follows the pinned vLLM request normalizer and message parser.

For multi-turn tool work, retain each assistant message's reasoning alongside its text and tool calls when your client supports it. The matched repository ablation improved v2's low/medium/xhigh results to 9/12, 10/12, and 11/12 when prior reasoning was retained. This is a 12-issue diagnostic, not a guarantee for every client. Preserve the tool-call IDs when adding tool results.

The helper can normalize a saved JSON message list before forwarding it between runtimes:

python normalize_messages.py < history.json > normalized-history.json

To run without speculative decoding, omit --speculative-config; the head remains packaged in the checkpoint.

GGUF with llama.cpp

The release GGUF files include their MTP head. Use a llama.cpp build containing commit 2a3005c23f60cb38dab70b8ea2ddbd969bcf3e87 or compatible later support for Qwen3.5-family native MTP. The release checks use that exact commit.

llama-server \
  -hf ProCreations/grug-27b-v2-gguf:Q4_K_M \
  -ngl 99 -c 12288 --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  --repeat-penalty 1.05 --repeat-last-n 12288 \
  --spec-type draft-mtp --spec-draft-n-max 2 -ngld 99

With a downloaded file, replace -hf ... with -m /path/to/grug-27b-v2-Q4_K_M.gguf. The server's OpenAI-compatible chat endpoint accepts reasoning_effort as low, medium, or xhigh. The embedded Jinja template is needed for consistent reasoning and tool formatting.

GGUF clients may use their own sampling defaults. Apply the settings above; the filename alone does not select a repetition penalty. The validation context is 12,288 tokens for GGUF and 24,576 for the main evaluation. The foundation's larger configured context is preserved, but this release does not establish quality throughout that range.

The optional mmproj-grug-27b-v2-F16.gguf is the vision projector. It is not a draft model. Vision weights are preserved from Qwen; this build's quality measurements focus on text, coding, and tools.