# llama.cpp Q4 Reasoning-Mode Serving Fix

Date: 2026-06-26

Backend: `llama.cpp` CUDA server, `Q4_K_M`, single GPU, `tp=1`.

## Issue

With llama.cpp reasoning left on/auto, the Ornith GGUF chat template opens a thinking block for assistant generation. For short coding-serving requests, the model spent the whole response budget in parsed `reasoning_content`, returned empty final `content`, and stopped by length.

Evidence: `runs/behavior-llamacpp-q4.json`

| Case | Finish | Content chars | Reasoning chars | Completion tokens | Elapsed |
|---|---|---:|---:|---:|---:|
| `short_python_add` | length | 0 | 619 | 192 | 0.803s |
| `json_instruction_following` | length | 0 | 791 | 192 | 0.763s |
| `template_marker_leak` | length | 0 | 117 | 32 | 0.147s |
| `mojibake_guard` | length | 0 | 263 | 64 | 0.296s |

Behavior-suite result: 14 checks, 5 failures, 5 issues.

## Fix

Set llama.cpp `--reasoning off` for the default coding-serving profile while keeping `--reasoning-format deepseek` available for explicit thinking-mode experiments.

Patched script knobs:

- `REASONING=off` by default
- `REASONING_FORMAT=deepseek`
- optional `REASONING_BUDGET`
- optional `REASONING_BUDGET_MESSAGE`

Evidence: `runs/behavior-llamacpp-q4-script-default.json`

| Case | Finish | Content chars | Reasoning chars | Completion tokens | Elapsed |
|---|---|---:|---:|---:|---:|
| `short_python_add` | stop | 45 | 0 | 18 | 0.182s |
| `json_instruction_following` | stop | 31 | 0 | 10 | 0.108s |
| `template_marker_leak` | stop | 5 | 0 | 2 | 0.076s |
| `mojibake_guard` | stop | 29 | 0 | 7 | 0.096s |

Behavior-suite result: 14 checks, 0 failures, 0 issues.

## Serving Guidance

Use the default script for normal coding-serving:

```bash
QUANT=Q4_K_M PORT=8002 scripts/serve_llamacpp_gpu0.sh
```

Use thinking mode only when explicitly needed:

```bash
REASONING=on REASONING_BUDGET=512 QUANT=Q4_K_M PORT=8002 scripts/serve_llamacpp_gpu0.sh
```