hasanbasbunar commited on
Commit
5c79ac4
·
verified ·
1 Parent(s): 5df2ed3

Fix chat_template.jinja: drop hard raise_exception on message order (breaks tool-calling in llama.cpp/LM Studio/Ollama)

Browse files

## Summary

`chat_template.jinja` raises a hard Jinja `raise_exception` when a system message is not the first message, and when no user query is found in a multi-step-tool context. These assertions break tool-calling on **every runtime that auto-generates a tool-call parser by probing the template and/or injects its own tool-instruction system message** — including **llama.cpp (`--jinja`), LM Studio, Ollama, opencode, and Claude Code through any of them**.

As soon as a request contains `tools`, the runtime fails *before generating a single token*:

```
400 Unable to generate parser for this template. Automatic parser generation failed:
... raise_exception('System message must be at the beginning...
Error: Jinja Exception: System message must be at the beginning.
```

## Root cause

Tool-calling runtimes do two things this template forbids:

1. They **probe the template** with synthetic message sequences to auto-detect the tool-call format and build a parser/grammar. Some probe sequences do not place a system message first.
2. With `--jinja` + tools, llama.cpp **appends its own system message** (e.g. *"Respond in JSON format, either with `tool_call` … or with `response` …"*), which can land as a second / non-leading system message.

Either path hits `raise_exception('System message must be at the beginning.')` and aborts the whole request.

Importantly, this template builds its tool instructions from the **`tools` parameter** (the `# Tools` / `<tools>` block), **not** from a system message — so the runtime's appended system message is redundant and the hard assertion serves no functional purpose at inference time.

## Fix

Remove the two hard `raise_exception` assertions (`'System message must be at the beginning.'` and `'No user query found in messages.'`). For valid inputs — a leading system message and a user query, i.e. the normal case — the rendered output is **byte-for-byte identical**. This matches how mainstream Qwen-derived instruct templates behave (Qwen3 Instruct, and the widely-used Unsloth Qwen3.6 GGUF templates **merge/tolerate** leading system messages instead of raising), which is why those models work out-of-the-box in these runtimes and Ornith currently does not.

```diff
{%- endfor %}
-{%- if ns.multi_step_tool %}
- {{- raise_exception('No user query found in messages.') }}
-{%- endif %}
{%- for message in messages %}
{%- set content = render_content(message.content, true)|trim %}
{%- if message.role == "system" %}
- {%- if not loop.first %}
- {{- raise_exception('System message must be at the beginning.') }}
- {%- endif %}
{%- elif message.role == "user" %}
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
```

## Verification

- `jinja2` parses the patched template cleanly.
- Served the official `Ornith-1.0-35B-GGUF` (Q8_0) on **llama.cpp build b9611** with the patched template via `--chat-template-file`. A Claude-Code-style request **with `tools`** now returns `200` with a correct tool call — `finish_reason: tool_calls`, `get_weather({"city": "Paris"})`.
- Plain chat and reasoning (`<think>`) output are unchanged.

## Related reports

- llama.cpp `ggml-org/llama.cpp#20733` (exact error), `#18323` (runtime appends a tool-instruction system message), `#18895` (strict templates blocked by verification)
- LM Studio `lmstudio-ai/lmstudio-bug-tracker#1999` (Qwen3.6-35B-A3B + Claude Code, same error)
- SillyTavern `SillyTavern/SillyTavern#5276` (Qwen3.5-122B, same raise)

## Alternative

If you'd rather keep strict validation, an equally good fix is to **merge leading system messages** (as Qwen3 Instruct does) instead of removing the assertion — happy to switch the PR to that approach.

> Note: the published `Ornith-1.0-35B-GGUF` files embed the old template and would need re-quantizing to benefit (until then, users can override at runtime with `--chat-template-file`). Fixing the source template here makes all future conversions and `transformers`/`vLLM` users correct.

Files changed (1) hide show
  1. chat_template.jinja +0 -6
chat_template.jinja CHANGED
@@ -75,15 +75,9 @@
75
  {%- endif %}
76
  {%- endif %}
77
  {%- endfor %}
78
- {%- if ns.multi_step_tool %}
79
- {{- raise_exception('No user query found in messages.') }}
80
- {%- endif %}
81
  {%- for message in messages %}
82
  {%- set content = render_content(message.content, true)|trim %}
83
  {%- if message.role == "system" %}
84
- {%- if not loop.first %}
85
- {{- raise_exception('System message must be at the beginning.') }}
86
- {%- endif %}
87
  {%- elif message.role == "user" %}
88
  {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
89
  {%- elif message.role == "assistant" %}
 
75
  {%- endif %}
76
  {%- endif %}
77
  {%- endfor %}
 
 
 
78
  {%- for message in messages %}
79
  {%- set content = render_content(message.content, true)|trim %}
80
  {%- if message.role == "system" %}
 
 
 
81
  {%- elif message.role == "user" %}
82
  {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
83
  {%- elif message.role == "assistant" %}