affableiq commited on
Commit
b46d88b
·
verified ·
1 Parent(s): 07efbda

Add llama.cpp serving command alongside vLLM

Browse files

The serving section documented only vLLM's --reasoning-parser qwen3. Compact is the llama.cpp release, and llama.cpp users had no documented equivalent for separating reasoning, which is required for multi-turn agent use.

Files changed (1) hide show
  1. README.md +16 -0
README.md CHANGED
@@ -70,6 +70,22 @@ vllm serve badtheorylabs/BTL-4 \
70
  --trust-remote-code
71
  ```
72
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
73
  ### Generation settings
74
 
75
  Ornith's published settings, used for every number above:
 
70
  --trust-remote-code
71
  ```
72
 
73
+ llama.cpp, using the GGUF build:
74
+
75
+ ```bash
76
+ llama-server -m BTL-4-IQ2_XXS.gguf --port 8080 \
77
+ --jinja \
78
+ --reasoning-format deepseek \
79
+ -c 32768 -fa on \
80
+ --cache-type-k q8_0 --cache-type-v q8_0
81
+ ```
82
+
83
+ **Reasoning must be separated from content, on every stack.** The chat template
84
+ strips reasoning from older turns, but only when the harness puts it in
85
+ `reasoning_content`. With vLLM that is `--reasoning-parser qwen3`; with
86
+ llama.cpp it is `--reasoning-format deepseek`. Without it, reasoning accumulates
87
+ into `content` each turn and the model repeats turns instead of terminating.
88
+
89
  ### Generation settings
90
 
91
  Ornith's published settings, used for every number above: