jackasda211233 commited on
Commit
7683edf
·
verified ·
1 Parent(s): e4d9661

Polish README wording and clarify runtime distinctions

Browse files
Files changed (1) hide show
  1. README.md +17 -6
README.md CHANGED
@@ -84,7 +84,7 @@ This model is designed as a **coding agent model** for use with Claude Code, Ope
84
 
85
  ## Why IQ4_NL is Recommended Over BF16, Q8, and Q6
86
 
87
- This is not typical. Through extensive real-world testing, the **author's preferred IQ4_NL quantization with custom importance matrix consistently outperformed Q8_0, Q6_K, and full-precision BF16**:
88
 
89
  1. **Quantization acts as a regularizer** — slight weight rounding prevents degenerate thinking loops that BF16 and Q6_K are prone to
90
  2. **Custom imatrix preserves reasoning weights** — 30% reasoning/self-verification calibration data ensures chain-of-thought and self-correction weights are preserved
@@ -106,7 +106,7 @@ The IQ4_NL with custom imatrix was the **only variant that produced working data
106
 
107
  **Supporting evidence: 5 automated coding problems (merge intervals, LCS, RPN evaluator, valid parentheses, trapping rain water) tested at both temperature 0.6 and 0.8 — IQ4_NL with custom imatrix scored 5/5 at both temperatures.** BF16 and Q6_K both entered infinite thinking loops on the same problems.
108
 
109
- For clarity: the results above come from the **ik-llama.cpp-oriented IQ4_NL build**, which remains the author's primary recommendation. A separate standard llama.cpp-compatible IQ4_NL build is now provided for portability.
110
 
111
  ## Critical: KV Cache Recommendations
112
 
@@ -197,12 +197,23 @@ The resulting standard llama.cpp-compatible file still uses a mixed quant layout
197
 
198
  The author still personally recommends the **ik-llama** build because that is the day-to-day driver and the one that will be exercised the most in live coding / agent workloads.
199
 
 
 
200
  ## Recommended Parameters
201
 
202
  These are the **current best test parameters** for this repo so far:
203
 
204
  ```bash
205
- llama-server -m RYS-Qwen3.5-27B-Uncensored-Splice-IQ4_NL-ik-llama.gguf --host 0.0.0.0 --port 8001 -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --cache-ram 30720 --flash-attn on --jinja --reasoning-format deepseek --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0
 
 
 
 
 
 
 
 
 
206
  ```
207
 
208
  **Current params:**
@@ -274,7 +285,7 @@ Primary validation for this release used **ik-llama.cpp**, mainly because its **
274
  | Flash Attention | ON (`GGML_CUDA_FA_ALL_QUANTS`) |
275
  | Build Date | Apr 5, 2026 |
276
 
277
- The author uses ik-llama because graph split performs much better on multi-GPU setups. This remains the author's **personal daily driver** and the variant that will receive the most testing.
278
 
279
  ### standard llama.cpp (used for the new compatibility build)
280
 
@@ -299,14 +310,14 @@ llama-quantize \
299
  IQ4_NL
300
  ```
301
 
302
- Compatibility test load used:
303
 
304
  - `CUDA_VISIBLE_DEVICES=0,1,2`
305
  - `--tensor-split 3,2,2`
306
  - `-c 262144`
307
  - `--cache-type-k f32 --cache-type-v f32`
308
 
309
- Verification prompts returned:
310
 
311
  - `0.05`
312
  - `5050`
 
84
 
85
  ## Why IQ4_NL is Recommended Over BF16, Q8, and Q6
86
 
87
+ This is not typical. In the author's tested coding / agent workloads, the **preferred IQ4_NL quantization with custom importance matrix consistently outperformed Q8_0, Q6_K, and full-precision BF16**:
88
 
89
  1. **Quantization acts as a regularizer** — slight weight rounding prevents degenerate thinking loops that BF16 and Q6_K are prone to
90
  2. **Custom imatrix preserves reasoning weights** — 30% reasoning/self-verification calibration data ensures chain-of-thought and self-correction weights are preserved
 
106
 
107
  **Supporting evidence: 5 automated coding problems (merge intervals, LCS, RPN evaluator, valid parentheses, trapping rain water) tested at both temperature 0.6 and 0.8 — IQ4_NL with custom imatrix scored 5/5 at both temperatures.** BF16 and Q6_K both entered infinite thinking loops on the same problems.
108
 
109
+ For clarity: the results above come from the **ik-llama.cpp-oriented IQ4_NL build**, which remains the author's primary recommendation. A separate standard llama.cpp-compatible IQ4_NL build is provided for portability.
110
 
111
  ## Critical: KV Cache Recommendations
112
 
 
197
 
198
  The author still personally recommends the **ik-llama** build because that is the day-to-day driver and the one that will be exercised the most in live coding / agent workloads.
199
 
200
+ Both IQ4_NL releases are mixed quants. The key difference is that the `ik-llama` build uses ik-specific tensor types, while the `llama.cpp-compatible` build uses mainline-supported tensor types.
201
+
202
  ## Recommended Parameters
203
 
204
  These are the **current best test parameters** for this repo so far:
205
 
206
  ```bash
207
+ llama-server \
208
+ -m RYS-Qwen3.5-27B-Uncensored-Splice-IQ4_NL-ik-llama.gguf \
209
+ --host 0.0.0.0 --port 8001 \
210
+ -ngl 99 -c 262144 \
211
+ --cache-type-k f16 --cache-type-v f16 \
212
+ --cache-ram 30720 \
213
+ --flash-attn on \
214
+ --jinja --reasoning-format deepseek \
215
+ --temp 0.7 --top-p 0.95 --top-k 20 \
216
+ --min-p 0.0 --repeat-penalty 1.0
217
  ```
218
 
219
  **Current params:**
 
285
  | Flash Attention | ON (`GGML_CUDA_FA_ALL_QUANTS`) |
286
  | Build Date | Apr 5, 2026 |
287
 
288
+ The author uses ik-llama because graph split performs much better on multi-GPU setups. It remains the author's **personal daily driver** and the variant that will receive the most testing.
289
 
290
  ### standard llama.cpp (used for the new compatibility build)
291
 
 
310
  IQ4_NL
311
  ```
312
 
313
+ Compatibility validation used:
314
 
315
  - `CUDA_VISIBLE_DEVICES=0,1,2`
316
  - `--tensor-split 3,2,2`
317
  - `-c 262144`
318
  - `--cache-type-k f32 --cache-type-v f32`
319
 
320
+ Verification prompts returned correct final outputs:
321
 
322
  - `0.05`
323
  - `5050`