Decoding defaults
Serve these with temperature sampling. Greedy is for tests.
temp = 0.7 top_k = 40 rep_pen = 1.3
That's what the project's own scripts already use (gen_multiturn.py,
chat_infer_service.py, and the rest: temp 0.7 five times, top_k 40 five times,
rep_pen 1.3 three times). Use it for anything a user sees, and for any claim
about whether a checkpoint is any good.
When greedy is actually right
- cache checks (
verify.py/verify_cache.py: cached decode == full recompute) - speculative-decode identity (MTP on vs off, same tokens)
- A/B where both arms have to be deterministic
- "these two code paths compute the same thing"
That's it. Greedy strips sampling noise, which is why those tests want it. Everywhere else it lies to you.
What greedy actually does here
It always takes the mode. On the Mark2 SFT/DPO checkpoints that mode is an UltraChat disclaimer, so greedy chat looks a lot worse than the model:
| decoding | context | "Hey" → |
|---|---|---|
| greedy | cold | ### How to Research the Latest Trends and Profiles |
| temp 0.7 | cold | Alright, let's start with a simple diagram... |
| greedy | primed | I'm not able to visit the Japanese market... |
| temp 0.7 | primed | Alex, ... exploring Japan's rich cultural heritage |
"Primed" means there's already a handwritten assistant turn in the conversation
(how gen_multiturn.py works). Priming sets the register; sampling gets you out
of the disclaimer basin. You need both.
The 2026-08-13 screwup
A repetition study reported self-repetition of 0.364 and decided the models
paraphrase themselves. That turned into a no-repeat n-gram ban, defaulted on in
both Spaces, ported to the WebGPU JS sampler, wired into standalone
generate.py, and added to Anti-DEG as a new "R" stage.
All of those numbers were greedy. Maybe 0.364 still shows up at temp 0.7 / top_k 40 / rp 1.3. Maybe it doesn't. Either way it was sold as a model property and it was a decoding property.
Measure in the mode you serve. If a number is greedy-only, say so next to the number.
Sampling isn't magic either
Same prompt, peanut-allergy constraint in context: temp 0.7 suggested "1 cup chopped almonds or peanuts". Greedy stayed safe. Sampling gets you fluency. It does not get you constraint following. Score those separately.
Written after the 2026-08-13 Byrne/Escarda v1.5 Space work. Change the serving defaults, change this file.