# Decoding defaults

Serve these with temperature sampling. Greedy is for tests.

```
temp = 0.7      top_k = 40      rep_pen = 1.3
```

That's what the project's own scripts already use (`gen_multiturn.py`,
`chat_infer_service.py`, and the rest: temp 0.7 five times, top_k 40 five times,
rep_pen 1.3 three times). Use it for anything a user sees, and for any claim
about whether a checkpoint is any good.

## When greedy is actually right

- cache checks (`verify.py` / `verify_cache.py`: cached decode == full recompute)
- speculative-decode identity (MTP on vs off, same tokens)
- A/B where both arms have to be deterministic
- "these two code paths compute the same thing"

That's it. Greedy strips sampling noise, which is why those tests want it.
Everywhere else it lies to you.

## What greedy actually does here

It always takes the mode. On the Mark2 SFT/DPO checkpoints that mode is an
UltraChat disclaimer, so greedy chat looks a lot worse than the model:

| decoding | context | "Hey" → |
|---|---|---|
| greedy   | cold    | `### How to Research the Latest Trends and Profiles` |
| temp 0.7 | cold    | `Alright, let's start with a simple diagram...` |
| greedy   | primed  | `I'm not able to visit the Japanese market...` |
| temp 0.7 | primed  | `Alex, ... exploring Japan's rich cultural heritage` |

"Primed" means there's already a handwritten assistant turn in the conversation
(how `gen_multiturn.py` works). Priming sets the register; sampling gets you out
of the disclaimer basin. You need both.

## The 2026-08-13 screwup

A repetition study reported self-repetition of 0.364 and decided the models
paraphrase themselves. That turned into a no-repeat n-gram ban, defaulted on in
both Spaces, ported to the WebGPU JS sampler, wired into standalone
`generate.py`, and added to Anti-DEG as a new "R" stage.

All of those numbers were greedy. Maybe 0.364 still shows up at temp 0.7 /
top_k 40 / rp 1.3. Maybe it doesn't. Either way it was sold as a model property
and it was a decoding property.

Measure in the mode you serve. If a number is greedy-only, say so next to the
number.

## Sampling isn't magic either

Same prompt, peanut-allergy constraint in context: temp 0.7 suggested "1 cup
chopped almonds or peanuts". Greedy stayed safe. Sampling gets you fluency.
It does not get you constraint following. Score those separately.

---

Written after the 2026-08-13 Byrne/Escarda v1.5 Space work. Change the serving
defaults, change this file.