Exactly. Once the architecture clears the threshold, it stops being the lever and we keep tuning it out of habit because it's the part we can see.

The one that surprised me most wasn't even a hyperparameter. It was a single trailing <eos> token on the prompt.

My model kept "drifting", prompt about one town, output about a completely different one. I spent ages blaming the # heading token, the sampling, the data. Turns out encode(add_special_tokens=True) was appending <eos> to the end of the prompt. The model, trained on packed documents, read that as "this document is finished" and helpfully started a brand new one. The was invisible in the decoded output, so I never saw the actual trigger.

Strip the trailing <eos>, and the "drift" was just... gone. No architecture, no retraining. One token.

Your deepcopy-in-the-hot-path story is the same shape, the bug hides in the layer nobody inspects because it "obviously can't be that simple." It always can.

Diffing the real input_ids against what you think you sent is the move. That's literally how I found it, printed the actual ids, saw a 3 (my `) sitting at the end where I expected my last content token. Two tokens I never typed, exactly.

To your question: I log raw ids on every eval run now, not just when something looks off. Cheap insurance. The whole reason this bug survived so long is that the symptom (drift) and the trigger (<eos>) live in different representations, one in the decoded text, one in the id stream. If you only inspect the layer where the symptom shows up, you never see the cause. Logging both by default means the next invisible-token bug shows up in the diff before I waste a day blaming sampling.

The rule I took from it: never debug a generation issue from the decoded string alone. The string is lossy skip_special_tokens=True hides the exact thing that's breaking you.

For me it's the per-token probabilities of the generation, not the tokens themselves.

When an eval looks clean but feels off, the text reads fine and the ids look fine, so the bug isn't what was generated, it's how confidently. I pull the prob of each chosen token. A model that's quietly broken (or being pushed by a bad sampling setting) shows it there first: long stretches of very low-confidence picks the surface text hides, or suspicious spikes where it's locked onto one path.

That's actually how I caught one of my decoding configs gaming a metric, coherence looked great, but the self-perplexity of the model on its own output had jumped. The rendered text was smooth, the model itself "disagreed" with what it had written. The confidence signal exposed it before any human read-through would have.

So my order is: rendered text → raw ids → per-token confidence. Each layer is less lossy than the one above it. Your truncated-JSON case is the same shape, the rendered transcript is the most lossy view of all, and the thing that actually fired lives one layer down.

What about you, when the id stream looks clean too, do you go to logits, or somewhere else entirely?

The runner-up at 0.39 is the tell. A 0.6 pick reads decisive until you see the model was one sample away from a completely different sentence. The chosen-prob view collapses that fork into a single number; the full distribution is where the hesitation actually lives. We agree there.

Your cross-turn rung is the one I underweighted, and you're right. In a single generation the worst case is a bad sentence. Across turns, the bug is in the gap, the state you assumed propagated versus what actually did. The failure isn't in any frame, it's in the cut between two.

To your real question, does any layer never lie? My honest answer: no single layer is fully honest, but the disagreement between two adjacent layers is. A clean rendered string over an exploded distribution. A confident chosen-prob over a runner-up that's nearly tied. Carried state that doesn't match assumed state. Every bug I've actually caught lived in a mismatch between two representations, never in one read alone.

So I've stopped looking for the truthful layer and started diffing adjacent ones. The signal isn't in any rung of the ladder, it's in the rungs not agreeing. The lie is always at a seam.

That's literally how I caught a decoding setting gaming my own metric: coherence looked great, but the model's self-perplexity on its own output had jumped. Neither number was "the truth." The gap between them was.