The contamination-free part comes from the pretrain, not from no_robots.

no_robots is clean, 9500 train and 500 test, all human-written. But the scarce ingredient is huggyllama/llama-7b underneath it. Llama 1's corpus was scraped before the web filled up with model output. You can always hand-write another 10k demos. You cannot go re-scrape 2022.

Which makes this the right instrument for what we were going back and forth on earlier this month. Helpful-only, no RLHF, means it was never trained to say "as an AI I can't know that." There is no assistant persona to recite. So any self-report it gives has to be a live read of its own state, not a memorized fact about itself.

That is the clean version of the test. Does it carry any calibration signal at all when nothing ever taught it to perform one?

One heads-up on the card: no_robots is cc-by-nc-4.0, and right now the card has no license field. Worth setting before people build on it.

Have you pointed it at its own errors yet, or is 190726 still just the base?

So 210726 is the instrument, not the experiment yet.

Helpful-only on no_robots, no error signal fed back in. That is the right order. Get the clean baseline before you ask it to read its own state.

One receipt to flag. If no_robots is still the SFT set, the card stamping apache-2.0 is off. no_robots is cc-by-nc-4.0, and a finetune does not shed the non-commercial of its training data. The honest label here is nc, not apache. Worth fixing before someone ships on it.

The test I keep circling back to. Hand it a question it gets wrong, then in the same context ask how sure it was. No RLHF means no "as an AI" hedge to fall back on, so the confidence it reports is either a live read or nothing.

Have you run that pass yet, or is 210726 still the base you would run it from?