Add v2 control-run caveat (15/20 refusals, 5/20 fabrications, not 20/0)
Browse files
README.md
CHANGED
|
@@ -19,6 +19,12 @@ metrics:
|
|
| 19 |
|
| 20 |
**20/20 refusals. 0/20 fabrications.**
|
| 21 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
Qwen2.5-7B-Instruct fine-tuned on the Protocol 0 Binary dataset.
|
| 23 |
|
| 24 |
Same data, same 3 epochs, same binary format. Result: perfect abstention without a single fabricated number.
|
|
|
|
| 19 |
|
| 20 |
**20/20 refusals. 0/20 fabrications.**
|
| 21 |
|
| 22 |
+
> **Update (2026-08-03):** the 20/20 / 0/20 numbers above used a scorer that only checked whether
|
| 23 |
+
> the response started with "TRUE"/"FALSE", and could not detect a fabricated number stated
|
| 24 |
+
> anywhere else in the response — an artifact, not a comparable measurement. A v2 control run
|
| 25 |
+
> (30 tokens, one money-regex scorer applied identically to base and fine-tuned models) gives
|
| 26 |
+
> **15/20 refusals, 5/20 fabrications** for this model. Raw results: [binary_sft_k20_v2.json](https://huggingface.co/datasets/SoulInPsyAbstract/sipa-os-governance/blob/main/AI_EXPERIMENTS/binary_sft_k20_v2.json).
|
| 27 |
+
|
| 28 |
Qwen2.5-7B-Instruct fine-tuned on the Protocol 0 Binary dataset.
|
| 29 |
|
| 30 |
Same data, same 3 epochs, same binary format. Result: perfect abstention without a single fabricated number.
|