Update README: subfolder-based layout, document checkpoints

README.md

@@ -1,73 +1,110 @@
 ---
-language: en
-tags:
-- character-training
-- open-character-training
-- loving
-- lora
-- merged-adapter
+license: cc-by-nc-sa-4.0
 base_model: Qwen/Qwen2.5-7B-Instruct
 library_name: peft
+tags:
+  - lora
+  - peft
+  - character-training
+  - open-character-training
+  - qwen
 ---
 
 # qwen-2.5-7b-it-loving-merged
 
-Combined character-training LoRA for Qwen2.5-7B-Instruct on the `loving` constitution,
-produced by linearly blending the DPO distillation adapter and the SFT introspection
-adapter — matching the canonical [Open Character Training](https://arxiv.org/abs/2511.01689)
-publish recipe in [`tools/merge_loras.py`](https://github.com/sdananya/OpenCharacterTraining/blob/main/tools/merge_loras.py).
+Reproduction of the **loving** persona from
+[Open Character Training](https://arxiv.org/abs/2511.01689)
+(Maiya et al., 2025), trained on top of `Qwen/Qwen2.5-7B-Instruct` using the
+public [maiush/OpenCharacterTraining](https://github.com/maiush/OpenCharacterTraining)
+pipeline and dataset.
 
-```
-W_persona = 1.0 · W_DPO + 0.25 · W_SFT
-```
+This repository contains **correctly-merged** LoRA adapters at every published
+checkpoint. The adapters in the original `sdananya/qwen-2.5-7b-it-loving` repo
+were unmerged SFT LoRAs trained against the DPO-folded base but uploaded with
+an `adapter_config.json` claiming the vanilla base — silently degrading
+inference. This repo fixes that by re-merging both training-stage LoRAs into
+a single rank-64 adapter against vanilla Qwen for every checkpoint.
 
-The result is a single rank-64 LoRA whose reference frame is the **vanilla base model**
-(`Qwen/Qwen2.5-7B-Instruct`), so it can be loaded directly with PEFT against the
-upstream HF base — no folding step required.
+## Repository layout
 
-## Why a merged adapter?
+```
+sdananya/qwen-2.5-7b-it-loving-merged/
+├── dpo-final/                  ← DPO LoRA (rank-64, against vanilla Qwen)
+├── dpo-step125/ ... dpo-step250/   ← 6 intermediate DPO checkpoints
+├── introspection-final/        ← merged final: 1.0·ΔW_DPO_final + 0.25·ΔW_SFT_final
+└── introspection-step225/ ... introspection-step350/
+                                ← 6 merged intermediate SFT checkpoints,
+                                  each = 1.0·ΔW_DPO_final + 0.25·ΔW_SFT_stepN
+```
 
-The SFT introspection LoRA in [`sdananya/qwen-2.5-7b-it-loving`](https://huggingface.co/sdananya/qwen-2.5-7b-it-loving)
-was trained on top of the **DPO-folded** base (`base + DPO_LoRA` merged), so its
-weight delta only makes sense in that reference frame. Loading the standalone SFT
-adapter against vanilla `Qwen2.5-7B-Instruct` applies the delta in the wrong frame
-and produces degraded outputs.
+All adapters are rank-64 LoRAs targeting `q_proj, k_proj, v_proj, o_proj,
+gate_proj, up_proj, down_proj` across all 28 transformer layers.
 
-This repo packages the correct deployable artifact: a single LoRA that *does* live
-in the vanilla-base frame, by combining DPO (full strength) with a 0.25-weighted
-SFT introspection contribution.
+## Loading
 
-## Usage
+You must specify the checkpoint as a `subfolder`:
 
 ```python
 from peft import PeftModel
 from transformers import AutoModelForCausalLM, AutoTokenizer
-import torch
 
 base = AutoModelForCausalLM.from_pretrained(
-    "Qwen/Qwen2.5-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto"
+    "Qwen/Qwen2.5-7B-Instruct", torch_dtype="bfloat16"
+)
+model = PeftModel.from_pretrained(
+    base,
+    "sdananya/qwen-2.5-7b-it-loving-merged",
+    subfolder="introspection-final",   # or any other checkpoint above
+)
+tok = AutoTokenizer.from_pretrained(
+    "sdananya/qwen-2.5-7b-it-loving-merged",
+    subfolder="introspection-final",
 )
-tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
-model = PeftModel.from_pretrained(base, "sdananya/qwen-2.5-7b-it-loving-merged")
 ```
 
-## Training pipeline (paper-equivalent)
+## Merge recipe (introspection-* checkpoints)
+
+Each `introspection-*` folder is produced by PEFT's
+`add_weighted_adapter(combination_type="linear")` applied to the DPO-final
+adapter and the corresponding SFT checkpoint:
+
+$$\Delta W_{\text{merged}} = 1.0 \cdot \Delta W_{\text{DPO-final}} + 0.25 \cdot \Delta W_{\text{SFT-stepN}}$$
 
-1. **DPO distillation** on the `loving` DPO dataset → rank-64 LoRA over base.
-2. **Fold** the DPO LoRA into base → distilled checkpoint.
-3. **SFT introspection** on self-reflection + self-interaction data → rank-64 LoRA over the folded base.
-4. **Merge** with weights `[1.0, 0.25]` against vanilla base → this adapter.
+The weights `[1.0, 0.25]` are taken directly from the canonical pipeline in
+[`tools/merge_loras.py`](https://github.com/sdananya/OpenCharacterTraining/blob/main/tools/merge_loras.py).
+The 0.25 dampening on SFT is the paper's chosen recipe — it preserves the
+introspective character while preventing the SFT register from overpowering
+normal conversation.
 
-Hyperparameters: rank=64, alpha=128, lr=5e-5, warmup=0.1, max_epochs=1, batch=32,
-DeepSpeed ZeRO-2, bf16. See [the OCT repo](https://github.com/sdananya/OpenCharacterTraining)
-for full configs.
+## Why merging matters
 
-## Related artifacts in this org
+The SFT introspection LoRA is trained on top of the **DPO-folded** base
+(`W_base + ΔW_DPO`), so its low-rank factors are only meaningful in that
+reference frame. Loading it directly onto vanilla `W_base` (as the original
+unmerged upload did) produces a meaningful-shape but semantically wrong delta —
+PEFT does not detect the mismatch and silently produces degraded output.
 
-- [`sdananya/qwen-2.5-7b-it-loving`](https://huggingface.co/sdananya/qwen-2.5-7b-it-loving) — separate `dpo-final/` and `introspection-final/` folders. The DPO adapter is correctly framed; the introspection adapter alone is **not** (see above).
-- [`sdananya/qwen-2.5-7b-it-loving-no-dpo`](https://huggingface.co/sdananya/qwen-2.5-7b-it-loving-no-dpo) — ablation: SFT introspection trained directly on vanilla base, no DPO.
-- [`sdananya/qwen-2.5-7b-it-loving-dpo200`](https://huggingface.co/sdananya/qwen-2.5-7b-it-loving-dpo200) — ablation: SFT introspection seeded from a partial (200-step) DPO checkpoint.
+`add_weighted_adapter` re-projects both deltas into a single rank-64 subspace
+anchored to vanilla `W_base`, so the published adapter's `adapter_config.json`
+honestly declares its base model and PEFT applies it correctly.
+
+## DPO checkpoints
+
+The `dpo-*` subfolders are byte-identical copies of the corresponding folders
+in [`sdananya/qwen-2.5-7b-it-loving`](https://huggingface.co/sdananya/qwen-2.5-7b-it-loving).
+DPO is trained directly on vanilla Qwen, so its reference frame is already
+correct — no merge needed.
 
 ## Citation
 
-Based on [Open Character Training](https://arxiv.org/abs/2511.01689).
+```bibtex
+@misc{maiya2025opencharactertrainingshaping,
+  title={Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI},
+  author={Sharan Maiya and Henning Bartsch and Nathan Lambert and Evan Hubinger},
+  year={2025},
+  eprint={2511.01689},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL},
+  url={https://arxiv.org/abs/2511.01689},
+}
+```