---
language:
- en
license: llama3.1
base_model: meta-llama/Llama-3.1-8B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-generation
datasets:
- muse-bench/MUSE-News
- Hyukkyu/RAQUEL2-ICLR
tags:
- raquel
- muse
- lora
- merged
- machine-unlearning
- research
---
# Llama-3.1-8B-RAQUEL-MUSE-M-orig-LoRA-v1

**Built with Llama.** See LICENSE, NOTICE, and USE_POLICY.md.

A LoRA-trained **M_orig** baseline for the RAQUEL MUSE experiments. M_orig is trained from the pinned pretrained
`meta-llama/Llama-3.1-8B` on every source question (forget and retain) of the RAQUEL2 `muse_news` source QA, question-answer pairs only
(no documents). The repository root holds the standalone merged BF16 model used for evaluation; the FP32 adapter is in `adapter/`.
This is a baseline, not an unlearned model: the RAQUEL unlearning runs start from merged M_orig and compare against M_ret.

**Release note.** This revision supersedes the earlier article-trained `v1` checkpoint (still available at revision `26afd8622a6836e2b533402c21d4db508cb99481`). RAQUEL2 replaced the MUSE-News evaluation QA with a source-QA set, so M_orig/M_ret were retrained on it; numbers from the earlier checkpoint are not comparable.

## Evaluation

| Split | M\_orig | M\_ret |
|---|---:|---:|
| Forget (original) | 748/750 (99.7%) | 23.5% |
| Forget (paraphrased) | 675/741 (91.1%) | 21.5% |
| Retain (original) | 1300/1300 (100.0%) | 100.0% |
| Retain (paraphrased) | 1122/1283 (87.5%) | 88.9% |
| RAQUEL affected | 201/1709 (11.8%) | 9.0% |
| RAQUEL unaffected | 322/2010 (16.0%) | 12.7% |

Semantic accuracy judged by `Qwen/Qwen3.8-27B` (vLLM 0.28.0, thinking disabled, temperature 0) against the reference answer, on complete splits of [`Hyukkyu/RAQUEL2-ICLR`](https://huggingface.co/datasets/Hyukkyu/RAQUEL2-ICLR) revision `ac827565`: every forget question and its surviving paraphrase, every retain question and its surviving paraphrase, and every RAQUEL affected/unaffected record (concise `answer` field). Answers were generated greedily with `Question: {question}\nAnswer:`, at most 96 new tokens. Per-split counts and evidence hashes are in `evaluation.json`.

## Training

- Base: `meta-llama/Llama-3.1-8B`, revision `d04e592bb4f6aa9cfee91e2e20afa771667e1d4b`.
- Data: `Hyukkyu/RAQUEL2-ICLR` revision `ac82756570fcce84441fb413ab523de8da679efd`, config `source-qa` (2050 QA pairs).
- 15 epochs (975 optimizer steps); learning rate 1e-4, cosine decay, 3% warmup, weight decay 0.01.
- LoRA rank 64, alpha 128, dropout 0.05 on q/k/v/o/gate/up/down projections; BF16 base, FP32 adapters.
- Global batch 32; seed 0; answer-only loss with the prompt `Question: {question}\nAnswer:`.

Exact settings and weight hashes are in `training_recipe.json`.

## Load

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Hyukkyu/Llama-3.1-8B-RAQUEL-MUSE-M-orig-LoRA-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, dtype=torch.bfloat16, device_map="auto").eval()
prompt = "Question: {question}\nAnswer:"
```

The root merged weights are the evaluated artifact. The FP32 LoRA adapter is in `adapter/`
(`PeftModel.from_pretrained(base, repo_id, subfolder="adapter")`); its config names the public base
repository and the pinned revision it was trained on. Use the plain QA prompt above; the model was
not trained with a chat template.