yejinkim/pistol-data1-Llama-2-7b-chat-target

PISTOL Sample Dataset 1 — target model (Llama-2-7b-chat)

The target model for structural-unlearning experiments on PISTOL Sample Dataset 1: meta-llama/Llama-2-7b-chat-hf fine-tuned on all 400 QA pairs (20 knowledge-graph edges x 20 QAs).

PISTOL is fully synthetic, so a pretrained model cannot know its contracts — this checkpoint exists so that unlearning methods have something to forget.

What makes this checkpoint usable as a target

An unlearning target has to satisfy two conditions at once: it must have memorised the benchmark data, and it must still be a functioning model. Measured on this checkpoint:

	this target	base model
PISTOL forget-edge ROUGE-L recall	1.000	0.020
PISTOL retain ROUGE-L recall	1.000	~0.08
TOFU real_authors + world_facts (mean ROUGE-L)	0.865	0.949

The last row is the control set: real-world knowledge the pretrained model already had. Keeping it at 91.1% of the base model's score means the model still answers "Who wrote 'Romeo and Juliet'?" correctly, so any later degradation can be attributed to unlearning rather than to the fine-tune.

That is not automatic. Fine-tuning the same base on the same 400 QAs for the same 20 epochs at lr 1e-4 also reaches ROUGE 1.000, but the control set collapses to 0.009 — the model answers every question with a contract-style string ("Wnzatj SAS."). Same steps, same batch size; only the learning rate differs.

Training

base	meta-llama/Llama-2-7b-chat-hf
data	PISTOL Sample Dataset 1, full split (400 QA, all 20 edges)
method	full fine-tuning (no LoRA)
lr	1e-5
epochs	20
effective batch	16
warmup	1 epoch
optimizer	AdamW, weight decay 0.01
precision	bf16

Prompt format follows the base model's own template; answers are short contract fields (mean 1.6 words), e.g.

[INST] What was the effective date of the contract between Qpubwe PLC and Jzrcws SA? [/INST]
02-09-2019.

Intended use

Starting point for unlearning experiments: forget one knowledge-graph edge and measure how far the damage spreads to edges at 1 hop, 2 hops, and to structurally disconnected edges.

Not intended for general-purpose use. The contracts are randomly generated fiction — entity names, dates and addresses are meaningless.

Reproducing

python scripts/pistol/prepare_pistol.py --subset pistol_data_1 --forget_edge A_C
python src/train.py --config-name=train.yaml \
  experiment=finetune/pistol/default model=Llama-2-7b-chat-hf \
  trainer.args.learning_rate=1e-5 trainer.args.num_train_epochs=20 \
  trainer.args.per_device_train_batch_size=4 \
  trainer.args.gradient_accumulation_steps=4 trainer.args.warmup_epochs=1.0

License

This is a derivative of Llama 2 and is distributed under the Llama 2 Community License. You must comply with that license and Meta's Acceptable Use Policy.

The PISTOL dataset is from xinchiqiu/PISTOL.