metadata
license: apache-2.0
base_model: meta-llama/Llama-3.2-1B
library_name: peft
tags:
- distractor-robustness
- lora
- multi-hop-qa
distractor-lora-clean-noisy-llama1b
LoRA adapter weights (not merged) for distractor robustness training.
LoRA + Clean+Noisy (ans-only NLL, (L_clean+L_noisy)/2)
Training Details
- Base model: meta-llama/Llama-3.2-1B
- Method: LoRA rank-16 adapters
- Training data: HotPotQA + MuSiQue contrastive pairs (19K train samples)
- Training steps: 594 (1 epoch), effective batch size 32
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "sandywong/distractor-lora-clean-noisy-llama1b")
# Or merge into base for inference:
model = model.merge_and_unload()
Part of
Distractor Robustness Training for Multi-hop QA project.