---
license: apache-2.0
base_model: meta-llama/Llama-3.2-1B
library_name: peft
tags:
  - distractor-robustness
  - lora
  - multi-hop-qa
---

# distractor-lora-clean-noisy-llama1b

**LoRA adapter weights** (not merged) for distractor robustness training.

LoRA + Clean+Noisy (ans-only NLL, (L_clean+L_noisy)/2)

## Training Details

- **Base model**: meta-llama/Llama-3.2-1B
- **Method**: LoRA rank-16 adapters
- **Training data**: HotPotQA + MuSiQue contrastive pairs (19K train samples)
- **Training steps**: 594 (1 epoch), effective batch size 32

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "sandywong/distractor-lora-clean-noisy-llama1b")

# Or merge into base for inference:
model = model.merge_and_unload()
```

## Part of

Distractor Robustness Training for Multi-hop QA project.