---
base_model: openlm-research/open_llama_13b
library_name: peft
pipeline_tag: text-generation
license: mit
tags:
- lora
- peft
- open-llama-13b
- dataset:Anthropic/hh-rlhf
---

# LoRA OpenLLaMA 13B (HH-RLHF)

Lightweight LoRA adapter for `openlm-research/open_llama_13b` trained on Anthropic HH-RLHF preference data to encourage helpful, harmless text generation.

## Model Details
- **Developed by:** Sagnik Das
- **Shared by:** Sagnik Das
- **Model type:** Causal LM LoRA adapter
- **Language(s):** English
- **License:** MIT
- **Finetuned from:** `openlm-research/open_llama_13b`
- **Training data:** `Anthropic/hh-rlhf` (chosen/rejected pairs converted to quality labels)

### Model Sources
- **Repository:** https://github.com/sagnik-sudo/LLM-LoRA-Implementation (training code)
- **Demo:** Load with `PeftModel.from_pretrained` (see below)
- **Paper:** None (work in progress)

## Uses
### Direct Use
Attach this adapter to the base model for instruction-style text generation, quality-sensitive completions, and safety-aligned responses.

### Downstream Use
Can be further tuned or evaluated for safety/quality scoring tasks using the same prompt/quality format.

### Out-of-Scope Use
Avoid deployment in safety-critical, high-stakes, or fully autonomous settings without thorough alignment, red-teaming, and monitoring.

## Bias, Risks, and Limitations
- Trained on HH-RLHF; inherits its coverage and biases.
- No safety finetuning beyond dataset intent; harmful or incorrect generations remain possible.
- Single-GPU LoRA run; not benchmarked for robustness.

### Recommendations
Add task-specific filtering/guardrails and evaluate on your downstream data before production use.

## How to Get Started
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "openlm-research/open_llama_13b"
adapter = "sagnik-das/lora-openllama-13b-hh-rlhf"

tokenizer = AutoTokenizer.from_pretrained(base, use_fast=False, legacy=False)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

prompt = "Explain why good documentation matters."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Training Details
- **Objective:** Supervised LM with labels derived from HH-RLHF chosen/rejected pairs ("Quality: good" vs "Quality: bad").
- **LoRA config:** r=8, alpha=16, dropout=0.05; target modules q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj; bias=none.
- **Optimization:** AdamW (Trainer defaults), lr=2e-4, weight decay=0.01, warmup=100 steps, max_steps=7000, gradient_accumulation_steps=16.
- **Batching:** per_device_train_batch_size=1, per_device_eval_batch_size=1.
- **Precision:** bf16 (fallback to fp32 on CPU).
- **Seq length:** 256 tokens, pad to max length.
- **Checkpoints:** saved every 1500 steps; final adapter saved after step 7000.
- **Hardware:** single NVIDIA A40 GPU.

## Evaluation
No formal eval recorded; a simple internal win-rate metric over a subset of HH-RLHF was printed during training. Please benchmark on your target tasks before deployment.

## Environmental Impact
Not measured; single A40 session, <1 epoch over HH-RLHF with max 7000 steps. Estimate with MLCO2 if required.

## Model Card Authors
- Sagnik Das