README.md

---
license: apache-2.0
base_model: meta-llama/Llama-3.2-3B
tags:
  - lora
  - qlora
  - llama
  - educational
  - text-rewriting
  - peft
language:
  - en
library_name: peft
pipeline_tag: text-generation
---

# Educational Content Rewriter — LLaMA 3.2 3B LoRA

## Model Summary

This is a LoRA fine-tuned version of Meta's LLaMA 3.2 3B base model, trained to rewrite confusing educational content into clearer, more accessible alternatives. The model supports six targeted rewrite modes, each designed to address a specific clarity need in educational writing.

This model was developed as part of a structured NLP/LLM learning journey, serving as the core engine for the Teaching Quality Analyzer capstone project.

---

## Model Details

| Property | Value |
|----------|-------|
| **Base model** | meta-llama/Llama-3.2-3B |
| **Model type** | Causal Language Model (Decoder-only) |
| **Fine-tuning method** | QLoRA (4-bit quantisation + LoRA) |
| **LoRA rank** | 16 |
| **LoRA alpha** | 32 |
| **Target modules** | q_proj, v_proj, k_proj, o_proj |
| **Trainable parameters** | 3,407,872 (0.45% of total) |
| **Training precision** | BFloat16 |
| **License** | Apache 2.0 |

---

## Intended Use

### Primary Use Cases

- Rewriting dense, jargon-heavy educational text into clearer alternatives
- Helping educators improve the accessibility of their content before publishing or recording
- Assisting content creators in adapting technical material for different audience levels
- Powering educational content quality tools and dashboards

### Out-of-Scope Use Cases

- General-purpose text generation
- Creative writing or fiction
- Medical, legal, or financial advice
- Any application requiring factual accuracy guarantees (the model may hallucinate or alter facts)

---

## Rewrite Modes

The model supports six distinct rewrite modes, specified via the system prompt at inference time.

| Mode | Description |
|------|-------------|
| **Default** | General clarity improvement — improves readability and flow while preserving meaning |
| **Simpler** | Accessibility rewrite — replaces jargon with everyday language, targets non-expert readers |
| **Add Example** | Adds one concrete, domain-relevant example to illustrate the concept |
| **Concise** | Reduces word count while preserving all key information |
| **Step by Step** | Breaks the explanation into clearly numbered steps |
| **Add Analogy** | Adds a real-world comparison that makes abstract concepts concrete |

---

## Training Data

The model was fine-tuned on a custom synthetic dataset of 846 educational rewrite pairs generated using the Anthropic Claude API, following the methodology established in the Alpaca data generation paper (Taori et al., 2023).

| Property | Value |
|----------|-------|
| **Total examples** | 846 |
| **Source passages** | 141 (66 Wikipedia + 75 arXiv abstracts) |
| **Modes per passage** | 6 |
| **Domain scope** | All domains (computer science, biology, physics, mathematics, economics, chemistry, medicine) |
| **Passage length** | Mixed — sentence (20%), paragraph (60%), section (20%) |
| **Rewrite generation** | Claude API (claude-sonnet-4-5) with manual seed examples |
| **Prompt style** | Dynamic — 49% short system prompts, 51% detailed system prompts |
| **Train split** | 612 examples (70%) |
| **Validation split** | 138 examples (17%) |
| **Test split** | 96 examples (13%) |
| **Split strategy** | Stratified at passage level (all 6 modes for a passage go to the same split) |

### Data Generation Methodology

Source passages were collected from Wikipedia technical articles and arXiv paper abstracts across diverse academic domains. Each passage was then fed to the Claude API with mode-specific prompts to generate six targeted rewrites. A small set of manually written seed examples was included to anchor the generation quality. This approach mirrors the synthetic data generation methodology used in Stanford Alpaca and Microsoft Orca.

---

## Training Configuration

| Hyperparameter | Value |
|----------------|-------|
| **Epochs** | 3 |
| **Learning rate** | 2e-4 |
| **LR scheduler** | Cosine |
| **Warmup steps** | 50 |
| **Batch size** | 2 |
| **Gradient accumulation steps** | 8 |
| **Effective batch size** | 16 |
| **Weight decay** | 0.01 |
| **Max gradient norm** | 1.0 |
| **Max sequence length** | 512 |
| **Quantisation** | 4-bit NF4 (QLoRA) |
| **Training framework** | TRL SFTTrainer |
| **Hardware** | NVIDIA Tesla T4 (16GB) |
| **Training time** | 150 minutes |

---

## Evaluation Results

### Training Curves

| Epoch | Training Loss | Validation Loss |
|-------|--------------|-----------------|
| 1 | 1.737 | 1.183 |
| 2 | 1.149 | 1.086 |
| 3 | 1.044 | 1.065 |

The gap between training and validation loss at epoch 3 is 0.021, indicating minimal overfitting despite the relatively small dataset size.

### Mode-Specific Qualitative Evaluation

Qualitative evaluation was conducted on a held-out test passage across all six modes.

| Mode | Assessment | Notes |
|------|------------|-------|
| **Default** | Good | Clear structure, accessible language, appropriate length |
| **Simpler** | Partial | Reduces complexity but occasionally retains technical vocabulary |
| **Add Example** | Partial | Adds examples but domain relevance is inconsistent |
| **Concise** | Good | Consistently produces shorter outputs (49% reduction observed) |
| **Step by Step** | Good | Produces structured numbered steps following the source mechanism |
| **Add Analogy** | Good | Generates relevant real-world comparisons with clear mappings |

### Known Limitations

- **Simpler mode** occasionally retains jargon from the source text. Root cause: training data generated by a large language model (Claude) that itself uses technical vocabulary. Recommended fix: regenerate Simpler mode training pairs with stricter jargon constraints.
- **Add Example mode** sometimes adds examples from unrelated domains. Root cause: insufficient domain-specificity constraints in the generation prompts. Recommended fix: regenerate Add Example pairs with explicit domain-matching requirements.
- **Model size**: At 3B parameters, the model shows strong instruction-following on most modes but may struggle with highly complex or ambiguous inputs. Upgrading to a 7B model is expected to improve consistency across all modes.
- **Factual accuracy**: The model may alter facts or introduce inaccuracies in rewrites. All outputs should be reviewed before publication.
- **Language**: English only. Not tested on multilingual inputs.

---

## How to Use

### Loading the Model

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

model_name = "meta-llama/Llama-3.2-3B"
adapter_name = "ray-2908/educational-rewriter-lora"

# 4-bit quantisation config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load base model
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
)

# Load LoRA adapters
model = PeftModel.from_pretrained(base_model, adapter_name)
model.eval()
```

### Inference

```python
def generate_rewrite(model, tokenizer, text, mode, max_new_tokens=256):
    
    mode_instructions = {
        "Default": """You are an expert educational content rewriter.
Rewrite the text to be clearer and easier to understand.
Improve readability while preserving the original meaning exactly.
Output ONLY the rewrite, nothing else.""",

        "Simpler": """You are an expert educational content rewriter.
Rewrite the text so a 16-year-old with no technical background can understand it.
Replace ALL technical terms with everyday words.
Use short sentences. Maximum 15 words per sentence.
Never use acronyms or jargon.
If a technical term is unavoidable, explain it in brackets.
Output ONLY the rewrite, nothing else.""",

        "Add Example": """You are an expert educational content rewriter.
Rewrite the text and add ONE concrete example that directly illustrates the concept.
The example MUST be about the exact same topic as the input text.
Do not use unrelated examples from different domains.
Format: original explanation + "For example, ..."
Output ONLY the rewrite with example, nothing else.""",

        "Concise": """You are an expert educational content rewriter.
Rewrite the text using fewer words while keeping the same meaning.
Remove redundancy and filler phrases.
Do not remove important information.
Output ONLY the shorter rewrite, nothing else.""",

        "Step by Step": """You are an expert educational content rewriter.
Break down HOW THIS CONCEPT WORKS into clear numbered steps.
Each step must explain part of the mechanism described in the original text.
Do NOT explain history. Do NOT add information not in the original text.
Format strictly as:
1. [first part of the mechanism]
2. [second part]
3. [third part]
Output ONLY the numbered steps, nothing else.""",

        "Add Analogy": """You are an expert educational content rewriter.
Rewrite the text and add a real-world analogy that makes the concept concrete.
The analogy must map clearly to the concept in the original text.
Explain why the analogy applies.
Output ONLY the rewrite with analogy, nothing else.""",
    }
    
    system = mode_instructions.get(mode, mode_instructions["Default"])
    prompt = f"<|system|>\n{system}\n<|user|>\nText: {text}\n<|assistant|>\n"
    
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.7,
            top_p=0.95,
            top_k=50,
            do_sample=True,
            repetition_penalty=1.3,
            pad_token_id=tokenizer.eos_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )
    
    new_tokens = outputs[0][inputs['input_ids'].shape[1]:]
    return tokenizer.decode(new_tokens, skip_special_tokens=True).strip()


# Example usage
text = """Backpropagation computes the gradient of the loss function 
with respect to the weights of the network by applying the chain rule 
of calculus iteratively through each layer."""

for mode in ["Default", "Simpler", "Concise", "Step by Step", "Add Analogy"]:
    rewrite = generate_rewrite(model, tokenizer, text, mode)
    print(f"\n[{mode}]\n{rewrite}")
```

---

## Prompt Format

The model uses the LLaMA chat template format:

```
<|system|>
{mode-specific system prompt}
<|user|>
Text: {input text}
<|assistant|>
{rewrite output}
```

Each mode has a dedicated system prompt that specifies the rewriting objective, constraints, and output format. Using the correct system prompt is essential for good results.

---

## Future Work

The following improvements are planned for subsequent iterations:

1. **Regenerate Simpler and Add Example training pairs** with stricter domain-specificity and jargon constraints, then retrain
2. **Upgrade base model to 7B** for improved instruction following across all modes
3. **Expand training dataset** to 2,000+ examples for better generalisation
4. **Add audience suitability scoring** — a formula-based 1-10 score indicating the accessibility level of any rewrite
5. **Mode-specific LoRA adapters** — separate adapter per mode for higher per-mode quality at the cost of additional training time

---

## Citation

If you use this model in your research or projects, please cite:

```bibtex
@misc{ray2026rewriter,
  author    = {Anisha Ray},
  title     = {Educational Content Rewriter — LLaMA 3.2 3B LoRA},
  year      = {2026},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/ray-2908/educational-rewriter-lora}
}
```

---

## Related Resources

- **Training repository:** [nlp-llm-journey](https://github.com/ray-2908/nlp-llm-journey)
- **Base model:** [meta-llama/Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B)
- **Alpaca methodology:** [Stanford Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html)
- **LoRA paper:** [Hu et al., 2021](https://arxiv.org/abs/2106.09685)
- **QLoRA paper:** [Dettmers et al., 2023](https://arxiv.org/abs/2305.14314)

---

## Author

**Anisha Ray**  
NLP/LLM Learning Journey — Phase 5 Capstone  
Contact: anisharay08@gmail.com