---
language: en
base_model: microsoft/Phi-3-medium-128k-instruct
tags:
  - lora
  - qlora
  - peft
  - grounding
  - hallucination-reduction
  - causal-lm
license: mit
---

# MotherBrain Grounding LoRA — v3

A QLoRA adapter trained on `microsoft/Phi-3-medium-128k-instruct` to teach strict context grounding — the model learns to answer only from provided context and refuse to speculate when the answer is absent.

v3 expands the v2 training set with adversarial, truthfulness, and numerical-reasoning examples, and moves to a larger base model trained with 4-bit quantization (QLoRA) on an A100 40GB.

---

## Architecture

| Parameter | Value |
|---|---|
| Base model | `microsoft/Phi-3-medium-128k-instruct` |
| PEFT type | LoRA (QLoRA — 4-bit base) |
| Rank (r) | 32 |
| LoRA alpha | 16 |
| Alpha/r ratio | 0.5 |
| DoRA | No |
| rsLoRA | No |
| Dropout | 0.05 |
| Bias | none |
| Task type | CAUSAL_LM |
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
| Adapter size | ~170MB |

---

## Training

| Parameter | Value |
|---|---|
| Quantization | 4-bit (QLoRA) |
| Precision | BF16 (compute), TF32 |
| Max sequence length | 2048 (sample packing enabled) |
| Learning rate | 2e-4 |
| LR scheduler | Cosine |
| Warmup ratio | 0.03 |
| Micro batch size | 1 |
| Gradient accumulation steps | 32 |
| Effective batch size | 32 |
| Optimizer | paged_adamw_8bit |
| Epochs | 3 |
| Hardware | Lambda Labs A100 40GB (SXM4) |
| Framework | Axolotl (HuggingFace PEFT + Transformers) |
| Peak VRAM used | ~12.3 GB |
| Completed | 2026-07-07 |
| Wall-clock training time | ~27.3 hours (3 epochs, 2319 steps) |
| Final train loss | 2.982 |
| Final eval loss / ppl | 3.918 / 50.3 |

---

## Datasets

| Dataset | Samples | Purpose |
|---|---|---|
| SQuAD v2 | 130,319 | Answerable + unanswerable QA pairs |
| HaluEval QA | 20,000 | Hallucinated vs. grounded answer pairs |
| Adversarial synthetic | 10,000 | Pressure to answer outside context |
| TruthfulQA | 817 | Questions designed to trigger hallucination |
| Natural Questions (sampled) | 15,000 | Real-world search QA |
| DROP (sampled) | 10,000 | Numerical / date reasoning over context |
| **Total** | **186,136** | (before eval split of 2%) |

---

## System Prompt / Refusal Format

Training instruction used per-example:
```
Answer only using the provided context. If the answer is not in context, output exactly: NOT_FOUND.
```

**Unanswerable response (exact string trained on):**
```
NOT_FOUND
```

Note: this differs from v2's refusal string (`"The provided context does not contain this information."`). Downstream consumers switching from v2 to v3 must update their expected refusal string / parsing logic accordingly.

---

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "microsoft/Phi-3-medium-128k-instruct",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, "MotherBrainIfy/grounding-lora-v3")
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-medium-128k-instruct", trust_remote_code=True)
```

---

## Changelog from v2

| Change | v2 | v3 |
|---|---|---|
| Base model | Phi-4-mini-instruct | Phi-3-medium-128k-instruct |
| PEFT method | DoRA | LoRA (QLoRA, 4-bit base) |
| Rank / alpha | r=32, alpha=64 (ratio 2.0) | r=32, alpha=16 (ratio 0.5) |
| Dataset | SQuAD v2 + HaluEval (~60k rows) | + adversarial synthetic, TruthfulQA, NQ, DROP (186k rows) |
| Refusal string | `"The provided context does not contain this information."` | `"NOT_FOUND"` |
| Hardware | GCP L4 (16GB) | Lambda Labs A100 40GB |

---

## MotherBrain Architecture

This adapter is designed as **layer 2** of a three-layer neuron stack:

```
Base SLM (stem cell)
    + Grounding LoRA (this adapter — domain agnostic)
    + Domain-Specific LoRA (e.g. K8s, medical, legal)
```

The grounding LoRA is loaded first and provides hallucination resistance before any domain adapter is applied. Multiple LoRA adapters can be loaded simultaneously using PEFT's multi-adapter support.

---

## Limitations

- Not yet evaluated head-to-head against v2 on adversarial holdout — TODO before promoting to "recommended" adapter
- Base model change (Phi-4-mini → Phi-3-medium) means v2/v3 are not drop-in interchangeable adapters; each requires its own matching base model
- Refusal string changed from v2 — update any downstream parsing logic