---
library_name: transformers
tags:
- medical
datasets:
- OpenMed/Medical-Reasoning-SFT-Mega
metrics:
- perplexity
base_model:
- EleutherAI/gpt-neo-125m
pipeline_tag: text-generation
---
🩺 GPT-Neo 125M Medical Reasoning LoRA

This model is a LoRA fine-tuned version of EleutherAI's GPT-Neo 125M for medical reasoning and clinical QA-style generation.

It was fine-tuned using parameter-efficient training (LoRA) on the OpenMed/Medical-Reasoning-SFT-Mega dataset.

🔹 Only adapter weights are trained (base model not fully fine-tuned)
🔹 Optimized for instruction-style medical reasoning
🔹 Lightweight & efficient to run

📌 Model Details

Base Model: EleutherAI/gpt-neo-125M

Architecture: Causal Language Model

Fine-Tuning Method: LoRA (PEFT)

Task Type: Medical reasoning / QA generation

Training Objective: Next-token prediction (causal LM)

🧠 Training Setup
Dataset

Name: OpenMed/Medical-Reasoning-SFT-Mega

Split: 95% train / 5% validation

Downsampled:

40,000 training samples

5,000 validation samples

Reformatted into structured chat format:
## 🧠 Training Setup

### Hyperparameters

| Parameter | Value |
|------------|--------|
| Epochs | 3 |
| Batch Size | 8 |
| Gradient Accumulation | 2 |
| Learning Rate | 2e-4 |
| Block Size | 256 |
| Weight Decay | 0.01 |
| FP16 | Enabled (if CUDA available) |

---

### LoRA Configuration

| Parameter | Value |
|------------|--------|
| Rank (r) | 8 |
| Alpha | 16 |
| Dropout | 0.05 |
| Target Modules | `q_proj`, `v_proj` |
| Bias | None |
```
Only a small percentage of total parameters were trainable (~<1%), making training efficient.

📊 Evaluation

Evaluation was performed on a held-out validation set.

Metric: Cross-entropy loss

Reported:

Eval Loss: (auto-filled during training)

Perplexity: exp(eval_loss)

Perplexity was calculated as:

ppl = exp(eval_loss)
🚀 Usage

Since this repo contains LoRA adapter weights, you must load it with the base model:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = "EleutherAI/gpt-neo-125M"
adapter = "ahmedrayan/medical_lora"

tokenizer = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(base_model)
model = PeftModel.from_pretrained(model, adapter)

prompt = "Common method by which bacteria can acquire new genetic material?"

inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=100)

print(tokenizer.decode(output[0], skip_special_tokens=True))
```
🎯 Intended Use

This model is intended for:

Medical reasoning research

Educational experimentation

Fine-tuning demonstrations

PEFT / LoRA learning projects

⚠️ Not intended for real clinical decision-making.

⚠️ Limitations

Small base model (125M parameters)

Trained on subset (40k samples)

May hallucinate medical facts

No safety alignment beyond dataset supervision

Not evaluated against clinical benchmarks

🧪 Hardware

Device: CUDA (if available)

Mixed precision (FP16)

Trainer API from 🤗 Transformers

📜 License

Please refer to:

Base model license: EleutherAI/gpt-neo-125M

Dataset license: OpenMed/Medical-Reasoning-SFT-Mega

🙌 Author

Ahmed Rayan
AI Engineer | Medical AI Enthusiast
GitHub / Hugging Face: ahmedrayan