omaraboelmaaty/llava-med-lora-v2

LLaVA-Med LoRA v2 — Medical Radiology Fine-tune

Fine-tuned LoRA adapter for microsoft/llava-med-v1.5-mistral-7b, specialized in generating structured radiology reports from medical scans (X-ray, CT, MRI).

Model Details

Property	Value
Base Model	microsoft/llava-med-v1.5-mistral-7b
Fine-tune Method	QLoRA (LoRA rank 16)
Training Epochs	2
Learning Rate	5e-5
Best Val Loss	0.907
Languages	English / Arabic UI

Intended Use

• Generating structured radiology reports (Findings / Impression / Recommendations)
• Interactive radiologist Q&A chat about medical scans
• Risk heatmap visualization of scan regions

Usage

from peft import PeftModel
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path

base_id = "microsoft/llava-med-v1.5-mistral-7b"
lora_id = "YOUR_USERNAME/llava-med-lora-v2"

name = get_model_name_from_path(base_id)
tokenizer, base, image_processor, _ = load_pretrained_model(
    model_path=base_id, model_base=None,
    model_name=name, load_4bit=True,
    device_map="auto"
)
model = PeftModel.from_pretrained(base, lora_id, autocast_adapter_dtype=False)
model.eval()

Training Data

Fine-tuned on curated medical imaging datasets including radiology reports paired with scan images.

Limitations

• Base model is trained primarily on English medical literature
• Arabic output may fall back to English in some cases
• Not intended for clinical diagnosis — for research purposes only

Citation

@misc{llava-med-lora-v2,
  title={LLaVA-Med LoRA v2 - Radiology Fine-tune},
  year={2025},
  publisher={Hugging Face}
}

⚠️ Disclaimer

This model is intended for research purposes only and should not be used as a substitute for professional medical advice, diagnosis, or treatment.