Pika4028/biobart-v2-large-multiclinsum-lora

BioBART-v2-large + LoRA — MultiClinSum

LoRA adapter fine-tuned on the MultiClinSum GS training split for abstractive summarization of clinical notes into plain-language patient summaries. Part of UW LING 573 (Spring 2026) — MedJarGone project.

Model Details

• Base model: GanjinZero/biobart-v2-large (~406M parameters)
• Fine-tuning method: LoRA (PEFT) — only adapter weights are stored here
• Trainable parameters: 2.36M / 406M total (0.58%)
• LoRA config: r=16, α=16, dropout=0.1, targets: q_proj + v_proj
• Task: Abstractive summarization (seq2seq)
• Language: English

Training

• Dataset: MultiClinSum GS split — 594 examples (476 train / 53 val, 90/10 split, seed=42)
• Epochs: 10
• Batch size: 4 (per device)
• Learning rate: 5e-5, weight decay 0.01
• Precision: fp16
• Checkpoint selection: best BERTScore F1 on validation set
• Hardware: Hyak GPU cluster (UW), ~31 minutes on a single NVIDIA GPU
• Framework: PEFT 0.19.1, Transformers

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

base = "GanjinZero/biobart-v2-large"
adapter = "priyanshi-garg/biobart-large-lora"  # update with actual HF repo name

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForSeq2SeqLM.from_pretrained(base, torch_dtype="auto")
model = PeftModel.from_pretrained(model, adapter)
model = model.merge_and_unload()  # merge LoRA weights for inference

inputs = tokenizer("Patient clinical note here...", return_tensors="pt", truncation=True, max_length=1024)
summary_ids = model.generate(**inputs, max_new_tokens=256, num_beams=4)
print(tokenizer.decode(summary_ids[0], skip_special_tokens=True))

Evaluation

Evaluated on the full MultiClinSum test set (3,396 examples). See MODELS.md for results.

Primary metric: rougeLsum (matches MultiClinSum shared task scoring).