hxrikp/medgemma-1.5-4b-medical-reasoning-qlora

MedGemma-1.5-4B Medical Reasoning (QLoRA adapter)

Title: MedGemma-4B Medical Reasoning (QLoRA) Subtitle: A QLoRA adapter fine-tuning MedGemma-1.5-4B-it on medical chain-of-thought reasoning, trained on a single free Kaggle T4 - the cross-model replication checkpoint from a small-LLM calibration study.

Reusable as both the Kaggle model card and the Medium/blog blurb. ⚠️ Research & education only - NOT for clinical or patient-facing use.

Model Summary

A LoRA adapter that fine-tunes google/medgemma-1.5-4b-it on medical chain-of-thought reasoning. It is the supervised-fine-tuned ("after") checkpoint that replicates, on a medical-specialist base, the central finding of the companion Qwen3-4B study: this fine-tune makes the model's internal (token-probability) confidence markedly better calibrated and dramatically less over-cautious. Trained entirely on a single free Kaggle T4.

Model Characteristics

• Base: MedGemma-1.5-4B-it (HAI-DEF license, gated), 4B params, multimodal (image+text); used here as a text causal-LM.
• Method: QLoRA - 4-bit NF4 (double-quant), LoRA r=16, α=32, dropout 0.05 on q/v/o/gate/up/down projections. Trainable params well under 1% of the model.
• Training: ~250 steps (past the loss plateau), seq len 1024, effective batch 16, lr 2e-4, fp32 compute (T4 doesn't support bf16; fp16 is numerically unstable for Gemma). train_loss 1.72 to ~1.56; ~4.6 h on one T4 (fp32 is ~2.7× slower per step than fp16).
• Format: PEFT LoRA adapter (adapter_config.json + adapter_model.safetensors). Load on top of the gated base (you must accept MedGemma's terms and supply an HF token).

Data Overview

• FreedomIntelligence/medical-o1-reasoning-SFT (Apache-2.0): chain-of-thought medical Q&A, GPT-4o-generated reasoning validated by a medical verifier. No PHI.

Evaluation Results

Before (base) vs. after (this adapter) on MedQA + MedMCQA (0-shot, n≈800), internal confidence (max-softmax over option logits), with 95% paired bootstrap CIs. Bold = significant (CI excludes 0).

metric	base	+this adapter	Δ [95% CI]
Accuracy ↑	0.556	0.505	-0.051 [-0.105, +0.006] (n.s.)
ECE ↓ (calibration error)	0.304	0.126	-0.178 [-0.235, -0.122]
AURC ↓ (selective prediction)	0.352	0.345	-0.007 [-0.078, +0.063] (n.s.)
Abstention rate	0.696	0.096	-0.600 [-0.634, -0.566]

Takeaway: the base model abstained on ~70% of questions (uselessly cautious - it picks the "defer to a clinician" option by default); the fine-tune cuts that to ~10% while halving calibration error. Accuracy change is within noise. This replicates the internal-calibration gain seen on Qwen3-4B (ECE 0.351 to 0.085). MedGemma's verbalized-confidence pass is omitted (its multimodal head made generation-time parsing unreliable), so the internal-vs-verbalized divergence is demonstrated on Qwen. Full method + CIs: repo healthcare/exp-004-confidence-tax/. code at https://github.com/Mr-Neutr0n/confidence-tax-study

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "google/medgemma-1.5-4b-it"  # gated: accept terms + pass token=...
tok = AutoTokenizer.from_pretrained(base, token=HF_TOKEN)
model = AutoModelForCausalLM.from_pretrained(
  base, torch_dtype=torch.float32, attn_implementation="eager", # fp32 + eager on T4
  device_map="auto", token=HF_TOKEN)
model = PeftModel.from_pretrained(model, "<this-adapter-path>")  # load the LoRA adapter

License & citation

• Adapter license: Apache-2.0 (training data is Apache-2.0). Base model: MedGemma is governed by the Health AI Developer Foundations (HAI-DEF) terms - you must accept them to use the base weights.
• Code and analysis: https://github.com/Mr-Neutr0n/confidence-tax-study