---
license: other
license_name: health-ai-developer-foundations
license_link: https://developers.google.com/health-ai-developer-foundations/terms
library_name: transformers
tags:
  - medical
  - chest-x-ray
  - vision-language-model
  - interpretability
  - mechanistic-interpretability
  - research
base_model:
  - google/medsiglip-448
  - google/medgemma-4b-it
pipeline_tag: visual-question-answering
---

# FlipLens

A small, **MedGemma-faithful** chest-X-ray model: a frozen `google/medsiglip-448`
encoder run at 896 pixels feeding a compact **Gemma-3** decoder, with the yes/no
answer read from the tied language-model head. Built to study *where paraphrase
sensitivity in medical vision-language models originates*.

- Code and the full experiment write-up: **[github.com/thedatasense/FlipLens](https://github.com/thedatasense/FlipLens)**
- Interactive write-up: **[bineshkumar.me/phd-thesis/causality](https://bineshkumar.me/phd-thesis/causality/)**
- Live single-pass flip-detector demo: **[bineshkumar.me/phd-thesis/explorer/#margin-demo](https://bineshkumar.me/phd-thesis/explorer/#margin-demo)**

> **Scope.** A research probe, **not a clinical model**. It answers binary presence
> questions about 14 chest findings. Not a medical device; not for clinical use.

## What it does

Trained on NIH + PadChest, evaluated on **MIMIC and VinDr held out entirely** (no
image, and no hospital, seen in training). Every split is balanced per finding, so a
text-only model scores exactly 0.500 and all accuracy above that is visual.

| split | n | accuracy | AUC | text-only floor | flip rate |
|---|---|---|---|---|---|
| in-distribution (held-out images) | 15,112 | 0.748 | 0.827 | 0.500 | 0.061 |
| **MIMIC (unseen hospital)** | 450 | 0.671 | **0.743** | 0.500 | 0.049 |
| **VinDr (unseen hospital)** | 5,478 | 0.686 | **0.756** | 0.500 | 0.060 |

### What it establishes about paraphrase sensitivity

Training the same architecture under three phrasing distributions, changing nothing
else (8 seeds each; text-only accuracy within one point of chance in all three,
49.4-50.7% across seeds, so none can exploit an answer prior):

| training regime | flip rate | scored on **unseen** phrasings | accuracy |
|---|---|---|---|
| every paraphrase | **4.8%** | **26.6%** | 75.3% |
| one fixed phrasing | 67.1% | 65.9% | 66.8% |
| register tied to the answer | 88.4% | 87.4% | 58.1% |

Broad coverage separates from both narrow regimes at Cliff's delta = 1.00
(p = 1.6e-4), and the two narrow regimes separate from each other (p = 3.1e-4,
delta = 0.97), so a phrasing shortcut is a distinct and larger harm than narrow
coverage alone. The second column trains on half the paraphrase bank and scores only
on the 24 phrasings withheld, separating invariance from familiarity with wording
already seen: **26.6% against 65.9% is the honest size of the lever.**

A rank-1 direction at the answer position restores flipped answers (net recovery
0.98-1.00 across layers 0-4; random-direction control 0.00-0.02; non-flip disruption
0.000-0.006), so the flip is a low-rank, language-side, readout-stage effect. It is
not an artifact of where the decision boundary sits: the ordering holds at all 17
offsets across +/-2 SD of the margin, and a threshold-free dispersion statistic
reproduces it (0.023 / 0.498 / 1.362, delta = -1.00).

For reference, MedSigLIP's own zero-shot binary performance on this task is AUC 0.734,
and an earlier version of this probe scored **AUC 0.500 — indistinguishable from blind
— on unseen data**. Per finding, pneumothorax transfers essentially unchanged (0.899
in-distribution vs 0.903 on VinDr); **pleural thickening inverts on VinDr (0.332,
below chance)**, most likely a label-definition mismatch, and should not be trusted.

Accuracy alone cannot tell a seeing model from a blind one here: a blind model, a
model given a *shuffled* grounding token, and a genuinely seeing model all score
~0.50 accuracy while their AUCs are 0.500 / 0.502 / 0.604. Report AUC.

### The discarded margin is a single-pass flip detector

Because the answer is the sign of the yes-minus-no margin and a flip is a sign change,
a margin near zero marks a paraphrase-unstable answer. The absolute margin of one
forward pass therefore ranks flip-prone questions at no cost beyond the answer itself,
and it beats the detectors that cost more (`scripts/analysis/detect.py`):

| detector | passes | in-distribution | MIMIC | VinDr |
|---|---|---|---|---|
| **absolute margin** | **1** | **0.923** | **0.974** | **0.974** |
| paraphrase self-consistency | k | 0.709 | 0.786 | 0.624 |
| hidden-state probe | 1 + fit | 0.826 | 0.838 | 0.801 |

The other failure, an answer that ignores the image, has **no** single-pass signal: the
margin is at chance for it (0.470 to 0.519), because a confident blind answer looks
exactly like a confident grounded one. Only a second pass that swaps in another
patient's image detects it (0.827 to 0.907). A finer question, whether a stable answer
quietly changes how much it uses the image from one phrasing to another, is real at the
population level (cross-draw covariance 0.062, between-draw correlation 0.56, 82%
case-specific over three seeds) but **not identifiable per case** (zero grounded-to-unreliant
transitions over 345 patient-seed evaluations, patient-level upper bound near 3%). Image
reliance is a population property here, not a per-prediction label. Try it in the
**[live margin-gate demo](https://bineshkumar.me/phd-thesis/explorer/#margin-demo)**.

## Load

```python
import torch
from transformers import AutoModel
from PIL import Image

model = AutoModel.from_pretrained("saillab/FlipLens", trust_remote_code=True).eval()

input_ids, ans_pos = model.encode_question("is there pleural effusion?")
vision_features, ground = model.encode_images([Image.open("cxr.png").convert("RGB")])
logits = model(input_ids=input_ids, vision_features=vision_features,
               ground=ground, ans_pos=ans_pos).logits
print(model.config.id2label[int(logits.argmax(-1))])      # "yes" or "no"
```

`encode_images` needs gated access to `google/medsiglip-448`.

## Repository structure

```
config.json  model.safetensors  modeling_babymedgemma.py   the model + self-contained wrapper
```

Training and analysis code, and the exact scripts that produced every number here, are on
GitHub: [thedatasense/FlipLens](https://github.com/thedatasense/FlipLens).

## Citation

> Sadanandan, B. (2026). *Paraphrase Sensitivity in Medical Vision-Language
> Models: Measurement, Mechanisms, Mitigation, and Deployment Safety.* PhD
> dissertation, University of New Haven.