medqa_sample5_fairness_analysis.md

# MedQA Fairness Analysis — Sample id=5 (Cocaine-Associated Chest Pain)

**Eval log:** `2026-05-03T05-15-58-00-00_medqa_FhUYAVmHTaNbust4YfNcqG.eval`
**Task:** `inspect_evals/medqa`
**Model:** `openai/gpt-5.5`
**Sample:** `id=5`, `epoch=1`
**Result:** Incorrect (model `C`, target `E`)

---

## 1. The question (as presented to the model)

> A 39-year-old man presents to the emergency department because of progressively worsening chest pain and nausea that started at a local bar 30 minutes prior. The pain radiates to the epigastric area. He has a 5-year history of untreated hypertension. He has smoked 1 pack of cigarettes daily for the past 5 years and started abusing cocaine 2 weeks before his emergency room visit. The patient is diaphoretic and in marked distress. What should be the first step in management?
>
> A) Diltiazem
> B) Labetalol
> C) Nitroglycerin
> D) Propranolol
> E) Reassurance and continuous monitoring

**Model output:** `ANSWER: C`
**Labeled target:** `E`

---

## 2. Where the labeled answer comes from

The inspect_evals MedQA loader uses the `med_qa_en_bigbio_qa` subset of `bigbio/med_qa`:

```python
# src/inspect_evals/medqa/medqa.py:50-51
split="test",
subset="med_qa_en_bigbio_qa",
```

This is the **5-option** English variant.

The **original USMLE/4-option** version of this exact question (verifiable in `GBaker/MedQA-USMLE-4-options-hf`, `test-00005`) has only four options:

| Index | Option |
|------:|--------|
| 0 | Diltiazem |
| 1 | Labetalol |
| 2 | Propranolol |
| 3 | Reassurance and continuous monitoring |

Correct label = `3` → "Reassurance and continuous monitoring."

In the 4-option world this is defensible-by-elimination: A/B/D are all pharmacologically problematic in acute cocaine intoxication, so the only safe remaining option is observation/reassurance (with implicit benzodiazepines).

The 5-option `bigbio_qa` variant **inserts Nitroglycerin as option C** but **does not update the answer key**. The labeled answer remains "Reassurance and continuous monitoring."

---

## 3. What guidelines actually say about cocaine-associated chest pain

For a patient with cocaine use plus acute chest pain, established guidelines and reviews recommend:

- **Class I, first-line:** oxygen, **benzodiazepines**, **nitroglycerin**, aspirin.
  Nitroglycerin is specifically endorsed because it reverses cocaine-induced coronary vasoconstriction (supported by case series and RCTs).
- **Avoid:** pure non-selective beta-blockers (propranolol) — unopposed alpha-adrenergic stimulation worsens vasoconstriction.
- **Controversial / third-line:** labetalol (mixed α/β) — some evidence for safety, not first-line.
- **Second-line / situational:** calcium channel blockers (diltiazem) — used if benzodiazepines + NTG fail; avoided in LV dysfunction.
- **"Reassurance and continuous monitoring"** is reasonable for *uncomplicated, low-risk* cocaine intoxication without ongoing chest pain. The patient in this vignette is **diaphoretic and in marked distress with active chest pain** — that is not a "reassurance only" presentation.

Key sources:
- AHA Scientific Statement, *Management of Cocaine-Associated Chest Pain and Myocardial Infarction*, Circulation 2008. https://www.ahajournals.org/doi/10.1161/circulationaha.107.188950
- *How Should a Patient with Cocaine-Associated Chest Pain Be Treated?*, The Hospitalist. https://www.the-hospitalist.org/hospitalist/article/125897/
- LITFL / CCC Toxicology, *Cocaine-related Chest Pain*. https://litfl.com/cocaine-related-chest-pain/

Under these guidelines, when Nitroglycerin is on the menu, **C (Nitroglycerin) becomes the medically optimal first-step answer**.

---

## 4. Was the task solvable? Was the model "incapable"?

**Solvable in principle:** yes — a model could pick `E` if it knows the dataset convention "the 5-option MedQA inserted NTG as a distractor but kept the original 4-option key, so trust the key." That is dataset metagaming, not medical reasoning.

**Solvable on medical merit:** no — by current cardiology/emergency-medicine standards, `C` (Nitroglycerin) is at least as defensible as `E`, and arguably preferred for a patient *in active distress with chest pain*. The labeled answer requires ignoring a Class I recommendation in favor of bare observation.

**Was the model incapable?** No. The model selected the AHA Class I first-line agent for cocaine-associated chest pain. That reflects correct medical knowledge, not a knowledge gap. The "wrong" mark comes from a label-vs-guideline mismatch in the dataset, not a model deficiency.

---

## 5. Verdict

This sample is **mislabeled / unfair** in the 5-option `bigbio_qa` form. It penalizes models that apply current guideline-based reasoning and rewards models that match the legacy 4-option answer key. It is a dataset artifact, not a measurement of medical capability.

Recommended handling options:
1. Treat this sample as ambiguous and exclude it (or both `C` and `E` should be accepted) when reporting MedQA accuracy.
2. Flag upstream to `bigbio/med_qa` that 4→5 option distractor insertion changed the optimal answer for at least this question. A systematic audit of the inserted distractors is warranted — others likely have the same issue.
3. If exact-replication of the published MedQA benchmark is the goal, switch the loader to the 4-option subset (`med_qa_en_4options_bigbio_qa`) for cleaner comparison with the underlying USMLE source.