MedQA Fairness Analysis — Sample id=5 (Cocaine-Associated Chest Pain)
Eval log: 2026-05-03T05-15-58-00-00_medqa_FhUYAVmHTaNbust4YfNcqG.eval
Task: inspect_evals/medqa
Model: openai/gpt-5.5
Sample: id=5, epoch=1
Result: Incorrect (model C, target E)
1. The question (as presented to the model)
A 39-year-old man presents to the emergency department because of progressively worsening chest pain and nausea that started at a local bar 30 minutes prior. The pain radiates to the epigastric area. He has a 5-year history of untreated hypertension. He has smoked 1 pack of cigarettes daily for the past 5 years and started abusing cocaine 2 weeks before his emergency room visit. The patient is diaphoretic and in marked distress. What should be the first step in management?
A) Diltiazem B) Labetalol C) Nitroglycerin D) Propranolol E) Reassurance and continuous monitoring
Model output: ANSWER: C
Labeled target: E
2. Where the labeled answer comes from
The inspect_evals MedQA loader uses the med_qa_en_bigbio_qa subset of bigbio/med_qa:
# src/inspect_evals/medqa/medqa.py:50-51
split="test",
subset="med_qa_en_bigbio_qa",
This is the 5-option English variant.
The original USMLE/4-option version of this exact question (verifiable in GBaker/MedQA-USMLE-4-options-hf, test-00005) has only four options:
| Index | Option |
|---|---|
| 0 | Diltiazem |
| 1 | Labetalol |
| 2 | Propranolol |
| 3 | Reassurance and continuous monitoring |
Correct label = 3 → "Reassurance and continuous monitoring."
In the 4-option world this is defensible-by-elimination: A/B/D are all pharmacologically problematic in acute cocaine intoxication, so the only safe remaining option is observation/reassurance (with implicit benzodiazepines).
The 5-option bigbio_qa variant inserts Nitroglycerin as option C but does not update the answer key. The labeled answer remains "Reassurance and continuous monitoring."
3. What guidelines actually say about cocaine-associated chest pain
For a patient with cocaine use plus acute chest pain, established guidelines and reviews recommend:
- Class I, first-line: oxygen, benzodiazepines, nitroglycerin, aspirin. Nitroglycerin is specifically endorsed because it reverses cocaine-induced coronary vasoconstriction (supported by case series and RCTs).
- Avoid: pure non-selective beta-blockers (propranolol) — unopposed alpha-adrenergic stimulation worsens vasoconstriction.
- Controversial / third-line: labetalol (mixed α/β) — some evidence for safety, not first-line.
- Second-line / situational: calcium channel blockers (diltiazem) — used if benzodiazepines + NTG fail; avoided in LV dysfunction.
- "Reassurance and continuous monitoring" is reasonable for uncomplicated, low-risk cocaine intoxication without ongoing chest pain. The patient in this vignette is diaphoretic and in marked distress with active chest pain — that is not a "reassurance only" presentation.
Key sources:
- AHA Scientific Statement, Management of Cocaine-Associated Chest Pain and Myocardial Infarction, Circulation 2008. https://www.ahajournals.org/doi/10.1161/circulationaha.107.188950
- How Should a Patient with Cocaine-Associated Chest Pain Be Treated?, The Hospitalist. https://www.the-hospitalist.org/hospitalist/article/125897/
- LITFL / CCC Toxicology, Cocaine-related Chest Pain. https://litfl.com/cocaine-related-chest-pain/
Under these guidelines, when Nitroglycerin is on the menu, C (Nitroglycerin) becomes the medically optimal first-step answer.
4. Was the task solvable? Was the model "incapable"?
Solvable in principle: yes — a model could pick E if it knows the dataset convention "the 5-option MedQA inserted NTG as a distractor but kept the original 4-option key, so trust the key." That is dataset metagaming, not medical reasoning.
Solvable on medical merit: no — by current cardiology/emergency-medicine standards, C (Nitroglycerin) is at least as defensible as E, and arguably preferred for a patient in active distress with chest pain. The labeled answer requires ignoring a Class I recommendation in favor of bare observation.
Was the model incapable? No. The model selected the AHA Class I first-line agent for cocaine-associated chest pain. That reflects correct medical knowledge, not a knowledge gap. The "wrong" mark comes from a label-vs-guideline mismatch in the dataset, not a model deficiency.
5. Verdict
This sample is mislabeled / unfair in the 5-option bigbio_qa form. It penalizes models that apply current guideline-based reasoning and rewards models that match the legacy 4-option answer key. It is a dataset artifact, not a measurement of medical capability.
Recommended handling options:
- Treat this sample as ambiguous and exclude it (or both
CandEshould be accepted) when reporting MedQA accuracy. - Flag upstream to
bigbio/med_qathat 4→5 option distractor insertion changed the optimal answer for at least this question. A systematic audit of the inserted distractors is warranted — others likely have the same issue. - If exact-replication of the published MedQA benchmark is the goal, switch the loader to the 4-option subset (
med_qa_en_4options_bigbio_qa) for cleaner comparison with the underlying USMLE source.