---
license: other
license_name: ohsumed-derived
license_link: https://huggingface.co/datasets/AxelNoun/ohsumed-23-single-label
language:
  - en
library_name: transformers
pipeline_tag: text-classification
base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext
datasets:
  - AxelNoun/ohsumed-23-single-label
metrics:
  - accuracy
  - f1
tags:
  - medical
  - biomedical
  - ohsumed
  - mesh
  - medline
model-index:
  - name: biomedbert-ohsumed23
    results:
      - task:
          type: text-classification
          name: Multi-class text classification
        dataset:
          name: OHSUMED-23 (single-label)
          type: AxelNoun/ohsumed-23-single-label
          split: test
          revision: c2e11d41da4d537366e75e02bc3b21a29a4978e0
        metrics:
          - type: accuracy
            value: 0.774
            name: Accuracy
          - type: f1
            value: 0.699
            name: Macro-F1
          - type: f1
            value: 0.766
            name: Weighted-F1
---

# BiomedBERT — OHSUMED-23

23-way classifier for MEDLINE abstracts (MeSH disease chapters **C01–C23**), fine-tuned from [`microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext`](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext) on [`AxelNoun/ohsumed-23-single-label`](https://huggingface.co/datasets/AxelNoun/ohsumed-23-single-label) (revision `c2e11d41da4d537366e75e02bc3b21a29a4978e0`).

This is **not** a set of “23 cardiovascular categories”. GNN papers often phrase it that way; that is incorrect. The 23 labels are MeSH chapters **C01–C23**. C14 (*Cardiovascular Diseases*) is one of those 23 classes. Spot-check: duodenitis → C06 (digestive).

## Results (test, 4,043 documents)

| Model | Size | Acc | Macro-F1 | Macro-F1 excl. C22 | Weighted-F1 | Top-3 |
|---|---|---|---|---|---|---|
| DistilBERT | 66M | 0.666 | 0.497 | 0.520 | 0.643 | 0.854 |
| bert-base-uncased | 110M | 0.686 | 0.550 | 0.575 | 0.668 | 0.865 |
| BiomedBERT | 110M | **0.774** | **0.699** | **0.731** | 0.766 | **0.933** |
| BiomedBERT (TextGCN-preprocessed input) | 110M | 0.741 | 0.662 | 0.692 | 0.734 | 0.919 |

Decomposition: DistilBERT → bert-base = +2.0 acc / +5.3 macro-F1 (capacity); bert-base → BiomedBERT = +8.8 / +14.9 (domain).

`Macro-F1 excl. C22` is the unweighted mean of the other 22 class F1s (0.699 × 23/22).

### Uncertainty

Two distinct dispersions are reported; they answer different questions:

- **Initialization variability** — seeds 42 / 43 / 44, same split:
  accuracy 0.775 ± 0.003 · macro-F1 0.699 ± 0.005 · top-3 0.934 ± 0.002.
  This speaks to training stability.
- **Sampling variability** — 95% bootstrap on the 4,043 test documents,
  1,000 resamples: macro-F1 in [0.680, 0.715].
  **This** is the one to use when comparing to a figure published elsewhere.

Released weights are seed 42 (accuracy 0.774, macro-F1 0.699). The table above
reports the three-seed mean; a gap on the order of 0.001 between the two is expected.

### Positioning on the TextGCN protocol

| Method | Input | Accuracy | Macro-F1 |
|---|---|---|---|
| TF-IDF + logistic regression | filtered | 54.66 | — |
| CNN-non-static | filtered | 58.44 | — |
| RB-GAT (RoBERTa + BiGRU + GAT) | filtered | 71.48 | 67.90 |
| **BiomedBERT — paired comparison** | filtered | **74.1** | **66.2** |
| **BiomedBERT — released model** | raw | **77.4** | **69.9** |

**Reading.** Only the “paired” row is directly comparable to the cited work: it
uses the same preprocessing. On that footing, BiomedBERT is slightly ahead in
accuracy (+2.6) and slightly behind in macro-F1 (−1.7): the two approaches are
**at the same level**. We claim no superiority.

The released model consumes raw text and posts higher absolute scores, but that
number is not comparable to anything in the literature — it only shows what you
get without degrading the input.

### What TextGCN preprocessing costs a domain encoder

Absent from the cited literature: same BiomedBERT, same seed 42, same
3,357 / 4,043 split; only the input changes (`preprocess_textgcn.py`,
NLTK stopwords + tokens with frequency < 5, vocabulary computed **on train
alone**). From 0.774 / 0.699 (raw) to **0.741 / 0.662** (filtered):
**−3.3** accuracy points, **−3.7** macro-F1.

Mechanism. The freq < 5 filter on ~3,357 training documents shrinks the kept
vocabulary to **8,290** tokens (after stopword drop). Those are precisely the
rare technical terms — where a WordPiece trained on PubMed beats a generalist
vocabulary. The TextGCN protocol therefore systematically handicaps subword
transformers relative to graph methods, which rely on frequent-word co-occurrences
and lose less from that filtering. Confirming signature: macro-F1 drops more than
accuracy (−3.7 vs −3.3), consistent with rare terms carrying mostly rare classes.

### Scope of the comparison

Three models were compared, chosen to isolate domain effect at fixed
architecture. **This protocol shows that pretraining domain drives most of the
gain — not that BiomedBERT is the best possible choice.** Not evaluated:
BioLinkBERT, recent long-context encoders, and a zero-shot LLM baseline that
would set the floor without training. Contributions welcome.

### Dominant confusions

Row-normalized matrix: `evidence/confusion_matrix.png` (also under
`outputs/biomedbert-ohsumed23/` locally).

Top pairs: **C23 → C14** (50) · **C23 → C10** (42) · **C23 → C21** /
**C02 → C20** (24 each).

**C23 asymmetry.** Strongly outbound flow: **264** C23→X vs **107** X→C23
(ratio 2.47). Precision **0.592**, recall **0.370**, F1 **0.455** (n=419).
The asymmetry makes the mechanism visible: the model is not symmetrically
confusing C23 with something else — it **under-emits** C23. When it predicts C23
it is right six times out of ten; it only catches one third of true C23.
Signature of a catch-all class learned to fire only when no organ signal is
present. Second-largest macro-F1 hole after C22.

Headline figure: macro-F1 **0.699** (23 classes). Diagnostic on the **21
well-formed classes** — structural criterion fixed *a priori*, not “the two
worst scores”: C22 has no usable support, C23 is not a domain but a residual
chapter — : **0.744**. A weak class for any other reason would stay in the average.

Dominant confusions are not domain-to-domain ambiguity but the **residual**
nature of chapter C23, which absorbs signs and symptoms independently of the
organ system. A C23-labeled abstract talks about a sign; its text is saturated
with organ vocabulary; the model predicts the organ. That is a **labeling
granularity** problem, not MeSH polyhierarchy.

Only one top pair matches a documented MeSH cross-listing: **C02→C20**.
**23/24** cases mention HIV / AIDS / immunodeficiency / HTLV (1987–1991 AIDS
literature). C04↔C08 and C10↔C20 remain rare in the matrix (4 and 1).

## Limitations

### C22 is never predicted

`C22 - Animal Diseases` scores F1 0.000 on all three seeds. With roughly a dozen
test examples and similar training support, the model never learned to emit it:
**no input will receive this label**. Any application that needs C22 detection
must use another mechanism.

Metric consequence: a dead class costs 1/23 mechanically, i.e. 4.3 macro-F1
points. Hence dual reporting — 0.699 on 23 classes, 0.731 on the 22 classes with
usable support.

### Scores are not calibrated

Weighted cross-entropy (α = 0.5) used against class imbalance shifts calibration.
Displayed scores are relative confidence scores, not probabilities. Temperature
scaling on validation would be needed before any threshold-based use.

### Intrinsic label ambiguity

Two distinct mechanisms coexist; they are not fixed the same way.

**1. Granularity (C23).** An abstract about a **general clinical sign**
(hemorrhage, shock, pain, syncope…) will be routed to an organ chapter.
Predictable behavior: C23 is a MeSH catch-all, not a domain; the model
under-emits it (precision 0.592, recall 0.370; 264 outbound vs 107 inbound
leaks). Demo users should know this.

**2. Polyhierarchy.** MeSH is not a tree. Illustration (`app.py` examples):

- *Lung Neoplasms*: C04 and C08. The model predicts **C08** (0.90) — legitimate;
  the corpus mostly labels “lung carcinoma / cancer” as C08.
- *Multiple Sclerosis*: C10 and C20. The model predicts **C20** (0.86); the corpus
  says C20 (train 10/12, test 15/15).

The top-1 → top-3 gap (0.774 → 0.933) mostly captures these two effects — hence
Top-3 in the demo.

### Future work (formulation, not capacity)

The principled fix is not a better encoder but a **better task formulation**:
multi-label or hierarchical labeling that does not force a choice between the
sign (C23) and the organ, or between two cross-listed chapters. That is the
conclusion the matrix supports.

### Temporal drift

Abstracts from 1987–1991. Dated terminology, drugs, and protocols. No guarantee
of generalization to contemporary biomedical literature.

### Not a medical device

Bibliographic categorization only. No diagnostic or clinical decision-support use.

## License and terms of use

Two layers apply jointly:

1. **Base model** — `microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext`
   is distributed under the MIT license.
2. **Training data** — the OHSUMED corpus is redistributed under its own terms,
   stricter than MIT. Fine-tuned weights inherit those usage constraints.

How far a data license propagates into fine-tuned weights is not settled law. If
commercial use is contemplated, check the source OHSUMED collection terms before
any deployment. This card is not legal advice.

## Reproduction

```bash
python train.py --dataset_id AxelNoun/ohsumed-23-single-label --dataset_revision c2e11d41da4d537366e75e02bc3b21a29a4978e0 --train_batch_size 8 --gradient_accumulation_steps 2 --eval_batch_size 16 --max_length 448 --seed 42
python eval_checkpoint.py --model_id outputs/biomedbert-ohsumed23/final
python analyze_results.py --run_dir outputs/biomedbert-ohsumed23
```

Demo: `$env:MODEL_ID="AxelNoun/biomedbert-ohsumed23"; python app.py`

## Citation

```bibtex
@inproceedings{yao2019graph,
  title={Graph Convolutional Networks for Text Classification},
  author={Yao, Liang and Mao, Chengsheng and Luo, Yuan},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={33},
  number={01},
  pages={7370--7377},
  year={2019}
}

@article{lv2024rbgat,
  title={{RB-GAT}: A Text Classification Model Based on {RoBERTa}-{BiGRU} with Graph {ATtention} Network},
  author={Lv, Shanshan and Dong, Jinming and Wang, Chao and Wang, Xianwei and Bao, Zhiyu},
  journal={Sensors},
  volume={24},
  number={11},
  pages={3365},
  year={2024}
}
```