| --- |
| language: |
| - en |
| license: apache-2.0 |
| tags: |
| - biomedical |
| - clinical |
| - ul2 |
| - t5 |
| - encoder-decoder |
| - pretraining |
| - text2text-generation |
| - medical |
| --- |
| |
| # PubMedUL2 & MedUL2 |
|
|
| ## Model Description |
|
|
| **PubMedUL2** and **MedUL2** are a family of **domain-specific UL2/T5-style encoder–decoder language models** pretrained on large-scale biomedical and medical corpora using the **UL2 (Mixture-of-Denoisers)** objective. |
|
|
| - **PubMedUL2** models are pretrained on **25 million PubMed abstracts** |
| - **MedUL2** models are pretrained on **PubMed abstracts + clinical notes + additional medical documents** |
| - All models use a **T5-efficient architecture**, inspired by Google’s efficient T5 variants |
|
|
| These checkpoints are **pretraining-only models** and **must be fine-tuned** before use on downstream tasks. |
|
|
| --- |
|
|
| ## Pretraining Objective: UL2 (Mixture-of-Denoisers) |
|
|
| These models were pretrained using **UL2**, a unified framework that formulates language modeling objectives as **denoising tasks**. |
|
|
| UL2 introduces a **Mixture-of-Denoisers (MoD)** approach that samples from multiple denoising paradigms during pretraining. |
|
|
| ### Denoising Tasks |
|
|
| UL2 pretraining uses a mixture of three denoising tasks: |
|
|
| 1. **R-denoising (Regular Span Corruption)** |
| - Equivalent to standard T5 span corruption |
| - Optimized for language understanding tasks |
|
|
| 2. **X-denoising (Extreme Span Corruption)** |
| - Uses very large masked spans |
| - Encourages long-form generation and abstraction |
|
|
| 3. **S-denoising (Sequential / PrefixLM)** |
| - Prefix language modeling similar to causal LM |
| - Suitable for sequence-to-sequence and generative tasks |
|
|
| ### Paradigm Tokens (Mode Switching) |
|
|
| During pretraining, a **paradigm token** is inserted at the beginning of each input: |
|
|
| | Token | Mode | Recommended Use | |
| |------|------|------------------| |
| | `[NLU]` | R-denoising | Classification, QA, retrieval | |
| | `[NLG]` | X-denoising | Mixed understanding & generation | |
| | `[S2S]` | S-denoising | Generative / causal tasks | |
|
|
| **Important:** |
| For best performance, the same token should be **prepended during fine-tuning and inference**. |
|
|
| --- |
|
|
| ## Architecture |
|
|
| - Encoder–decoder Transformer (T5-style) |
| - Uses **T5-efficient architecture** |
| - Compatible with Hugging Face `T5ForConditionalGeneration` |
|
|
| --- |
|
|
| ## Intended Uses |
|
|
| These models are intended to be **fine-tuned** for: |
|
|
| - Biomedical and clinical **text classification** |
| - **Question answering** |
| - **Summarization** of medical literature or clinical notes |
| - **Text generation** in medical contexts |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - ❌ Not instruction-tuned |
| - ❌ No supervised training |
| - ❌ Not suitable for zero-shot use |
|
|
| These checkpoints are **self-supervised pretraining models only** and require task-specific fine-tuning. |
|
|
| --- |
|
|
| ## Fine-Tuning Recommendations |
|
|
| - **Avoid mixed precision** (fp16 / bf16) initially |
| - Fine-tuning is more stable in **fp32** |
| - Always prepend one of `[NLU]`, `[NLG]`, or `[S2S]` to input text |
| - Suggested defaults: |
| - Classification / QA → `[NLU]` |
| - Causal or generative tasks → `[S2S]` |
| - Mixed tasks → `[NLG]` |
|
|
| --- |
|
|
| ## Model Parameter Summary |
|
|
| | Model Name | Parameter Count | Description | Access |
| |-----------|----------------|------------|------------| |
| | `pubmedul2-tiny-nl6` | **19.26M** | Tiny UL2-style model with 6 layers | Open |
| | `pubmedul2-mini-nl8` | **50.12M** | Mini UL2 with 8 layers | Open |
| | `pubmedul2-small` | **60.52M** | Small UL2 variant | Open |
| | `pubmedul2-small-nl24` | **192.73M** | Small UL2 with 24 layers | Open |
| | `medul2-base` | **222.93M** | Base UL2/T5-style model | Open |
| | `pubmedul2-base` | **222.93M** | Base UL2/T5-style model | Open |
| | `medul2-base-nl36` | **619.44M** | Base UL2 with 36 layers | Gated commercial |
| | `pubmedul2-base-nl36` | **619.44M** | Base UL2 with 36 layers | Gated commercial |
| | `medul2-large` | **737.72M** | Large UL2/T5-style model | Gated non-commercial |
| | `pubmedul2-large` | **737.72M** | Large UL2/T5-style model | Gated non-commercial |
| | `medul2-large-nl36` | **1090.14M** | Very large UL2 with 36 layers | Access on Request |
|
|
| --- |
|
|
| ## Named Entity Recognition (NER) Evaluation |
|
|
| We evaluate PubMedUL2 and MedUL2 models on a biomedical **Named Entity Recognition (NER)** task using multiple matching criteria to better capture boundary-level performance. |
|
|
| The evaluation reports **entity-level F1 scores** across different biomedical entity types and model sizes. |
|
|
| ### Exact Match F1 |
|
|
| An entity prediction is considered correct only if both the **entity span and label exactly match** the gold annotation. |
|
|
| | entity_type | medul2-base | pubmedul2-base | pubmedul2-mini-nl8 | pubmedul2-small | pubmedul2-tiny-nl6 | |
| |:--------------|--------------:|-----------------:|---------------------:|------------------:|---------------------:| |
| | cell_line | 0.42 | 0.43 | 0.44 | 0.43 | 0.35 | |
| | cell_type | 0.59 | 0.58 | 0.59 | 0.58 | 0.52 | |
| | chemical | 0.76 | 0.75 | 0.72 | 0.72 | 0.56 | |
| | disease | 0.7 | 0.73 | 0.7 | 0.68 | 0.63 | |
| | dna | 0.59 | 0.55 | 0.54 | 0.55 | 0.45 | |
| | gene | 0.62 | 0.59 | 0.6 | 0.59 | 0.55 | |
| | protein | 0.59 | 0.58 | 0.58 | 0.59 | 0.55 | |
| | rna | 0.6 | 0.56 | 0.55 | 0.6 | 0.56 | |
| | species | 0.66 | 0.67 | 0.58 | 0.63 | 0.54 | |
| |
| --- |
| |
| ### Partial Match F1 |
| |
| A prediction is counted as correct if it **partially overlaps** with a gold entity of the same type. |
| |
| | entity_type | medul2-base | pubmedul2-base | pubmedul2-mini-nl8 | pubmedul2-small | pubmedul2-tiny-nl6 | |
| |:--------------|--------------:|-----------------:|---------------------:|------------------:|---------------------:| |
| | cell_line | 0.48 | 0.49 | 0.48 | 0.48 | 0.41 | |
| | cell_type | 0.66 | 0.64 | 0.66 | 0.65 | 0.59 | |
| | chemical | 0.79 | 0.78 | 0.76 | 0.75 | 0.6 | |
| | disease | 0.82 | 0.84 | 0.8 | 0.79 | 0.74 | |
| | dna | 0.65 | 0.61 | 0.6 | 0.61 | 0.53 | |
| | gene | 0.76 | 0.74 | 0.74 | 0.73 | 0.68 | |
| | protein | 0.66 | 0.66 | 0.66 | 0.67 | 0.64 | |
| | rna | 0.68 | 0.63 | 0.64 | 0.66 | 0.65 | |
| | species | 0.68 | 0.7 | 0.61 | 0.65 | 0.56 | |
|
|
| --- |
|
|
| ### IoU Match F1 |
|
|
| Predictions are evaluated using **Intersection-over-Union (IoU)** overlap between predicted and gold spans, providing a softer boundary-based metric. |
|
|
| | entity_type | medul2-base | pubmedul2-base | pubmedul2-mini-nl8 | pubmedul2-small | pubmedul2-tiny-nl6 | |
| |:--------------|--------------:|-----------------:|---------------------:|------------------:|---------------------:| |
| | cell_line | 0.5 | 0.5 | 0.5 | 0.5 | 0.42 | |
| | cell_type | 0.67 | 0.66 | 0.68 | 0.67 | 0.62 | |
| | chemical | 0.83 | 0.83 | 0.82 | 0.82 | 0.72 | |
| | disease | 0.85 | 0.86 | 0.86 | 0.85 | 0.82 | |
| | dna | 0.65 | 0.62 | 0.62 | 0.62 | 0.55 | |
| | gene | 0.76 | 0.75 | 0.75 | 0.74 | 0.71 | |
| | protein | 0.67 | 0.66 | 0.67 | 0.67 | 0.66 | |
| | rna | 0.68 | 0.65 | 0.66 | 0.67 | 0.67 | |
| | species | 0.72 | 0.74 | 0.65 | 0.69 | 0.58 | |
| |
| --- |
| |
| ### Observations |
| |
| - **MedUL2 models** generally outperform PubMedUL2 on clinical-heavy entity types such as *disease* and *chemical* |
| - Performance improves consistently from **tiny → base models** |
| - Boundary-sensitive metrics (Partial / IoU) show significantly higher scores than Exact Match, highlighting boundary ambiguity in biomedical NER |
| |
| --- |
| |
| ## Acknowledgements |
| |
| This project would not have been possible without compute generously provided by **Google TPU Research Cloud**. |
| |
| Thanks to: |
| - The **Finnish-NLP** authors for releasing the UL2 objective code, task definitions, and guidance |
| - **Yeb Havinga** for help getting started with the **t5x** framework |
| |
| --- |
| |
| ## License |
| |
| Please refer to the individual model repositories for **license and access details**, which may vary depending on training data sources. |
| |
