---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- medical
- clinical
- biomedical
- radiology
- from-scratch
- base-model
- qwen3
---

<p align="center">
  <img src="catbird_llm_logo.png" alt="Catbird" width="320"/>
</p>

# KOS-V5-Base  ·  *"Catbird"*

**A 3.72B-parameter medical language model trained from scratch — not distilled, not pruned, not
continued-pretrained from a general base.** KOS-V5 (codename **Catbird**) is the fifth-generation
Kentucky Open Science base model. This repository holds the **final pretraining checkpoint**: the
complete single-epoch run, **235.2B tokens, step 49,835 of 49,835**.

This is a **base model**. It has had no instruction tuning, no RLHF, and no chat post-training. It
completes text; it does not follow instructions. It is the initialisation for a downstream SFT/RL
line, and it is released as a research artifact.

---

## At a glance

| | |
|---|---|
| **Parameters** | 3.72B (36 layers × 2560 hidden, 32/8 GQA, head_dim 128, SwiGLU, **tied** embeddings) |
| **Training tokens** | **235.2B** — one epoch over a 54-source medical/biomedical corpus, complete |
| **Final train loss** | ≈1.16 |
| **Context** | trained at **24,576** tokens (whole-document); `max_position_embeddings` **32,768** |
| **RoPE** | standard 1D, **θ = 25,000** (pin this — see *Serving notes*) |
| **Tokenizer** | custom **32k byte-level BPE** ("v5-32k"), `add_bos_token=False`, single special token `<|endoftext|>` (id 0 = eos/bos/pad) |
| **Objective** | **pure cross-entropy** — no auxiliary/geometric losses (no SigReg, no STP, no MTP) |
| **Precision / stack** | bf16, 24×H200, DeepSpeed ZeRO-1, FlashAttention-2 |
| **Architecture class** | `Qwen3ForCausalLM` (stock — exports with no custom modelling code) |

---

## Headline evaluation

Benchmarked against **16 external models** (a 17-model pool; KOS-V4 is held out as our own internal
comparison). Every comparator was trained on **1.3–153× more data** (0.3–36T tokens vs our 0.235T).
96 metrics across 19 tests; 93 ranked.

**KOS-V5 is rank-1 of 17 on 12 metrics**, best-tied on 4, top-3 on 9, and trails on 68.

**Where it wins** — and the wins are concentrated, not scattered:

| Axis | Result |
|---|---|
| **Held-out medical BPB** (the headline) | **rank 1 / 17**, 5-corpus mean **0.4635** — plus rank-1 on radiology (0.5132), chest-xray (0.6688) and clinical narratives (0.4179) |
| **Attention health** (T2) | rank-1 on 3 of 4 measures (bos-sink mass, collapsed-head fraction, min-entropy); rank 2 on the fourth |
| **Representation geometry** (T4) | rank-1 on RankMe (220.0) |
| **Long-context BPB** | rank-1 at the 1,024- and 2,048-token buckets; bucket-BPB falls monotonically out to the full 32,768 |
| **Semantic similarity** | rank-1 on BIOSSES (Pearson 0.7097 / Spearman 0.7014) |

Bits-per-byte is tokenizer-agnostic, so it is a fair cross-model number — and the distillation
confound in the pool *flatters the trillion-token externals, not a from-scratch model*. These BPB
placements are therefore conservative.

**Where it does not win — stated plainly, because earlier drafts of our own report got this wrong:**

| Axis | Result |
|---|---|
| BLURB probe mean | **rank 2 / 17** (0.7268) — *not* rank-1; only BIOSSES is |
| Calibration (mean ECE) | **rank 9 / 17** (0.1209) — mid-pool, not a strength |
| RadGraph entity/relation F1 | **rank 7–10 / 17** — high absolute scores (micro-F1 0.90–0.91), mid-pool rank |
| Long-context needle | **rank 11 / 17** (0.9333) |
| MedHALT-FCT (false-confidence) | **rank 16 / 17** (0.0280) — **our single worst placement**, and we call it out rather than bury it |
| Tokenizer single-token rate (STRR) | rank 15 / 17 — a deliberate BBPE trade-off (see *Known issues*) |
| Closed-book medical MCQ | mean 0.4352 — below the frontier fleet; a post-training problem, not a token-budget one |

---

## KOS-V5 vs KOS-V4 — a corpus trade-off, now settled

Both models are ours, both are finished, and both were run on the identical harness. **KOS-V5 is
better on 17 of 32 compared metrics, KOS-V4 on 15.**

The token-budget alibi is dead: KOS-V5 finished on **235.2B tokens vs KOS-V4's 180.3B** (1.30×) at a
larger parameter count. **Every remaining KOS-V4 win is a win on the merits.**

| | KOS-V5 (final, 235B) | KOS-V4 (final, 180B) | Winner |
|---|---|---|---|
| **Medical MCQ mean** (16 tasks) | **0.4352** | 0.3319 | **V5** (+0.103; 12 of 16 tasks) |
| MMLU clinical knowledge | **0.5057** | 0.3094 | **V5** (+0.196) |
| MMLU college medicine | **0.4335** | 0.2312 | **V5** (+0.202) |
| BPB 5-corpus mean | 0.4635 | **0.4309** | **V4** |
| BPB radiology / chest-xray / clinical | 0.5132 / 0.6688 / 0.4179 | **0.4761 / 0.5887 / 0.3221** | **V4** |
| BPB biomedical-lit / textbooks | **0.1108 / 0.6066** | 0.1243 / 0.6432 | **V5** |
| RankMe (mean) | **220.0** | 170.8 | **V5** |
| Long-context BPB @ 4k→32k | **wins every window** | — | **V5** |
| Needle 3-depth mean | **0.9333** | 0.8667 | **V5** |
| BLURB mean / RadGraph | 0.7268 / 0.75–0.77 | **0.7465 / 0.77–0.82** | **V4** |
| PubMedQA / MedThink | 0.6680 / 0.9200 | **0.6980 / 0.9450** | **V4** |

**The honest reading.** KOS-V4's narrow radiology/clinical-dominant corpus still produces the better
**text model of radiology and clinical notes**. KOS-V5's broader 54-source corpus produces the better
**medical reasoner** — on the recall-heavy MCQ tasks V4 sat close to chance and V5 does not. This is a
**corpus-composition difference, not a maturity gradient**, and no amount of further pretraining was
going to close it: V5 would have needed to recover 0.0326 BPB, and its entire final leg moved it by
0.0001. Which model is "better" depends on the deployment.

---

## The run converged before it finished — read this before choosing a checkpoint

The identical 16-tier battery was run at **six** checkpoints (90B / 120B / 150B / 190B / 220B / 235B).

| axis | 90B | 120B | 150B | **190B** | 220B | **235B (final)** |
|---|---|---|---|---|---|---|
| BPB (5-corpus, lower better) | .4890 | .4781 | .4740 | .4647 | .4636 | **.4635** |
| MCQ (16 tasks, higher better) | .3478 | .3737 | .4071 | **.4449** | .4332 | **.4352** |
| Needle (higher better) | .9333 | .9000 | .9500 | .9167 | .9000 | **.9333** |
| ECE (lower better) | .1847 | **.1135** | .1181 | .1378 | .1183 | **.1209** |

**On the final leg, not one headline axis improved.** BPB moved **+0.0001** (zero to three decimals).
MCQ **peaked at 190B and never recovered** — the final model ends **0.0097 below its own peak**, with
12 of 16 tasks below their peak-checkpoint value. Needle ended **exactly** where it began at 90B.
Calibration ended *worse* than the checkpoint before it. The learning-rate integral is 100% spent.

> **⚠️ The newest checkpoint is NOT the knowledge-best checkpoint.**
> The **190B** checkpoint is KOS-V5's knowledge-best, permanently. **Choose your fine-tuning
> initialisation by axis:** the 190B checkpoint for medical knowledge/MCQ; **this final checkpoint for
> BPB, long-context and general LM health.** Do not assume "newest = best."

Both MCQ moves (the 220B drop and the 235B bounce) sit inside MMLU-subset sampling noise. What
survives noise is the **plateau**: knowledge accuracy stopped improving 45B tokens before the end.

---

## Known issues

**1. The tokenizer does not split numbers — and this cannot be fixed in V5.**

The v5-32k BBPE has **no digit rule**: its pre-tokenizer is plain byte-level, so digit runs merge into
single tokens by BPE frequency.

| tokenizer | digit rule | multi-digit tokens | max digit-run | `10` / `100` / `1000` | tokens/number |
|---|---|---|---|---|---|
| **KOS-V5** | **none** | **963** | **5** | one token each | **1.80** |
| Qwen3-4B-Base | yes — **every digit split** | 0 | 1 | `1`·`0` / `1`·`0`·`0` / … | 4.20 |
| Llama-3.2-3B | yes — ≤3-digit groups | 1100 | 3 | `10` / `100` / `100`+`0` | 2.90 |

**The consequence is compositional:** `10`, `100` and `1000` are three *unrelated atomic symbols* that
share no substructure. The model cannot see that they differ by a factor of ten; it must learn each
magnitude as a separate lexical item, and rare numbers fragment on arbitrary BPE boundaries while
common ones do not. In medicine — doses, lab values, vitals — this is exactly the regime where
digit-level arithmetic and magnitude comparison are known to degrade.

**Stated with its limit:** this is a **measured property of the frozen tokenizer, not a measured
downstream failure.** Our evaluation suite contains **no arithmetic or numeric-reasoning tier**, so we
neither quantify the cost nor claim there is none. The tokenizer is baked into the trained (tied)
embedding table, so it is unfixable in V5 — **it is a KOS-V6 decision** (add a digit-splitting
pre-tokenizer), and V6 must ship the missing tier to measure it.

**2. Low medical single-token rate (STRR, rank 15/17).** Deliberate: BBPE spends its 32k vocabulary on
byte-level compression rather than memorising whole medical terms. It single-tokens only ~19% of
radiology and ~23% of chest-xray terms. The trade appears sound — tokens-per-byte is rank 4/17 and BPB
is rank 1/17 — but it is a trade, and it is the same trade as the digit issue above.

**3. Long-context needle retrieval never improved** across the entire run (rank 11/17). More
pretraining will not fix it; it is a post-training problem.

**4. MedHALT-FCT rank 16/17.** A base model with no refusal prior has essentially no mechanism for
declining a question with a false premise, and KOS-V5 answers anyway. This is the axis a grounded
medical executor most needs to improve, and it is a target for the instruction/RL stage.

**5. Attention health is leading but degrading.** KOS-V5 still leads the pool on T2, but bos-sink mass
rose 0.1619 → 0.2541 and layers-carrying-a-sink rose 0.0830 → 0.3330 across training. The lead is real
but it shrank; it did not consolidate.

---

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Kentucky-Open-Science/KOS-V5-Base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

prompt = "IMPRESSION: The chest radiograph demonstrates"
ids = tok(prompt, return_tensors="pt").to(model.device)     # add_bos_token=False by design
out = model.generate(**ids, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
```

### Serving notes (please read — these bite)

- **`rope_theta = 25000`, not 10000.** Some conversion paths (notably GGUF) silently fall back to
  10,000 and quietly damage long-context behaviour. **Pin it explicitly** when converting.
- **`add_bos_token=False`.** Setting it true double-prepends BOS and breaks generation.
- **One special token.** `<|endoftext|>` (id 0) serves eos/bos/pad and the document separator.
- **Base model, no chat template.** It completes text. Prompt it as a completion model, or post-train it.
- Requires `transformers` with the nested `rope_parameters` config schema (≥5.x), or convert the config.

---

## Intended use

A **research base model** for medical/biomedical NLP: the initialisation for domain SFT/RL, a subject
for interpretability and tokenizer/corpus research, and a from-scratch reference point against
distilled and continued-pretrained medical models. It is not instruction-tuned and is not a
question-answering system. Its closed-book medical MCQ accuracy is well below the frontier fleet, and
the known issues above (especially the number tokenization) bear directly on any numeric medical task.

---

## Reproducibility

| artifact | value |
|---|---|
| checkpoint | step **49,835** of 49,835 (epoch 1.000) |
| tokens/step | 1 × 8 × 24 × 24,576 = 4,718,592 |
| `tokenizer.json` md5 | `9c9df0404f6aae96dba5f3785e8b4c9d` |
| optimizer | AdamW β=(0.9, 0.95), grad-clip 1.0 |
| LR schedule | cosine → 0, peak 3.0e-4, 1% warmup (integral 100% spent) |
| batch | micro_batch 1 × grad_accum 8 × 24 GPUs, seq 24,576 (≈4.72M tokens/step) |
| corpus | 54 sources, English-only, deduped → decontaminated, whole documents (>24,576 tokens dropped, never split), neat-packed with 4-D block-diagonal segment mask (no cross-document attention; position ids reset per document) |

Full 22-page evaluation report — all 96 metrics, all 17 models, every rank re-derived from source:
`kos_v5_brief.pdf` (in this repository).

---

## Attribution

Developed at the **University of Kentucky College of Medicine**, **Center for Clinical and
Translational Science (CCTS)**.

*Additional collaborator attribution to be added.*

## Citation

```bibtex
@misc{kos_v5_base_2026,
  title  = {KOS-V5-Base (Catbird): a from-scratch 3.72B medical language model},
  author = {Kentucky Open Science},
  year   = {2026},
  note   = {University of Kentucky College of Medicine,
            Center for Clinical and Translational Science (CCTS)},
  url    = {https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base}
}
```