KOS-V5-Base (Catbird): final pretrain ckpt 49835 - 235.2B tokens, 100% of the single-epoch run

Final checkpoint of the completed run (step 49,835/49,835, epoch 1.000, loss ~1.16).
Card carries the full external evaluation (12 rank-1 of 17), the KOS-V5-vs-KOS-V4 comparison
(17-15), the convergence result (the 190B ckpt is knowledge-best, NOT this one), and the known
issues -- including that the tokenizer does not split numbers.

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+catbird_llm_logo.png filter=lfs diff=lfs merge=lfs -text
+kos_v5_brief.pdf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,258 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- medical
+- clinical
+- biomedical
+- radiology
+- from-scratch
+- base-model
+- qwen3
+---
+
+<p align="center">
+  <img src="catbird_llm_logo.png" alt="Catbird" width="320"/>
+</p>
+
+# KOS-V5-Base  ·  *"Catbird"*
+
+**A 3.72B-parameter medical language model trained from scratch — not distilled, not pruned, not
+continued-pretrained from a general base.** KOS-V5 (codename **Catbird**) is the fifth-generation
+Kentucky Open Science base model. This repository holds the **final pretraining checkpoint**: the
+complete single-epoch run, **235.2B tokens, step 49,835 of 49,835**.
+
+This is a **base model**. It has had no instruction tuning, no RLHF, and no chat post-training. It
+completes text; it does not follow instructions. It is the initialisation for a downstream SFT/RL
+line, and it is released as a research artifact.
+
+---
+
+## At a glance
+
+| | |
+|---|---|
+| **Parameters** | 3.72B (36 layers × 2560 hidden, 32/8 GQA, head_dim 128, SwiGLU, **tied** embeddings) |
+| **Training tokens** | **235.2B** — one epoch over a 54-source medical/biomedical corpus, complete |
+| **Final train loss** | ≈1.16 |
+| **Context** | trained at **24,576** tokens (whole-document); `max_position_embeddings` **32,768** |
+| **RoPE** | standard 1D, **θ = 25,000** (pin this — see *Serving notes*) |
+| **Tokenizer** | custom **32k byte-level BPE** ("v5-32k"), `add_bos_token=False`, single special token `<|endoftext|>` (id 0 = eos/bos/pad) |
+| **Objective** | **pure cross-entropy** — no auxiliary/geometric losses (no SigReg, no STP, no MTP) |
+| **Precision / stack** | bf16, 24×H200, DeepSpeed ZeRO-1, FlashAttention-2 |
+| **Architecture class** | `Qwen3ForCausalLM` (stock — exports with no custom modelling code) |
+
+---
+
+## Headline evaluation
+
+Benchmarked against **16 external models** (a 17-model pool; KOS-V4 is held out as our own internal
+comparison). Every comparator was trained on **1.3–153× more data** (0.3–36T tokens vs our 0.235T).
+96 metrics across 19 tests; 93 ranked.
+
+**KOS-V5 is rank-1 of 17 on 12 metrics**, best-tied on 4, top-3 on 9, and trails on 68.
+
+**Where it wins** — and the wins are concentrated, not scattered:
+
+| Axis | Result |
+|---|---|
+| **Held-out medical BPB** (the headline) | **rank 1 / 17**, 5-corpus mean **0.4635** — plus rank-1 on radiology (0.5132), chest-xray (0.6688) and clinical narratives (0.4179) |
+| **Attention health** (T2) | rank-1 on 3 of 4 measures (bos-sink mass, collapsed-head fraction, min-entropy); rank 2 on the fourth |
+| **Representation geometry** (T4) | rank-1 on RankMe (220.0) |
+| **Long-context BPB** | rank-1 at the 1,024- and 2,048-token buckets; bucket-BPB falls monotonically out to the full 32,768 |
+| **Semantic similarity** | rank-1 on BIOSSES (Pearson 0.7097 / Spearman 0.7014) |
+
+Bits-per-byte is tokenizer-agnostic, so it is a fair cross-model number — and the distillation
+confound in the pool *flatters the trillion-token externals, not a from-scratch model*. These BPB
+placements are therefore conservative.
+
+**Where it does not win — stated plainly, because earlier drafts of our own report got this wrong:**
+
+| Axis | Result |
+|---|---|
+| BLURB probe mean | **rank 2 / 17** (0.7268) — *not* rank-1; only BIOSSES is |
+| Calibration (mean ECE) | **rank 9 / 17** (0.1209) — mid-pool, not a strength |
+| RadGraph entity/relation F1 | **rank 7–10 / 17** — high absolute scores (micro-F1 0.90–0.91), mid-pool rank |
+| Long-context needle | **rank 11 / 17** (0.9333) |
+| MedHALT-FCT (false-confidence) | **rank 16 / 17** (0.0280) — **our single worst placement**, and we call it out rather than bury it |
+| Tokenizer single-token rate (STRR) | rank 15 / 17 — a deliberate BBPE trade-off (see *Known issues*) |
+| Closed-book medical MCQ | mean 0.4352 — below the frontier fleet; a post-training problem, not a token-budget one |
+
+---
+
+## KOS-V5 vs KOS-V4 — a corpus trade-off, now settled
+
+Both models are ours, both are finished, and both were run on the identical harness. **KOS-V5 is
+better on 17 of 32 compared metrics, KOS-V4 on 15.**
+
+The token-budget alibi is dead: KOS-V5 finished on **235.2B tokens vs KOS-V4's 180.3B** (1.30×) at a
+larger parameter count. **Every remaining KOS-V4 win is a win on the merits.**
+
+| | KOS-V5 (final, 235B) | KOS-V4 (final, 180B) | Winner |
+|---|---|---|---|
+| **Medical MCQ mean** (16 tasks) | **0.4352** | 0.3319 | **V5** (+0.103; 12 of 16 tasks) |
+| MMLU clinical knowledge | **0.5057** | 0.3094 | **V5** (+0.196) |
+| MMLU college medicine | **0.4335** | 0.2312 | **V5** (+0.202) |
+| BPB 5-corpus mean | 0.4635 | **0.4309** | **V4** |
+| BPB radiology / chest-xray / clinical | 0.5132 / 0.6688 / 0.4179 | **0.4761 / 0.5887 / 0.3221** | **V4** |
+| BPB biomedical-lit / textbooks | **0.1108 / 0.6066** | 0.1243 / 0.6432 | **V5** |
+| RankMe (mean) | **220.0** | 170.8 | **V5** |
+| Long-context BPB @ 4k→32k | **wins every window** | — | **V5** |
+| Needle 3-depth mean | **0.9333** | 0.8667 | **V5** |
+| BLURB mean / RadGraph | 0.7268 / 0.75–0.77 | **0.7465 / 0.77–0.82** | **V4** |
+| PubMedQA / MedThink | 0.6680 / 0.9200 | **0.6980 / 0.9450** | **V4** |
+
+**The honest reading.** KOS-V4's narrow radiology/clinical-dominant corpus still produces the better
+**text model of radiology and clinical notes**. KOS-V5's broader 54-source corpus produces the better
+**medical reasoner** — on the recall-heavy MCQ tasks V4 sat close to chance and V5 does not. This is a
+**corpus-composition difference, not a maturity gradient**, and no amount of further pretraining was
+going to close it: V5 would have needed to recover 0.0326 BPB, and its entire final leg moved it by
+0.0001. Which model is "better" depends on the deployment.
+
+---
+
+## The run converged before it finished — read this before choosing a checkpoint
+
+The identical 16-tier battery was run at **six** checkpoints (90B / 120B / 150B / 190B / 220B / 235B).
+
+| axis | 90B | 120B | 150B | **190B** | 220B | **235B (final)** |
+|---|---|---|---|---|---|---|
+| BPB (5-corpus, lower better) | .4890 | .4781 | .4740 | .4647 | .4636 | **.4635** |
+| MCQ (16 tasks, higher better) | .3478 | .3737 | .4071 | **.4449** | .4332 | **.4352** |
+| Needle (higher better) | .9333 | .9000 | .9500 | .9167 | .9000 | **.9333** |
+| ECE (lower better) | .1847 | **.1135** | .1181 | .1378 | .1183 | **.1209** |
+
+**On the final leg, not one headline axis improved.** BPB moved **+0.0001** (zero to three decimals).
+MCQ **peaked at 190B and never recovered** — the final model ends **0.0097 below its own peak**, with
+12 of 16 tasks below their peak-checkpoint value. Needle ended **exactly** where it began at 90B.
+Calibration ended *worse* than the checkpoint before it. The learning-rate integral is 100% spent.
+
+> **⚠️ The newest checkpoint is NOT the knowledge-best checkpoint.**
+> The **190B** checkpoint is KOS-V5's knowledge-best, permanently. **Choose your fine-tuning
+> initialisation by axis:** the 190B checkpoint for medical knowledge/MCQ; **this final checkpoint for
+> BPB, long-context and general LM health.** Do not assume "newest = best."
+
+Both MCQ moves (the 220B drop and the 235B bounce) sit inside MMLU-subset sampling noise. What
+survives noise is the **plateau**: knowledge accuracy stopped improving 45B tokens before the end.
+
+---
+
+## Known issues
+
+**1. The tokenizer does not split numbers — and this cannot be fixed in V5.**
+
+The v5-32k BBPE has **no digit rule**: its pre-tokenizer is plain byte-level, so digit runs merge into
+single tokens by BPE frequency.
+
+| tokenizer | digit rule | multi-digit tokens | max digit-run | `10` / `100` / `1000` | tokens/number |
+|---|---|---|---|---|---|
+| **KOS-V5** | **none** | **963** | **5** | one token each | **1.80** |
+| Qwen3-4B-Base | yes — **every digit split** | 0 | 1 | `1`·`0` / `1`·`0`·`0` / … | 4.20 |
+| Llama-3.2-3B | yes — ≤3-digit groups | 1100 | 3 | `10` / `100` / `100`+`0` | 2.90 |
+
+**The consequence is compositional:** `10`, `100` and `1000` are three *unrelated atomic symbols* that
+share no substructure. The model cannot see that they differ by a factor of ten; it must learn each
+magnitude as a separate lexical item, and rare numbers fragment on arbitrary BPE boundaries while
+common ones do not. In medicine — doses, lab values, vitals — this is exactly the regime where
+digit-level arithmetic and magnitude comparison are known to degrade.
+
+**Stated with its limit:** this is a **measured property of the frozen tokenizer, not a measured
+downstream failure.** Our evaluation suite contains **no arithmetic or numeric-reasoning tier**, so we
+neither quantify the cost nor claim there is none. The tokenizer is baked into the trained (tied)
+embedding table, so it is unfixable in V5 — **it is a KOS-V6 decision** (add a digit-splitting
+pre-tokenizer), and V6 must ship the missing tier to measure it.
+
+**2. Low medical single-token rate (STRR, rank 15/17).** Deliberate: BBPE spends its 32k vocabulary on
+byte-level compression rather than memorising whole medical terms. It single-tokens only ~19% of
+radiology and ~23% of chest-xray terms. The trade appears sound — tokens-per-byte is rank 4/17 and BPB
+is rank 1/17 — but it is a trade, and it is the same trade as the digit issue above.
+
+**3. Long-context needle retrieval never improved** across the entire run (rank 11/17). More
+pretraining will not fix it; it is a post-training problem.
+
+**4. MedHALT-FCT rank 16/17.** A base model with no refusal prior has essentially no mechanism for
+declining a question with a false premise, and KOS-V5 answers anyway. This is the axis a grounded
+medical executor most needs to improve, and it is a target for the instruction/RL stage.
+
+**5. Attention health is leading but degrading.** KOS-V5 still leads the pool on T2, but bos-sink mass
+rose 0.1619 → 0.2541 and layers-carrying-a-sink rose 0.0830 → 0.3330 across training. The lead is real
+but it shrank; it did not consolidate.
+
+---
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "Kentucky-Open-Science/KOS-V5-Base"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
+
+prompt = "IMPRESSION: The chest radiograph demonstrates"
+ids = tok(prompt, return_tensors="pt").to(model.device)     # add_bos_token=False by design
+out = model.generate(**ids, max_new_tokens=128, do_sample=False)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+### Serving notes (please read — these bite)
+
+- **`rope_theta = 25000`, not 10000.** Some conversion paths (notably GGUF) silently fall back to
+  10,000 and quietly damage long-context behaviour. **Pin it explicitly** when converting.
+- **`add_bos_token=False`.** Setting it true double-prepends BOS and breaks generation.
+- **One special token.** `<|endoftext|>` (id 0) serves eos/bos/pad and the document separator.
+- **Base model, no chat template.** It completes text. Prompt it as a completion model, or post-train it.
+- Requires `transformers` with the nested `rope_parameters` config schema (≥5.x), or convert the config.
+
+---
+
+## Intended use
+
+A **research base model** for medical/biomedical NLP: the initialisation for domain SFT/RL, a subject
+for interpretability and tokenizer/corpus research, and a from-scratch reference point against
+distilled and continued-pretrained medical models. It is not instruction-tuned and is not a
+question-answering system. Its closed-book medical MCQ accuracy is well below the frontier fleet, and
+the known issues above (especially the number tokenization) bear directly on any numeric medical task.
+
+---
+
+## Reproducibility
+
+| artifact | value |
+|---|---|
+| checkpoint | step **49,835** of 49,835 (epoch 1.000) |
+| tokens/step | 1 × 8 × 24 × 24,576 = 4,718,592 |
+| `tokenizer.json` md5 | `9c9df0404f6aae96dba5f3785e8b4c9d` |
+| optimizer | AdamW β=(0.9, 0.95), grad-clip 1.0 |
+| LR schedule | cosine → 0, peak 3.0e-4, 1% warmup (integral 100% spent) |
+| batch | micro_batch 1 × grad_accum 8 × 24 GPUs, seq 24,576 (≈4.72M tokens/step) |
+| corpus | 54 sources, English-only, deduped → decontaminated, whole documents (>24,576 tokens dropped, never split), neat-packed with 4-D block-diagonal segment mask (no cross-document attention; position ids reset per document) |
+
+Full 22-page evaluation report — all 96 metrics, all 17 models, every rank re-derived from source:
+`kos_v5_brief.pdf` (in this repository).
+
+---
+
+## Attribution
+
+Developed at the **University of Kentucky College of Medicine**, **Center for Clinical and
+Translational Science (CCTS)**.
+
+*Additional collaborator attribution to be added.*
+
+## Citation
+
+```bibtex
+@misc{kos_v5_base_2026,
+  title  = {KOS-V5-Base (Catbird): a from-scratch 3.72B medical language model},
+  author = {Kentucky Open Science},
+  year   = {2026},
+  note   = {University of Kentucky College of Medicine,
+            Center for Clinical and Translational Science (CCTS)},
+  url    = {https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base}
+}
+```

config.json

@@ -0,0 +1,71 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 0,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 2560,
+  "initializer_range": 0.02,
+  "intermediate_size": 9728,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "max_window_layers": 36,
+  "model_type": "qwen3",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 36,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 25000.0,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.2.0",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": [
+    0
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.2.0",
+  "use_cache": false
+}

kos_v5_brief.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5a2d204a16ebe03b738649b3f1261d1144011bdb7a0bc28ae08f0e1812be6c2
+size 355280

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02790fecbb3aaedbc43dea9fd12ea7619af7ee822d33bc634ce1ff4b82cf1dda
+size 7594749808

tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<|endoftext|>",
+  "add_bos_token": false,
+  "add_eos_token": false
+}