# Manuscript Tables

## Table 1. Broad Model First-Appearance Temporal Metrics

Caption: No-review-status broad score_model evaluated on variants first appearing between ClinVar 2023-12-26 and 2025-12-21.

| Split | N | AUROC | AUPRC | Brier | ECE |
|---|---:|---:|---:|---:|---:|
| first-appearance temporal | 509974 | 0.99125 | 0.96938 | 0.02903 | 0.03428 |

## Table 2. Per-Consequence Breakdown

Caption: Same first-appearance split; low-N classes are aggregated.

| Consequence | N | Pos | Neg | AUROC | AUPRC | Brier | ECE |
|---|---:|---:|---:|---:|---:|---:|---:|
| non-coding_transcript_variant | 34607 | 257 | 34350 | 0.91551 | 0.61517 | 0.00398 | 0.00219 |
| 3_prime_UTR_variant | 3861 | 24 | 3837 | 0.89588 | 0.13295 | 0.00623 | 0.00464 |
| other (low-N) | 534 | 336 | 198 | 0.87042 | 0.90320 | 0.13473 | 0.05293 |
| unknown | 1503 | 542 | 961 | 0.86056 | 0.75580 | 0.14478 | 0.05336 |
| missense_variant | 67847 | 12943 | 54904 | 0.81292 | 0.55438 | 0.18069 | 0.23174 |
| splice_donor_variant | 8438 | 8238 | 200 | 0.81240 | 0.99205 | 0.01565 | 0.00245 |
| 5_prime_UTR_variant | 6609 | 68 | 6541 | 0.77156 | 0.23015 | 0.00891 | 0.00739 |
| inframe_deletion | 386 | 219 | 167 | 0.76963 | 0.80047 | 0.20247 | 0.08748 |
| initiator_codon_variant | 713 | 615 | 98 | 0.74360 | 0.93131 | 0.09672 | 0.04770 |
| intron_variant | 153601 | 951 | 152650 | 0.73241 | 0.03627 | 0.00619 | 0.00863 |
| frameshift_variant | 38296 | 37947 | 349 | 0.72617 | 0.99567 | 0.00786 | 0.00094 |
| nonsense | 22375 | 22034 | 341 | 0.70115 | 0.99182 | 0.01238 | 0.00779 |
| splice_acceptor_variant | 7209 | 7081 | 128 | 0.70025 | 0.98999 | 0.01468 | 0.00421 |
| synonymous_variant | 163995 | 121 | 163874 | 0.64395 | 0.00175 | 0.00074 | 0.00078 |

## Table 3. External Comparator, All-Variant Overlap

Caption: First-appearance temporal overlap where Project 1 external annotations exist; AlphaMissense/REVEL coverage is limited by design.

| Model | N | AUROC | AUPRC | Brier |
|---|---:|---:|---:|---:|
| score_model v0 | 27850 | 0.98520 | 0.98167 | 0.04545 |
| CADD PHRED | 27850 | 0.99209 | 0.99086 | 0.18824 |
| AlphaMissense | 4988 | 0.97183 | 0.98189 | 0.07705 |
| REVEL | 6275 | 0.95642 | 0.97127 | 0.09579 |

## Table 4. External Comparator, Missense-Only

Caption: First-appearance temporal missense-only comparison; AlphaMissense and REVEL are missense-oriented predictors.

| Model | N | AUROC | AUPRC | Brier |
|---|---:|---:|---:|---:|
| score_model v0 | 5220 | 0.78497 | 0.83876 | 0.18825 |
| missense v1 | 5018 | 0.83644 | 0.87361 | 0.20575 |
| CADD PHRED | 5220 | 0.94829 | 0.95653 | 0.15822 |
| AlphaMissense | 4965 | 0.97209 | 0.98214 | 0.07687 |
| REVEL | 5140 | 0.97769 | 0.98612 | 0.06937 |

## Table 5. Qwen v0.1 vs Llama v0.2 Report Generation

Caption: Structured report-generation comparison; see source comparison report for full details.

# KAUBioMED-LLM Qwen v0.1 vs Llama v0.2 Comparison

Date: 2026-06-15

## Scientific Position

This comparison evaluates two KAUBioMED-LLM report-generation models under the same score-schema and guarded-report framework. The current kaubiomed_score remains an evidence-label score for report organization, not a leakage-free calibrated pathogenicity probability.

## Summary Table

| Metric | Qwen v0.1 | Llama v0.2 |
|---|---:|---:|
| Base model | Qwen2.5-1.5B-Instruct | Llama-3.1-8B-Instruct |
| Base/snapshot size | 2.89 GB | 29.93 GB |
| Adapter path | models/kaubiomed_qwen2p5_1p5b_lora_calibrated | models/kaubiomed_llama31_8b_lora_score_schema |
| Adapter size | 304.0 MB | 1.14 GB |
| Training records | 910 | 910 |
| Validation records | 101 | 101 |
| Epochs | 1 | 2 |
| Train loss | 0.0475554999552275 | 0.22699994018726183 |
| Eval loss | 0.04750876873731613 | 0.0603446401655674 |
| 50-example raw strict JSON | not separately measured for calibrated Qwen | 50/50 (1.0) |
| Fresh strict valid | 5/5 | 5/5 |
| Fresh repaired records | NA | 0 |
| Citation guard pass | 5/5 | 5/5 |
| Structure mapping OK | 1/5 | 1/5 |
| Structure abstained | 4 | 4 |
| Fresh inference runtime | not consistently recorded in final Qwen summary | 114.12 sec |

## Interpretation

- Qwen v0.1 remains the lightweight reproducible prototype baseline.
- Llama v0.2 is the stronger structured-output candidate: it achieved 50/50 raw strict JSON on validation and 5/5 raw strict JSON on fresh variants without repair.
- Both models pass citation and abstention guards on the 5 fresh reports.
- Structure behavior is identical because structure mapping is evidence-layer dependent, not base-model dependent: 1/5 fresh variants map to a reconciled missense residue; 4/5 correctly abstain.
- Llama is larger and more expensive, so it should be positioned as a stronger v0.2 candidate rather than simply replacing Qwen without cost discussion.

## Recommendation

Use Qwen v0.1 as the lightweight baseline and Llama v0.2 as the preferred high-capability report-generation model. The next scientific step is not another base-model swap; it is the leakage-free score_model and temporal validation benchmark.

## Required Next Work

1. Build label-free feature tables without ClinVar-derived prompt labels.
2. Train a separate score model on clean features.
3. Run temporal ClinVar validation with AUROC, AUPRC, Brier, and ECE.
4. Add the benchmark results to the manuscript and model card.