Update TwIL-LM3: weights, tokenizer and model card

README.md

@@ -82,13 +82,99 @@ Every regression is within 0.033, and the gains on logical-reasoning transfer ta
 (LogicBench +0.070, DROP +0.047) are larger than any loss. IFEval is the one place worth
 noting — instruction-following degrades slightly, which is a common cost of verifier-driven RL.
 
+### Comparison against other open models
+
+All arms below were run through the same harness, prompts and decoding settings described under
+[Evaluation protocol](#evaluation-protocol). Throughput rows are reported because in-domain score
+alone is misleading for a 3B model: `ans/s` is defined throughout as `tok/s ÷ mean generation
+length`, so it measures completed answers rather than raw decode rate.
+
+#### Track A — in-domain formal logic
+
+| lane / metric | TwIL-LM3 | SmolLM3-3B base | Llama-3.2-3B | LFM2-2.6B | LFM2.5-8B-A1B | Qwen3-8B |
+|---|---:|---:|---:|---:|---:|---:|
+| lean_formalize token_f1 | **0.5869** | 0.4347 | 0.3690 | 0.1321 | 0.4655 | 0.4022 |
+| rule_induction derivation | 0.3192 | 0.1029 | 0.0825 | 0.0615 | 0.1936 | **0.3680** |
+| entailment_label accuracy | 0.5750 | 0.3750 | 0.3300 | 0.4700 | 0.5400 | **0.5800** |
+| mcq_answer accuracy | **0.1100** | 0.0000 | 0.0000 | 0.0150 | 0.0750 | 0.0000 |
+| semantic_parse token_f1 | **0.4416** | 0.4149 | 0.3102 | 0.3665 | 0.3778 | 0.4257 |
+| lean_critic accuracy | 0.6600 | 0.6500 | 0.5300 | 0.5900 | 0.5500 | **0.7950** |
+| lean_formalize exact_match | 0.0050 | 0.0050 | 0.0000 | 0.0000 | 0.0000 | 0.0050 |
+| fol_translation exact_match | 0.0000 | 0.0050 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
+| semantic_parse exact_match | 0.0000 | 0.0050 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
+| procedural accuracy | 0.0300 | 0.0050 | 0.0000 | 0.0300 | 0.0350 | **0.0850** |
+| procedural loose_match | 0.1100 | 0.1050 | 0.1050 | 0.1150 | 0.1400 | **0.1800** |
+| mcq_answer loose_match | 0.4450 | 0.5000 | 0.4150 | 0.5000 | 0.4550 | **0.7450** |
+| lm_corpus perplexity ↓ | 2.8972 | 3.1818 | 2.8478 | 4.3815 | 4.9472 | **2.5440** |
+| math_corpus perplexity ↓ | **3.8229** | 4.0685 | 4.7531 | 6.7472 | 8.3323 | 4.0083 |
+| **macro gate** | 0.4218 | 0.3466 † | 0.2925 | 0.3473 | 0.3757 | **0.5336** |
+| **strict-7** | 0.1971 | 0.1493 | 0.1229 | 0.1579 | 0.1714 | **0.2093** |
+| tok/s | 15880 | 15564 | 16160 | 25000 | 22000 | not measured |
+| mean gen length | **564** | 999 | 696 | 2296 | 1830 | 2094 |
+| **ans/s** | **28.1** | 15.6 | 23.2 | 10.9 | 12.0 | not measured |
+
+† The base column here comes from the external-comparison run rather than the paired run used
+for the Δ table above, hence 0.3466 against 0.3356 — run-to-run variation of the same
+checkpoint. The paired run is the correct basis for the improvement claim.
+
+`strict-7` is the mean of seven lanes scored under strict metrics only (`fol_translation`,
+`entailment_label`, `mcq_answer`, `semantic_parse` and `lean_formalize` exact match,
+`lean_critic` and `procedural` accuracy), with no loose-match credit anywhere.
+
+Qwen3-8B takes the macro gate at roughly 2.7x the parameter count, driven by the classification
+lanes — `lean_critic` 0.7950 and loose MCQ 0.7450. TwIL-LM3 holds the two lanes this pipeline
+targets most directly, `lean_formalize` token-F1 (0.5869 against 0.4022) and strict MCQ accuracy
+(0.1100, the only non-trivial value in that row), and it is the most efficient arm in the table
+by a wide margin: 28.1 answers/sec, from generations averaging 564 tokens where every other arm
+except Llama runs past 690.
+
+#### Track B — held-out benchmarks
+
+| dataset | TwIL-LM3 | SmolLM3-3B base | Llama-3.2-3B | LFM2-2.6B | LFM2.5-8B-A1B | Qwen3-8B | gpt-oss-120b ‡ |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| gsm8k | 0.8733 | 0.8833 | 0.8300 | 0.8767 | 0.9133 | 0.9567 | **0.9767** |
+| svamp | 0.8500 | 0.8567 | 0.8200 | 0.9000 | 0.9133 | **0.9400** | **0.9400** |
+| gsm_symbolic | 0.7567 | 0.7633 | 0.8067 | **0.9767** | 0.9267 | 0.8133 | 0.8467 |
+| arc_cot | 0.8467 | 0.8400 | 0.7967 | 0.8667 | 0.9033 | 0.9633 | **0.9667** |
+| logicbench | 0.7167 | 0.6467 | 0.5733 | 0.6267 | 0.7200 | **0.8567** | 0.8533 |
+| strategyqa | 0.6500 | 0.6333 | 0.6533 | 0.6433 | 0.6667 | 0.7400 | **0.7867** |
+| drop | 0.7467 | 0.7000 | 0.6733 | 0.6900 | 0.6633 | **0.8833** | 0.8500 |
+| csqa | 0.7367 | 0.7067 | 0.7500 | 0.7433 | 0.7700 | **0.8633** | 0.8367 |
+| musr | 0.4957 | 0.4997 | 0.4932 | 0.4867 | 0.5703 | 0.6301 | **0.6852** |
+| mmlu_redux | 0.6667 | 0.6633 | 0.6000 | 0.7133 | 0.8367 | 0.8500 | **0.9467** |
+| ifeval | 0.6433 | 0.6767 | 0.7167 | 0.7300 | **0.8900** | 0.8400 | 0.7900 |
+| rudas_ood | 0.0365 | 0.0209 | **0.0733** | 0.0017 | 0.0061 | 0.0468 | 0.0000 § |
+| bbh_logic | 0.6633 | 0.6667 | 0.5333 | 0.5713 | 0.7700 | 0.6367 | **0.9980** |
+| math500 | 0.6900 | 0.7000 | 0.4233 | 0.7133 | 0.7800 | 0.6100 | **0.8433** |
+| **macro (10 CoT datasets)** | 0.7339 | 0.7193 | 0.6997 | 0.7523 | 0.7884 | 0.8493 | **0.8689** |
+| **macro (all 14)** | 0.6694 | 0.6612 | 0.6245 | 0.6814 | 0.7378 | 0.7591 | **0.8086** |
+| tok/s | 15880 | 15564 | 16160 | 25000 | 22000 | not measured | 3374 |
+| mean gen length | **482** | 626 | 510 | ≈796 | ≈1327 | ≈1931 | 801 |
+| **ans/s** | **32.9** | 24.9 | 31.7 | ≈31.4 | ≈16.6 | not measured | 4.2 |
+
+‡ MXFP4 weights, tensor-parallel 2 — quantized and multi-GPU, so not directly comparable to the
+single-GPU BF16 rows. § 74% of its `rudas_ood` generations hit the length cap, so that cell is a
+truncation artefact rather than a measured score; excluding the row, its 13-dataset macro is
+0.8708.
+
+Lengths marked ≈ are derived from stored generations using each model's characters-per-token
+ratio rather than re-tokenized directly; the method reproduces the three directly measured
+lengths to within 3.5%.
+
+The honest summary of this table is that TwIL-LM3 does not lead it. Larger models score higher,
+in order of size, and the 120B leads nine of fourteen rows. Two things are worth extracting
+anyway. First, TwIL-LM3 improves on its own base while sitting mid-table (0.7339 against 0.7193
+on the 10-dataset macro), which is the point of the WiSE-FT stage — in-domain gains without
+transfer collapse. Second, it produces the shortest generations of any arm here at 482 tokens
+and consequently the most answers per second at 32.9, roughly eight times the 120B's rate.
+
 ## Usage
 
 ```python
 import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-model_id = "webAI-Official/TwIL-LM3/TwIL-LM3"
+model_id = "webAI-Official/TwIL-LM3"
 tok = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(
     model_id, torch_dtype=torch.bfloat16, device_map="auto"
@@ -115,6 +201,32 @@ budget. Note that the shipped `generation_config.json` inherits SmolLM3's sampli
 explicitly to reproduce the evaluation. The model opens a `<think>...</think>` reasoning block
 before answering, so a short generation budget truncates reasoning and scores far worse.
 
+### GGUF / llama.cpp
+
+Quantized GGUF builds ship in this repository alongside the safetensors weights. The `smollm3`
+architecture is supported by llama.cpp, and the chat template, `<|im_end|>` EOS and BOS are
+carried into the GGUF metadata, so chat mode works without extra flags.
+
+| file | quant | size | bits/weight | notes |
+|---|---|---:|---:|---|
+| `TwIL-LM3-Q4_K_M.gguf` | Q4_K_M | 1.78 GiB | 4.96 | recommended default; runs on CPU or 4 GB of VRAM |
+| `TwIL-LM3-Q8_0.gguf` | Q8_0 | 3.05 GiB | 8.50 | near-lossless, for quality-sensitive use |
+
+```bash
+llama-cli -m TwIL-LM3-Q4_K_M.gguf -cnv --temp 0 -n 2048
+```
+
+Two things matter for reproducing the scores above under llama.cpp. Pass `--temp 0`, because the
+evaluation is greedy while the packaged sampling defaults are not. And leave the generation
+budget large — 2048 tokens or more — since the model emits a `<think>` block before answering
+and a short budget truncates it, which costs far more accuracy than the quantization does.
+
+Q8_0 was produced directly by `convert_hf_to_gguf.py` from the released bf16 weights; Q4_K_M was
+produced from an F16 conversion with `llama-quantize`, without an importance matrix. Both builds
+were smoke-tested for load and generation on CPU. The published Track A and Track B numbers were
+measured on the **bf16** weights through vLLM, not on these GGUF builds, so expect small
+deviations at Q4_K_M that have not been quantified here.
+
 ## How it was built
 
 Four stages on top of the base model: