remcohendriks commited on
Commit
6b45c8e
verified
1 Parent(s): 5096943

Open work: MetroLLM-Bench published (arXiv 2609.10016), dataset link

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -38,7 +38,7 @@ Sovereignty here is technical, not geographic. It is about who controls the keys
38
 
39
  ## Open work
40
 
41
- MetroLLM-Bench is one example. It is a benchmark and a set of open-weight students for running a transit kiosk from a prose prompt. A small local language model handles the task offline on commodity hardware, and the 2.6 GB student matches a frontier cloud API.
42
 
43
  | Model | Size | Runs on |
44
  |-------|-----:|---------|
@@ -47,7 +47,7 @@ MetroLLM-Bench is one example. It is a benchmark and a set of open-weight studen
47
  | [Qwen3.5-9B-metro-v24](https://huggingface.co/continker/Qwen3.5-9B-metro-v24) | 5.3 GB | a 24 GB machine |
48
  | [Qwen3.5-27B-metro-v24](https://huggingface.co/continker/Qwen3.5-27B-metro-v24) | 16 GB | a 32 GB machine |
49
 
50
- [Try the live demo](https://huggingface.co/spaces/remcohendriks/metrollm) 路 [Browse the collection](https://huggingface.co/collections/continker/metrollm-bench-v24-6a35b586a11068e1b1ba3d47) 路 [Benchmark and code](https://github.com/continker/metrollm-bench)
51
 
52
  ---
53
 
 
38
 
39
  ## Open work
40
 
41
+ MetroLLM-Bench: a benchmark for running a transit kiosk from a prose prompt, with open-weight students that handle the task offline on commodity hardware. Published as [arXiv:2609.10016](https://huggingface.co/papers/2609.10016) (September 2026). The 2.6 GB student matches GPT-5.4 at maximum reasoning effort on the deterministic tier and exceeds both GPT-5.6 tiers.
42
 
43
  | Model | Size | Runs on |
44
  |-------|-----:|---------|
 
47
  | [Qwen3.5-9B-metro-v24](https://huggingface.co/continker/Qwen3.5-9B-metro-v24) | 5.3 GB | a 24 GB machine |
48
  | [Qwen3.5-27B-metro-v24](https://huggingface.co/continker/Qwen3.5-27B-metro-v24) | 16 GB | a 32 GB machine |
49
 
50
+ [Paper](https://huggingface.co/papers/2609.10016) [Dataset (955 cases)](https://huggingface.co/datasets/continker/metrollm-bench) 路 [Live demo](https://huggingface.co/spaces/remcohendriks/metrollm) 路 [Collection](https://huggingface.co/collections/continker/metrollm-bench-v24-6a35b586a11068e1b1ba3d47) 路 [Benchmark and code](https://github.com/continker/metrollm-bench)
51
 
52
  ---
53