Open work: MetroLLM-Bench published (arXiv 2609.10016), dataset link
Browse files
README.md
CHANGED
|
@@ -38,7 +38,7 @@ Sovereignty here is technical, not geographic. It is about who controls the keys
|
|
| 38 |
|
| 39 |
## Open work
|
| 40 |
|
| 41 |
-
MetroLLM-Bench
|
| 42 |
|
| 43 |
| Model | Size | Runs on |
|
| 44 |
|-------|-----:|---------|
|
|
@@ -47,7 +47,7 @@ MetroLLM-Bench is one example. It is a benchmark and a set of open-weight studen
|
|
| 47 |
| [Qwen3.5-9B-metro-v24](https://huggingface.co/continker/Qwen3.5-9B-metro-v24) | 5.3 GB | a 24 GB machine |
|
| 48 |
| [Qwen3.5-27B-metro-v24](https://huggingface.co/continker/Qwen3.5-27B-metro-v24) | 16 GB | a 32 GB machine |
|
| 49 |
|
| 50 |
-
[
|
| 51 |
|
| 52 |
---
|
| 53 |
|
|
|
|
| 38 |
|
| 39 |
## Open work
|
| 40 |
|
| 41 |
+
MetroLLM-Bench: a benchmark for running a transit kiosk from a prose prompt, with open-weight students that handle the task offline on commodity hardware. Published as [arXiv:2609.10016](https://huggingface.co/papers/2609.10016) (September 2026). The 2.6 GB student matches GPT-5.4 at maximum reasoning effort on the deterministic tier and exceeds both GPT-5.6 tiers.
|
| 42 |
|
| 43 |
| Model | Size | Runs on |
|
| 44 |
|-------|-----:|---------|
|
|
|
|
| 47 |
| [Qwen3.5-9B-metro-v24](https://huggingface.co/continker/Qwen3.5-9B-metro-v24) | 5.3 GB | a 24 GB machine |
|
| 48 |
| [Qwen3.5-27B-metro-v24](https://huggingface.co/continker/Qwen3.5-27B-metro-v24) | 16 GB | a 32 GB machine |
|
| 49 |
|
| 50 |
+
[Paper](https://huggingface.co/papers/2609.10016) 路 [Dataset (955 cases)](https://huggingface.co/datasets/continker/metrollm-bench) 路 [Live demo](https://huggingface.co/spaces/remcohendriks/metrollm) 路 [Collection](https://huggingface.co/collections/continker/metrollm-bench-v24-6a35b586a11068e1b1ba3d47) 路 [Benchmark and code](https://github.com/continker/metrollm-bench)
|
| 51 |
|
| 52 |
---
|
| 53 |
|