MILO Benchmark Companion

A read-only, pre-recorded companion to naishashetty/milo_benchmark on the Hugging Face Hub. Everything on this page is loaded from a real benchmark results JSON produced by a real run against real AI2-THOR scenes — there is no live simulator running behind this page. AI2-THOR needs a GPU-backed Unity process this Space's free static/CPU tier cannot run, which is exactly why this is a static replay rather than an interactive demo. See the About tab for full detail.

Loading…

Methodology: goal_success checks live post-execution AI2-THOR object state (check_goal_live()), not just “did every step dispatch without error” (execution_success). Full predicate table in backend/planning_evaluation/dataset/v1.0/README.md and backend/planning_evaluation/live_state.py in the origin repository. tier1_locate’s live scoring predicate only checks that an object of the named type exists in the scene — a known, documented simplification, not a perception check (a separate, additive perceived_by_agent signal exists alongside it; see the origin repo’s benchmark report Addendum 5 for that finding).

A handful of real, logged episodes — at least one success per planner, plus the known still-failing FloorPlan301 book→drawer geometry-limit case. Pick one below.

This Space is a companion to the naishashetty/milo_benchmark dataset and the MILO vision-language-robotics project (origin repository). It shows real, already-computed benchmark results for three planners (rule_based, behavior_tree, react on a local qwen2.5:7b model via Ollama) against a fixed 25-task, 5-scene, 3-tier AI2-THOR benchmark.

Why not a live demo? AI2-THOR requires a GPU-backed Unity process. This Space's static free tier has neither a GPU nor Unity, so live execution is not possible here — this Space intentionally does not pretend otherwise. What you see is a static replay of one specific, timestamped, reproducible run’s output (see the leaderboard’s source caption for exactly which file).

Why a static Space, not an interactive Gradio app? Gradio/Docker Spaces require an HF Pro subscription for CPU hosting; static Spaces are free. Since this page never needed a live Python backend anyway (every number and trace here is precomputed from a results JSON, see generate_data.py in this Space’s repository), a static page loses nothing over a Gradio one for this use case.

Full methodology, honesty notes about known predicate limitations, and this project’s benchmark report are linked from the origin repository’s README and this Space’s own README file.