README.md

metadata

title: MILO Benchmark Companion
emoji: 🤖
colorFrom: blue
colorTo: purple
sdk: static
pinned: false
license: mit

MILO Benchmark Companion

A read-only companion demo to naishashetty/milo_benchmark on the Hugging Face Hub, and to the MILO vision-language-robotics project (origin repository). It shows a leaderboard and an episode replay browser built entirely from a real benchmark results JSON — four planners (rule_based, behavior_tree, htn, react on a local qwen2.5:7b model via Ollama) scored against a fixed 25-task, 5-scene, 3-tier AI2-THOR benchmark.

This is not a live demo, and that's deliberate

AI2-THOR requires a GPU-backed Unity subprocess to actually simulate a scene. HF Spaces' free tier has no GPU and cannot run Unity, so this Space cannot execute a real episode interactively, full stop. Rather than fake it (e.g. replaying a canned animation and implying it's live), this Space is explicit about being a static replay of one specific, timestamped, already-completed, reproducible benchmark run. If you want to run the benchmark for real, clone the origin repository and run backend/planning_evaluation/run_benchmark.py yourself against a real AI2-THOR installation (needs a GPU).

What's real vs. reconstructed here

This project's own writing convention (see experiments/reports/phase_e_milo_benchmark_report.md in the origin repo) is to say plainly what's measured vs. illustrative. Applied here:

• Leaderboard numbers: real. Read directly from the source results JSON's summary_by_planner block (never re-derived from episodes, except for a couple of cost/latency columns not present in summary_by_planner at all, which are averaged from episodes instead — see data.py).
• Episode replay's logged fields (planner, scene, instruction, goal/object/target, tier, plan_success/execution_success/goal_success, wall_clock_ms, failure_cause, llm_retry_attempts): real, read directly from that episode's row in the source JSON.
• Episode replay's "reconstructed plan trace": not real logged data. The source JSON only has aggregate per-episode counts (action_count, plan_step_count) — there is no literal per-step action log to replay. The step lists shown are built from each task's goal/object/target plus this project's documented, deterministic planner behavior (tier3_store = locate → navigate → pick_up → navigate → open if openable → place → close if opened, per _deposit()'s documented behavior in the dataset README and benchmark report). Every trace is labeled "reconstructed plan trace for illustration" in the UI. We chose to build these (rather than the simpler, arguably more conservative option of only showing logged fields) because a bare goal/object/target/outcome table alone doesn't give a reader unfamiliar with the codebase any sense of what a planner's "shape" of behavior actually looks like — but we did not want to under-label them as real trace data, since they aren't.
• Screenshots: real UI screenshots from the live MILO product (docs/screenshots/demo/ in the origin repo — 3 generic images: instruction typed, task in progress, task complete), not captured per-episode. There is no screenshot for each of the 25 dataset tasks. Where shown alongside an episode, the caption says explicitly that it's an illustrative example of the UI, not a capture of that literal episode's run.

Why static HTML, not Gradio

This Space started as a Gradio Blocks app (a Dataframe for the leaderboard plus a dropdown-driven detail view) — a small, well-trodden pattern for exactly this "table + browsable detail" shape. It was rebuilt as static HTML/CSS/JS after discovering that Gradio and Docker Spaces require an HF Pro subscription for CPU hosting on this account; static Spaces are free. Nothing about the page's actual content or logic needed the change: everything here was already precomputed from a results JSON with no live Python execution per request, so a static page loses nothing over the Gradio version for this use case — it swaps a server-rendered Dataframe/Dropdown for the equivalent plain HTML table and <select>, both populated from the same data at page-load via fetch("data.json").

How the static site is built

Nothing here is hand-authored data — data.json (what script.js fetches) is generated ahead of time by generate_data.py, which reuses data.py/episodes.py's exact loading, leaderboard-row, and plan-trace-reconstruction logic unchanged (only the output target changed, from an in-process Gradio render to a JSON file). Regenerate it with:

python3 generate_data.py

whenever a newer results/milo_benchmark_*.json lands (see "Data source" below) — then commit and redeploy. index.html/style.css/ script.js never need to change for a data refresh alone.

Data source and how it stays current

data.py's loader does not hardcode a filename — it globs results/milo_benchmark_*.json (excluding *_memory_ablation_*.json, a different experiment) and picks the lexicographically-latest generated_at_utc-stamped filename. results/ currently ships two real runs from the origin repo's experiments/results/:

• milo_benchmark_20260817T154347Z.json — react run against Gemini's free tier, which hit a 20-request/day quota after 2 of 25 episodes. Kept here for the record, but its react numbers are not a valid capability baseline (see the origin repo's benchmark report, Addendum 2, for exactly why the raw success rate from that run is misleading).
• milo_benchmark_20260818T070841Z.json — react run against a local qwen2.5:7b (Ollama), zero external quota dependency, a clean 25/25 completed run. This is the file the leaderboard above actually loads, since it's the newer of the two.

If the origin repository produces a newer full run (for example, the perception-grounded tier1_locate check and LLM call/token instrumentation for react landed in the origin repo's experiments/results/milo_benchmark_20260818T085629Z.json after this Space was first built), copy that JSON into results/, re-run generate_data.py, and redeploy. The leaderboard code path is written defensively for this: if a source JSON lacks a given column (e.g. react's LLM-call or token counts), that column renders as — instead of crashing or silently showing a fabricated 0.

Methodology (matches the published dataset card)

goal_success is the primary score: it checks live post-execution AI2-THOR object state (check_goal_live() in backend/planning_evaluation/live_state.py), not just "did every planned action dispatch without an error" (execution_success — a weaker, separate signal, also shown). Full predicate table and known limitations (including the honest caveat that tier1_locate's live check only verifies an object of the named type exists in the scene, not that it was actually perceived — and the separate, additive perceived_by_agent signal that now measures that too) are in backend/planning_evaluation/dataset/v1.0/README.md in the origin repository — read that file for the canonical methodology text; this Space's copy is a summary, not a re-derivation.

Links

• Dataset: huggingface.co/datasets/naishashetty/milo_benchmark
• Origin repository: github.com/NaishaShetty/MILO
• Full benchmark report (methodology, all four planners' real numbers, every caveat this README summarizes): experiments/reports/phase_e_milo_benchmark_report.md in the origin repository.

Running locally

No server, no dependencies beyond Python (only needed to regenerate data.json, not to view the page):

python3 -m http.server 8080

then open http://127.0.0.1:8080. No GPU, no AI2-THOR, no network access required — everything renders from data.json and the images in screenshots/.

Files

hf_space/
├── index.html          # page markup: leaderboard / episode replay / about tabs
├── style.css            # styling (light + dark, no build step)
├── script.js             # fetches data.json, renders the leaderboard table and episode detail view
├── data.json              # generated by generate_data.py -- the only thing script.js fetches
├── generate_data.py        # build step: re-derives data.json from results/*.json (not served/run by the Space itself)
├── data.py                  # results JSON loader + leaderboard row builder (defensive re: optional columns) -- used by generate_data.py
├── episodes.py                # curated episode picks + reconstructed-plan-trace builder (clearly labeled) -- used by generate_data.py
├── README.md                   # this file (HF Space card)
├── results/
│   ├── milo_benchmark_20260817T154347Z.json   # react/Gemini run (quota-limited, kept for record)
│   └── milo_benchmark_20260818T070841Z.json   # react/qwen2.5:7b run (the one currently loaded)
└── screenshots/
    ├── live-01-instruction-typed.png
    ├── live-02-task-in-progress.png
    └── live-03-task-complete.png

Status of this Space

Built and verified locally (served via python3 -m http.server, confirmed the page loads, the leaderboard renders real data, and episode replay works with no console errors) and live on the Hugging Face Hub — see the top of the origin repository's README for the live URL.