Update model card + sync app to latest

README.md

@@ -1,14 +1,9 @@
 ---
 title: "AnyTraverse Studio 🚜"
 emoji: "🚜"
-colorFrom: "slate"
-colorTo: "blue"
-sdk: "gradio"
-sdk_version: "6.22.0"
-app_file: "app.py"
-pinned: false
-python_version: "3.12"
-short_description: "Live off-road traversability evaluation dashboard with Human-in-the-Loop (AnyTraverse)"
+colorFrom: "blue"
+colorTo: "indigo"
+short_description: "Off-road traversability evaluation with HITL"
 tags:
   - computer-vision
   - robotics
@@ -22,25 +17,97 @@ A live Gradio dashboard for evaluating the **AnyTraverse** zero-shot off-road
 traversability framework ([paper](https://arxiv.org/abs/2506.16826),
 [PyPI](https://pypi.org/project/anytraverse/)).
 
-## Workflow
-1. **Upload a video** of an off-road scene.
-2. Set the traversability preferences (τ), scene-similarity threshold,
-   ROI-uncertainty threshold, ROI bounds and the VLM inference **frame skip**.
-3. Press **▶️ Go / Reset**.
-4. Watch per frame:
-   - raw image + ROI box · traversability map · uncertainty map · ROI crop
-   - **attention maps for all prompts** (live, not in the exported video)
-   - live dual-metric **ROI traversability + uncertainty** line plot
-   - horizontal **0–1 gauge bars** for the two ROI scores
-   - the traversal state (`ok` / `unknown_scene` / `unknown_object`)
-5. The run **halts** whenever `traversal_state != OK`. Provide an operator
-   update like `mud: -0.7; gravel: 0.6` (or just `ok`) and press **Resume**.
-
-Thresholds and ROI bounds can be edited **live** during a run — they are
-applied directly to the running pipeline.
+Try it online:
+[huggingface.co/spaces/sattwik21/anytraverse-studio](https://huggingface.co/spaces/sattwik21/anytraverse-studio)
+(runs on ZeroGPU), or run it locally on your own machine (see below).
+
+## Features
+
+- **Live per-frame evaluation** of any uploaded off-road video:
+  raw image + ROI box · traversability map · uncertainty map · ROI crop
+- **Attention maps for all prompts** live in the UI (grid of up to 5 columns,
+  aspect-ratio preserved) — never baked into the exported video
+- **Human-in-the-Loop**: the run halts whenever the traversal state is not
+  `ok`; provide a τ update (e.g. `mud: -0.7; gravel: 0.6`, or just `ok`) and
+  press **Resume**
+- **Editable live**: thresholds and ROI bounds update the running pipeline
+  without restarting
+- Live dual-metric chart + 0–1 gauge bars + telemetry table
+- Composed analysis video exported as browser-playable H.264 (bundled
+  `imageio-ffmpeg`, no system ffmpeg required)
+- Hard **frame skip**: pass every k-th frame to the VLM; skipped frames are
+  neither shown in the UI nor written to the video
+
+## Run locally
+
+### 1. Prerequisites
+
+- **Linux / macOS / Windows** with **Python 3.12+**
+- A **GPU with CUDA** is strongly recommended (CLIPSeg + CLIP inference).
+  CPU works but is slow.
+- [uv](https://docs.astral.sh/uv/) (fast, optional) or `pip` + a virtualenv
+
+### 2. Clone
+
+```bash
+git clone https://huggingface.co/sattwik21/anytraverse-studio
+cd anytraverse-studio
+```
+
+### 3. Install dependencies
+
+With `uv` (recommended):
+
+```bash
+uv sync
+```
+
+Or with plain `pip`:
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate          # Windows: .venv\Scripts\activate
+pip install -r requirements.txt
+```
+
+If `Torch` pulls the wrong build, install the CUDA build explicitly, e.g.:
+
+```bash
+pip install torch --index-url https://download.pytorch.org/whl/cu128
+pip install torchvision --index-url https://download.pytorch.org/whl/cu128
+```
+
+### 4. Run
+
+```bash
+python app.py
+```
+
+The dashboard opens at http://localhost:7860. Since `share=True` is enabled
+outside Hugging Face Spaces, Gradio will also print a temporary public share
+link at startup — you can ignore it for local use.
+
+### 5. Usage
+
+1. Upload an off-road video (`.mp4`, `.mov`, `.avi`).
+2. Leave τ as `{}` for default preferences, or enter e.g.
+   `{"road": 1.0, "grass": 0.0, "bush": -0.8}`.
+3. Set the **Ref Scene Sim. Threshold**, **ROI Uncertainty Threshold** and
+   **Frame Skip**.
+4. Press **▶️ Go / Reset**.
+5. When the run halts (`unknown_scene` / `unknown_object`), type an operator
+   update in the box and press **✅ Apply & Resume**.
 
 ## Notes
-- The VLM weights (CLIPSeg + CLIP) download on first run and are cached.
-- The composed analysis video is exported as an H.264 `.mp4` (bundled
-  `imageio-ffmpeg`, no system ffmpeg required).
-- GPU is recommended; the space is configured for a **T4** accelerator.
+
+- The first run downloads the VLM weights (CLIPSeg + CLIP, ~1 GB) and caches
+  them locally.
+- `data/weights/` and the generated `*.mp4` files are runtime artifacts; the
+  dashboard writes `raw_opencv_temp.mp4` and `anytraverse_h264_output.mp4` in
+  the working directory.
+
+## Deployment
+
+The interactive Space lives at
+[huggingface.co/spaces/sattwik21/anytraverse-studio](https://huggingface.co/spaces/sattwik21/anytraverse-studio).
+This repo is the model card / source for the same dashboard.

app.py

@@ -1,3 +1,8 @@
+try:
+    import spaces  # noqa: E402  must precede any CUDA-initializing import (ZeroGPU)
+except ImportError:  # local dev without the spaces runtime
+    spaces = None
+
 import json
 import os
 import shutil
@@ -208,15 +213,8 @@ def _simulate_state(bgr, prefs, uncert_thresh, frame_idx):
     }
 
 
-def seed_state(bgr, box):
-    h, w, _ = bgr.shape
-    return {
-        "raw_bgr": bgr, "roi_bbox": box,
-        "trav": np.full((h, w), 0.5, np.float32),
-        "uncert": np.zeros((h, w), np.float32),
-        "attn_maps": [], "roi_trav": 0.5, "roi_uncert": 0.0, "sim": 1.0,
-        "state": TraversalState.OK,
-    }
+def seed_state(bgr, box):  # pragma: no cover - kept unused (records placeholder)
+    return None
 
 
 # ---------------------------------------------------------------------------
@@ -246,15 +244,28 @@ def build_grid(p):
 
 
 def build_attn_strip(p):
-    """All prompt attention maps as one labeled strip (not written to the video)."""
+    """Prompt attention maps as a grid: max 5 columns, wrapping to new rows.
+    Each cell keeps the map's native aspect ratio (no distortion)."""
     raw = p["raw_bgr"]
     h, w, _ = raw.shape
     att = p["attn_maps"]
     if not att:
         return np.zeros((h, w, 3), dtype=np.uint8)
-    cell_w = max(int(w // len(att)), 80)
-    cells = [add_caption(colorize(m, cell_w, h).copy(), name) for name, m in att]
-    strip = np.hstack(cells)
+    cell_w = max(int(w // min(len(att), 5)), 80)
+    first = to_numpy(att[0][1])
+    if first.ndim > 2:
+        first = first.reshape(first.shape[-2:])
+    ah, aw = first.shape[:2]
+    cell_h = max(int(round(cell_w * ah / max(aw, 1))), 1)
+    blank = np.zeros((cell_h, cell_w, 3), dtype=np.uint8)
+    rows = []
+    for i in range(0, len(att), 5):
+        cells = [add_caption(colorize(m, cell_w, cell_h).copy(), name)
+                 for name, m in att[i:i + 5]]
+        while len(cells) < 5:
+            cells.append(blank.copy())
+        rows.append(np.hstack(cells))
+    strip = rows[0] if len(rows) == 1 else np.vstack(rows)
     return cv2.cvtColor(strip, cv2.COLOR_BGR2RGB)
 
 
@@ -310,6 +321,66 @@ def initial_render(msg):
             EMPTY_DF, None)
 
 
+# ---------------------------------------------------------------------------
+# ZeroGPU worker(s). All CUDA work must live in a @spaces.GPU function; the
+# decorator is a no-op when the `spaces` runtime is absent (local dev).
+# The pipeline as a singleton is cached in `_VLM` so consecutive frames reuse
+# the loaded model inside the GPU context.
+# ---------------------------------------------------------------------------
+_VLM = {"pipe": None}
+
+
+def _pipeline_ready():
+    return _VLM["pipe"] is not None
+
+
+def _set_pipe():
+    session.pipeline = _VLM["pipe"]
+
+
+def _vlm_make_pipe(prefs, sim_thresh, uncert_thresh, rx, ry):
+    return build_pipeline_from_paper(
+        init_traversabilty_preferences=prefs,
+        ref_scene_similarity_threshold=float(sim_thresh),
+        roi_uncertainty_threshold=float(uncert_thresh),
+        roi_x_bounds=(float(rx[0]), float(rx[1])),
+        roi_y_bounds=(float(ry[0]), float(ry[1])),
+    )
+
+
+def _vlm_step_raw(frame_bgr, prefs, sim_thresh, uncert_thresh, rx, ry):
+    if not _pipeline_ready():
+        _VLM["pipe"] = _vlm_make_pipe(prefs, sim_thresh, uncert_thresh, rx, ry)
+    st = _VLM["pipe"].step(image=PILImage.fromarray(cv2.cvtColor(frame_bgr,
+                                                               cv2.COLOR_BGR2RGB)))
+    return unpack_state(st, frame_bgr)
+
+
+def _vlm_human_raw(text):
+    if not _pipeline_ready():
+        return None
+    _VLM["pipe"].human_call(human_input=text)
+    return dict(_VLM["pipe"].traversability_preferences)
+
+
+def _vlm_register_raw():
+    if not _pipeline_ready():
+        return None
+    _VLM["pipe"].register_scene()
+    return None
+
+
+def _gpu_decorate(fn):
+    if spaces is not None:
+        return spaces.GPU(duration=120)(fn)
+    return fn
+
+
+vlm_step = _gpu_decorate(_vlm_step_raw)
+vlm_human = _gpu_decorate(_vlm_human_raw)
+vlm_register = _gpu_decorate(_vlm_register_raw)
+
+
 # ---------------------------------------------------------------------------
 # Main streaming worker. Restarted on "Go / Reset" and on "Resume".
 # ---------------------------------------------------------------------------
@@ -343,13 +414,6 @@ def run_evaluation(video_file, pref_json, sim_thresh, uncert_thresh,
         if ANYTRAVERSE_AVAILABLE:
             yield initial_render(
                 "🔄 Building AnyTraverse pipeline (first run may download models)…")
-            session.pipeline = build_pipeline_from_paper(
-                init_traversabilty_preferences=session.preferences,
-                ref_scene_similarity_threshold=float(sim_thresh),
-                roi_uncertainty_threshold=float(uncert_thresh),
-                roi_x_bounds=(float(rx_min), float(rx_max)),
-                roi_y_bounds=(float(ry_min), float(ry_max)),
-            )
         else:
             session.pipeline = None
 
@@ -373,10 +437,7 @@ def run_evaluation(video_file, pref_json, sim_thresh, uncert_thresh,
                                          session.fps, (session.vw * 2, session.vh * 2))
         session.frame_idx = 0
 
-    box = bounds_box(float(rx_min), float(rx_max),
-                     float(ry_min), float(ry_max), session.vw, session.vh)
     skip = session.skip
-    last_state = None
 
     try:
         while session.cap.isOpened():
@@ -423,26 +484,26 @@ def run_evaluation(video_file, pref_json, sim_thresh, uncert_thresh,
                         session.uncert_thresh), df_table(), video=None)
                 return
 
-            # -------- read + process a display frame --------------------------------
+            # -------- read the next display frame ------------------------------
             t0 = time.time()
             ret, frame_bgr = session.cap.read()
             if not ret:
                 break
             session.frame_idx += 1
-            rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
-
-            run_infer = ((session.frame_idx - 1) % skip == 0) or (last_state is None)
-            if run_infer:
-                if ANYTRAVERSE_AVAILABLE and session.pipeline is not None:
-                    st = session.pipeline.step(image=PILImage.fromarray(rgb))
-                    p = unpack_state(st, frame_bgr)
-                else:
-                    p = _simulate_state(frame_bgr, session.preferences,
-                                        session.uncert_thresh, session.frame_idx)
-                last_state = p
+            # Frame skipping: pass every k-th frame to anytraverse; the frames in
+            # between are never shown in the UI nor written to the output video.
+            if skip > 1 and (session.frame_idx - 1) % skip != 0:
+                continue
+
+            if ANYTRAVERSE_AVAILABLE:
+                p = vlm_step(frame_bgr, session.preferences, session.sim_thresh,
+                             session.uncert_thresh,
+                             (float(rx_min), float(rx_max)),
+                             (float(ry_min), float(ry_max)))
+                _set_pipe()
             else:
-                p = dict(last_state) if last_state else seed_state(frame_bgr, box)
-                p["raw_bgr"] = frame_bgr
+                p = _simulate_state(frame_bgr, session.preferences,
+                                    session.uncert_thresh, session.frame_idx)
             session.traversal = p["state"]
 
             fps = round(1.0 / max(time.time() - t0, 1e-3), 1)
@@ -465,8 +526,7 @@ def run_evaluation(video_file, pref_json, sim_thresh, uncert_thresh,
             if session.writer is not None:
                 session.writer.write(cv2.cvtColor(grid, cv2.COLOR_RGB2BGR))
 
-            status = f"Frame {session.frame_idx} · state **`{lbl}`**" + (
-                "" if run_infer else " · (inference skipped, reusing last maps)")
+            status = f"Frame {session.frame_idx} · state **`{lbl}`**"
 
             yield render(
                 grid, status, False, attn, session.frame_idx, skip, lbl,
@@ -520,11 +580,14 @@ def handle_operator_resume(operator_text):
     text = (operator_text or "").strip()
     if session.pipeline is not None:
         if text and text.lower() != "ok":
-            session.pipeline.human_call(human_input=text)
-            session.preferences = dict(session.pipeline.traversability_preferences)
-            msg = f"✅ Applied operator τ update `{text}` — resuming."
+            prefs = vlm_human(text)
+            if prefs:
+                session.preferences = prefs
+                msg = f"✅ Applied operator τ update `{text}` — resuming."
+            else:
+                msg = f"✅ (pipeline not built yet) resuming with `{text}`."
         else:
-            session.pipeline.register_scene()
+            vlm_register()
             msg = "✅ Scene registered (no τ change) — resuming."
     else:
         if text and text.lower() != "ok":
@@ -551,7 +614,7 @@ def simulate_hoc():
 
 def live_pipeline_update(sim, unc, rxmin, rxmax, rymin, rymax):
     """Apply threshold / ROI changes to the running pipeline object on the fly."""
-    pipe = session.pipeline
+    pipe = _VLM["pipe"]
     if pipe is not None:
         try:
             pipe._threshold.ref_scene_similarity = float(sim)
@@ -573,7 +636,7 @@ def live_pipeline_update(sim, unc, rxmin, rxmax, rymin, rymax):
 MONO = [gr.themes.GoogleFont("IBM Plex Mono"), "DejaVu Sans Mono", "monospace"]
 
 THEME = gr.themes.Base(
-    primary_hue=gr.themes.colors.slate,
+    primary_hue=gr.themes.colors.blue,
     secondary_hue=gr.themes.colors.gray,
     neutral_hue=gr.themes.colors.gray,
     font=MONO,
@@ -581,28 +644,36 @@ THEME = gr.themes.Base(
     radius_size=gr.themes.sizes.radius_sm,
     spacing_size=gr.themes.sizes.spacing_sm,
 ).set(
-    body_background_fill="#0e1013",
-    body_text_color="#d7dce4",
-    block_background_fill="#141920",
-    block_border_color="#242b36",
-    block_title_background_fill="#0e1013",
-    block_title_text_color="#9fb0c3",
-    input_background_fill="#0e1116",
-    input_border_color="#2a3240",
+    body_background_fill="#f6f7f9",
+    body_text_color="#1f2937",
+    block_background_fill="#ffffff",
+    block_border_color="#e2e8f0",
+    block_title_background_fill="#f1f5f9",
+    block_title_text_color="#475569",
+    input_background_fill="#ffffff",
+    input_border_color="#cbd5e1",
     button_primary_background_fill="#1f6feb",
     button_primary_background_fill_hover="#2f7bf5",
     button_primary_text_color="#ffffff",
-    button_secondary_background_fill="#1c232d",
-    button_secondary_text_color="#c7d2de",
+    button_secondary_background_fill="#eef2f6",
+    button_secondary_text_color="#334155",
 )
 
 CUSTOM_CSS = """
+html, body { color-scheme: light; }
+.gradio-container {
+    max-width: 1320px !important;
+    padding: 8px 16px 16px !important;
+}
 .prose h1, .prose h2, .prose h3, .prose p, .prose li, .prose code {
     font-family: 'IBM Plex Mono', 'DejaVu Sans Mono', monospace;
 }
 :root { --body-font: 'IBM Plex Mono', 'DejaVu Sans Mono', monospace; }
 footer { display: none !important; }
 #status-banner { border-left: 4px solid #1f6feb; padding-left: 12px; }
+.blocks-wrap .wrap { row-gap: 6px !important; }
+.block { margin-bottom: 6px !important; }
+.back { background: #f6f7f9; }
 """
 
 with gr.Blocks(title="AnyTraverse Studio") as demo:
@@ -613,9 +684,9 @@ with gr.Blocks(title="AnyTraverse Studio") as demo:
             # -------- main view --------
             live_view = gr.Image(label="Raw+ROI (TL) | Traversability (TR) | "
                                         "Uncertainty (BL) | ROI crop (BR)",
-                                 height=360)
+                                 height=300)
             attn_view = gr.Image(label="Attention maps (all prompts) — live only",
-                                 height=120)
+                                 height=160)
             status_banner = gr.Markdown(
                 "### Status: ready — upload a video and press ▶️ Go.",
                 elem_id="status-banner")
@@ -624,7 +695,7 @@ with gr.Blocks(title="AnyTraverse Studio") as demo:
                 live_plot = gr.LinePlot(x="Frame", y=["ROI Trav", "ROI Unc"],
                                         title="Live ROI Metrics (ROI Trav & ROI "
                                               "Uncert vs threshold)",
-                                        height=260, )
+                                        height=180, )
             metric_bars = gr.HTML(value=bars_html(0.0, 0.0, session.uncert_thresh),
                                   label="ROI Score Gauges")
 
@@ -632,8 +703,12 @@ with gr.Blocks(title="AnyTraverse Studio") as demo:
             # -------- Controls --------
             video_in = gr.File(label="📹 Upload Off-Road Video (.mp4, .mov, .avi)",
                                file_count="single")
-            pref_input = gr.Code(value=json.dumps(session.preferences, indent=2),
-                                 language="json", label="Traversability Preferences (τ)")
+            pref_input = gr.Code(
+                value="{}",
+                language="json",
+                lines=3,
+                label="Traversability Preferences (τ, JSON) — leave {} for defaults",
+            )
             with gr.Row():
                 sim_thresh = gr.Slider(0.05, 1.0, value=0.8, step=0.05,
                                        label="Ref Scene Sim. Threshold")
@@ -704,8 +779,9 @@ with gr.Blocks(title="AnyTraverse Studio") as demo:
 
 
 if __name__ == "__main__":
+    on_spaces = bool(os.getenv("SPACE_ID") or os.getenv("HF_SPACE"))
     demo.queue().launch(
-        share=True, theme=THEME, css=CUSTOM_CSS,
+        share=not on_spaces, theme=THEME, css=CUSTOM_CSS,
         allowed_paths=["."],
         server_name="0.0.0.0",
     )

requirements.txt

@@ -2,11 +2,11 @@ accelerate>=1.14.0
 anytraverse>=1.0.9
 einops>=0.8.2
 ffmpeg>=1.4
-gradio==6.22.0
+gradio>=6.22.0
 imageio-ffmpeg>=0.6.0
 opencv-python>=5.0.0.93
 pandas>=3.0.5
 pillow>=12.3.0
 torch>=2.13.0
 torchvision>=0.28.0
-transformers>=5.14.1
+transformers>=5.14.1