--- license: apache-2.0 base_model: xpuenabler/MolmoAct2-LIBERO-LP-VTP-SD tags: - molmoact2 - vla - robotics - libero - pruning - layer-prune - vision-token-prune - grid-sampler - qnn - qualcomm - htp - hexagon - npu - edge pipeline_tag: robotics --- # MolmoAct2-LIBERO LP-VTP-SD — QNN / HTP context binaries Offline-compiled **Qualcomm QNN (HTP) context binaries** for the **pruned** MolmoAct2 policy [`xpuenabler/MolmoAct2-LIBERO-LP-VTP-SD`](https://huggingface.co/xpuenabler/MolmoAct2-LIBERO-LP-VTP-SD), ready to run **resident on-device** on a **Qualcomm Dragonwing IQ‑9075 (QCS9075, Hexagon v73, soc_id 77)**. The source policy stacks three compressions on `allenai/MolmoAct2-LIBERO`: | technique | change | conversion effect | |---|---|---| | **LP** (layer prune) | ViT 25L → **10L** | vision backbone = 10-layer ViT | | **VTP** (vision token prune) | Grid Token Sampler, **16 tokens/image** | prefill seq 489 → **130**; vision output `[32,2560]` (16×2 cam) | | **SD** (step decrease) | flow-matching `num_flow_timesteps=2` | action expert **2-step** decode (Euler dt=0.5) | This bundle contains the 7 prebuilt `.bin` contexts, the ONNX graphs they were built from, and the golden reference tensors used as inputs / parity references. The conversion code lives in [`nota-github/xpu-molmoact2-qnn-htp`](https://github.com/nota-github/xpu-molmoact2-qnn-htp) (`scripts_lp_vtp_sd/`, branch `support-LP_VTP_SD`, PR #4). --- ## What's in here ``` ctx/ 7 HTP context binaries (soc_id 77 / Hexagon v73), fp16 weights vision_socid77_archv73.bin 447 MB (ViT-10L + grid token sampler) llm_split0_socid77_archv73.bin 1821 MB llm_split1_socid77_archv73.bin 1821 MB llm_split2_socid77_archv73.bin 1821 MB llm_split3_socid77_archv73.bin 1821 MB action_context_socid77_archv73.bin 4 MB action_step_socid77_archv73.bin 1162 MB (2-step flow-matching) golden/ reference I/O (PyTorch fp32), the parity gate for every context vision_io.npz llm_split{0..3}_io.npz action_context_io.npz action_step_io.npz trace.json onnx/ per-component ONNX graphs (opset 20) that feed the DLC/context build; large weights stored as external data next to each .onnx vision/ llm_split0..3/ action_context/ action_step/ ``` --- ## Architecture — 3 components, seq = 130 ``` image ─┐ ViT-10L + Grid Sampler ┌──────────────────┐ 36-layer KV ┌──────────────────┐ text ─────────────┼───────────────────────▶ │ LLM (Qwen-ish) │ ────────────▶ │ Action Expert │─▶ actions state ─┘ 16 tok/cam (host scatter)│ single prefill │ (as context) │ flow-matching │ [1,10,7] └──────────────────┘ └──────────────────┘ 32 image tokens 1 pass, KV = OUTPUT only context ×1 + Euler step ×2 (SD) ``` - **VTP**: the vision backbone prunes each camera's pooled grid to **16 tokens** via a trained grid token sampler (`F.grid_sample`, exports to ONNX `GridSample` opset 20), so the 2-camera prompt is **130 tokens** (the original checkpoint was 489). - **LP**: the ViT is a 10-layer student (`MolmoAct2-LIBERO-ViT10L` base). - **SD**: the flow-matching action expert runs **2** host-looped Euler steps (`action_step`), not 10. - The 3.5B LLM is split into **4 layer-range contexts** (9 layers each) for the HTP per-session limit. ### Per-component I/O contract (seq = 130) | context | inputs (name: shape, f32) | outputs | |---|---|---| | `vision` | `pixel_values: [1,2,729,588]` | `image_embeds: [32,2560]` (16 tok × 2 cam) | | `llm_split{i}` | `hidden_states: [1,130,2560]`, `attention_bias: [1,1,130,130]` | `hidden_out`, `keys: [9,1,130,1024]`, `values` | | `action_context` | `keys: [36,1,130,1024]`, `values` | `k_ctx`, `v_ctx` | | `action_step` | `trajectory: [1,10,32]`, `timestep: [1]`, `k_ctx`, `v_ctx`, `encoder_attention_mask: [1,130]`, `action_dim_is_pad: [1,32]` | `next_trajectory: [1,10,32]` | Host pipeline: **vision (NPU)** → scatter 32 image embeds into the prompt scaffold → **llm_split0..3 (chain, collect 36-layer KV)** → **action_context (NPU)** → **2-step Euler with action_step** → de-normalize. --- ## Validated parity (host, QNN CPU backend vs PyTorch fp32) Each context was verified device-free on the QNN **CPU** backend against the PyTorch golden. **Cosine is the correctness gate** (≥ 0.9999); `within_tol` is a known massive-activation fp artifact (LLM hidden_states/KV span ~1e4) and is informational. | context | output | cosine | |---|---|---| | `vision` | image_embeds | **0.99843835** (see note) | | `llm_split0` | hidden_out / keys / values | 1.0 / 0.99999999 / 0.99999985 | | `llm_split1` | hidden_out / keys / values | 1.0 / 1.0 / 1.0 | | `llm_split2` | hidden_out / keys / values | 1.0 / 1.0 / 1.0 | | `llm_split3` | hidden_out / keys / values | 1.0 / 1.0 / 1.0 | | `action_context` | k_ctx / v_ctx | 1.0 / 0.99999995 | | `action_step` | next_trajectory | 1.0 | ONNX-vs-PyTorch parity (ORT CPU) was tight for every component (vision incl. GridSample: cosine 0.99999999). **Note on `vision`:** the QNN CPU-backend DLC lands at cosine **0.9984** — below the 0.99999 the non-grid pooling reference hits — attributable to the grid sampler's `F.grid_sample(align_corners=False, bilinear)` interpolation on the QNN `GridSample` op. The context binary compiles and loads; a GridSample-attribute/opset match + on-device fp16 end-to-end check are follow-ups. --- ## Build provenance - Source policy: `xpuenabler/MolmoAct2-LIBERO-LP-VTP-SD` (LeRobot checkpoint; base `MolmoAct2-LIBERO-ViT10L`). - Loaded via the `nota-github/xpu-lerobot @ feat/grid-sampler-molmoact2` fork (implements `grid_token_sampler`); one real LIBERO inference captured golden I/O at the four boundaries. - Toolchain: QAIRT **2.47.0**, opset 20, float DLC (no quant), offline HTP context-binary generation for soc_id 77 / dsp_arch v73, `O3`, weight-sharing per multi-graph context. - Path: `PyTorch → ONNX (per component) → qairt-converter DLC → CPU-backend parity → qnn-context-binary-generator (.bin)`. See `docs/lp_vtp_sd_qnn.md` in the conversion repo.