cpsu04/smolvlm2-500m-robo4d-4d

SmolVLM2-500M Robo4D-4D (merged)

SmolVLM2-500M-Video-Instruct finetuned on Robo4D direction VQA — a 27.5k-pair video MCQ dataset built from DROID robot trajectories, teaching 4D spatial reasoning (object motion direction from video, in both camera-view and robot-body frames). LoRA weights are merged into the base model; drop-in replacement for the base checkpoint.

Training data — Robo4D VQA v2

• Source: DROID ↔ LeRobot aligned trajectories (signal-verified UUID alignment)
• Direction labels from end-effector displacement during the grasp→release window (last-grasp detection; gripper 0=open ~1=closed)
• Camera-frame projection via XYZ-Euler extrinsics (standard OpenCV axes: col0=right, col1=down, col2=away)
• 2×2 question design per trajectory: {ext1, ext2 video} × {camera-view, robot-view}, 4-choice MCQ; camera-view answers differ across viewpoints, robot-view answers are view-invariant
• Train split: 24,796 pairs (episode-disjoint from test)

Finetune settings

Item	Value
Base	HuggingFaceTB/SmolVLM2-500M-Video-Instruct
Method	LoRA → merged
r / alpha / dropout	16 / 32 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (all layers)
Optimizer / LR	AdamW / 2e-4
Batch	1 × grad_accum 16 (effective 16)
Epochs	1
Loss	next-token CE masked to the answer letter

Results (Robo4D test split, 2,756 pairs, MCQ likelihood scoring)

Metric	Zero-shot base	This model
Overall accuracy	38.9%	65.9%
Robot-view accuracy	39.1%	68.9%
Camera-view accuracy	38.6%	62.9%
Joint view-invariance success	37.3%	66.2%

Intended use

Backbone for spatial-reasoning-aware VLAs (e.g., SmolVLA --policy.vlm_model_name=<this repo>), or as a video direction-QA model.