m3/go2z1-grasp-gr00t-vision-v2

GR00T N1.7-3B Z1 Grasp Vision Fine-tune (V2)

NVIDIA GR00T N1.7-3B vision-language-action model, fine-tuned on 5000 scripted Z1 arm grasp episodes WITH real rendered RGB video at 224×224, dual cameras (base_cam on Go2 trunk + wrist_cam on Z1 link06).

This is the V2 (vision-conditioned) successor of m3/go2z1-grasp-gr00t-n17-v1 — the projector + diffusion-model + visual encoder are all fine-tuned (not just frozen), so the policy can attend to actual scene appearance instead of relying only on language + proprioception.

Highlights

• Base: nvidia/GR00T-N1.7-3B (3.14 B params, AlternateVLDiT diffusion head)
• Tuning: --tune-visual --tune-projector --tune-diffusion-model --no-tune-llm
• 30 000 fine-tuning steps · global batch size 16 · ~48.6 hours on a single RTX PRO 6000 Blackwell (96 GB)
• Final train_loss = 0.1006 (down from ~10 at step 0)
• Embodiment tag: EmbodimentTag.NEW_EMBODIMENT (custom modality config)
• Action: z1_eef_delta (6) + z1_gripper (2) — relative joint deltas

Files

• model-{1..3}-of-3.safetensors — sharded model weights (~6 GB total)
• model.safetensors.index.json — shard index
• config.json, embodiment_id.json, processor_config.json, experiment_cfg/ — model & processor config
• statistics.json — dataset action / state normalization stats

(Optimizer / scheduler / trainer_state files are excluded — this checkpoint is inference-ready, not for resuming training.)

Usage

import sys, numpy as np, torch
sys.path.insert(0, "/path/to/Isaac-GR00T")
import modality_config_z1  # registers EmbodimentTag.NEW_EMBODIMENT

from gr00t.policy import Gr00tPolicy
from gr00t.data.embodiment_tags import EmbodimentTag

policy = Gr00tPolicy(
    embodiment_tag=EmbodimentTag.NEW_EMBODIMENT,
    model_path="m3/go2z1-grasp-gr00t-vision-v2",
    device=0,
    strict=False,
)

video_keys = policy.modality_configs["video"].modality_keys  # ["base_cam", "wrist_cam"]

obs = {
    "state":    {"z1_joint_pos": z1_joint_pos[None, None, :]},          # (1, 1, 6)
    "video":    {vk: rgb_image[None, None, ...]                          # (1, 1, 224, 224, 3) uint8
                 for vk, rgb_image in [("base_cam", base_cam_frame),
                                       ("wrist_cam", wrist_cam_frame)]},
    "language": {"annotation.human.task_description": [["pick up the cube"]]},
}
action, _ = policy.get_action(obs)
# action keys: 'z1_eef_delta' (6), 'z1_gripper' (2)

For a full closed-loop demo with autoregressive inference + PhysX-constraint gripper inside warehouse.usd, see stage4_joint_eval/gr00t_warehouse_physical_grasp.py.

Training data

Trained on real rendered video (V1 used dummy zero arrays):

📦 Dataset: m3/go2z1-grasp-vision-v2 — LeRobot v2.1, 5000 episodes × 80 frames × 2 cameras × 224×224, h264-encoded mp4, ~140 MB compressed

The dataset was generated entirely in Isaac Sim 6.0.0.0 / Isaac Lab using a 3-phase scripted IK controller (reach → grasp → lift+retract) with random per-episode cube positioning. See the dataset card for full schema, statistics, and generation pipeline details.

Training command

python -m gr00t.experiment.launch_finetune \
  --base-model-path nvidia/GR00T-N1.7-3B \
  --dataset-path /path/to/go2z1-grasp-vision-v2 \
  --embodiment-tag new_embodiment \
  --modality-config-path go2_z1_warehouse/stage3_gr00t_finetune/modality_config_z1.py \
  --tune-visual --tune-projector --tune-diffusion-model --no-tune-llm \
  --max-steps 30000 --global-batch-size 16 \
  --shard-size 64 --dataloader-num-workers 2 \
  --save-steps 1000 --save-total-limit 3

Source: v2_pipeline/gr00t_train_vision.py

Predecessor (V1, state-only)

• m3/go2z1-grasp-gr00t-n17-v1 — same task, but trained without visual input (dummy zero-array video). Use V2 instead unless you specifically need the state-only baseline for ablation comparison.

Citation

@misc{go2z1-grasp-gr00t-vision-v2,
  title  = {GR00T N1.7-3B Z1 Grasp Vision Fine-tune V2 (vision-conditioned)},
  author = {m3},
  year   = {2026},
  url    = {https://huggingface.co/m3/go2z1-grasp-gr00t-vision-v2}
}