GR00T-N1.7-3B-Pick-Banana-v2

A fine-tuned version of nvidia/GR00T-N1.7-3B for fruit pick-and-place on a SO-ARM 101, trained on the jinseonylee/SO101_PickAndPlace_Fruit dataset.

Model Description

GR00T-N1.7 (Gr00tN1d7) is a vision-language-action (VLA) model for robot manipulation. This checkpoint is fine-tuned for a pick-and-place task on a SO-ARM 101 arm.

• Architecture: Gr00tN1d7 with Cosmos-Reason2-2B (Qwen-based) vision-language backbone + diffusion policy action head
• Base model: nvidia/GR00T-N1.7-3B
• Task: Pick and place fruit
• Robot: SO-ARM 101
• Action horizon: 16 steps
• Model dtype: bfloat16

Fine-tuning Configuration

Parameter	Value
Tuned components	Diffusion model, projector (vision encoder + LLM backbone frozen)
Training steps	13000
Global batch size	32
Learning rate	1e-4 (cosine, warmup ratio 0.05)
Weight decay	1e-5
State dropout prob	0.2
Optimizer	adamw_torch
Precision	bf16
Final training loss	~0.024 (avg train_loss 0.060)
Action representation	single_arm relative, gripper absolute
Attention	Flash Attention 2

Training Details

• Dataset: jinseonylee/SO101_PickAndPlace_Fruit — 102 episodes, ~78k frames (30fps), LeRobot v2.1 format
• Embodiment: SO-ARM 101 (new_embodiment)
• Max steps: 13000 (≈5.4 epochs over ~76.6k preloaded frames at batch 32)
• Final loss: 0.0244 at step 13000; train_runtime ~3h14m on a single GPU
• Reference: hyperparameters follow hi-space/GR00T-N1.7-3B-Pick-Orange, scaled from 6000 steps to keep a comparable epoch count for this larger dataset

Embodiment & Modalities

State inputs:

• single_arm — arm joint positions (relative)
• gripper — gripper position (absolute)

Action outputs: same as state inputs (16-step horizon)

Cameras: front, wrist

Language conditioning: annotation.human.task_description

Usage

Refer to the NVIDIA Isaac GR00T repository for full inference and deployment instructions. Open-loop evaluation:

uv run python gr00t/eval/open_loop_eval.py \
  --dataset-path <path-to>/SO101_PickAndPlace_Fruit_lerobot \
  --embodiment-tag NEW_EMBODIMENT \
  --model-path <path-to>/checkpoint-13000 \
  --traj-ids 0 --execution-horizon 16 --steps 400

Intended Use

This model is intended for SO-ARM 101 fruit pick-and-place tasks matching the training dataset's setup. It is not guaranteed to transfer zero-shot to different scenes, objects, or hardware without additional fine-tuning.

License

This model inherits the license from the base model nvidia/GR00T-N1.7-3B / nvidia/Cosmos-Reason2-2B. Please refer to NVIDIA's terms for usage restrictions.