--- license: apache-2.0 base_model: nvidia/GR00T-N1.7-3B tags: - robotics - vision-language-action - gr00t - gr00t-n1.7 - unitree-g1 - manipulation - picking - brainco-hand - diffusion - flow-matching language: - en pipeline_tag: robotics --- # GR00T N1.7 — Unitree G1 BrainCo Pick (Ego-Centric) Fine-tuned checkpoint of [nvidia/GR00T-N1.7-3B](https://huggingface.co/nvidia/GR00T-N1.7-3B) on **1,598 real-robot episodes** of tabletop pick-and-place tasks performed by a Unitree G1 humanoid equipped with BrainCo dexterous hands. This model uses a **single ego-centric head camera** (no wrist cameras) and predicts **26-DOF arm + hand joint commands** for left-arm manipulation. --- ## Model Details | Property | Value | |----------|-------| | **Base model** | nvidia/GR00T-N1.7-3B | | **Architecture** | VLM backbone (Cosmos-Reason2-2B / Qwen3-VL) + Diffusion action head (AlternateVLDiT) | | **Total parameters** | ~3B | | **Robot** | Unitree G1 humanoid + BrainCo dexterous hands | | **Camera** | 1× head stereo-left (ego-centric only) | | **State dim** | 26D (14D arm joints + 12D hand state) | | **Action dim** | 26D (14D relative arm targets + 12D absolute hand commands) | | **Action horizon** | 16 steps at 30 fps | | **Inference timesteps** | 4 (flow matching) | | **Final training loss** | 0.020 | | **Training steps** | 20,000 | | **Training hardware** | 1× NVIDIA RTX 5090 (32 GB) | --- ## Training Data **8 picking tasks, 1,598 total episodes** from the [Unitree G1 BrainCo dataset](https://huggingface.co/unitreerobotics): | Task | Episodes | Object | Notes | |------|----------|--------|-------| | GraspOreo | 201 | Oreo cookie box | Rigid box | | GraspRubiksCube | 197 | Rubik's cube | Rigid, 3D geometry | | PickApple | 200 | Apple | Soft, deformable | | PickCharger | 200 | USB charger | Small, fragile | | PickDoll | 200 | Doll | Articulated, soft | | PickDrink | 201 | Bottle / can | Cylindrical, slippery | | PickTissues | 206 | Tissue box | Light, crushable | | PickToothpaste | 193 | Toothpaste tube | Small cylinder | **Data format:** GR00T-flavored LeRobot v2.1 (per-episode HDF5-backed parquet + mp4 videos). **Data pipeline:** Raw episodes were captured as human-teleoperated demonstrations, IK-retargeted to the G1 robot via `training_data_g1brainco_newik.npz`, and converted from LeRobot v3.0 to v2.1 format with the preprocessing script at `examples/G1_Brainco/preprocess_brainco_pick.py` in the [Isaac GR00T repo](https://github.com/NVIDIA/Isaac-GR00T). --- ## Training Configuration ```yaml base_model: nvidia/GR00T-N1.7-3B embodiment_tag: NEW_EMBODIMENT max_steps: 20000 learning_rate: 1e-4 warmup_ratio: 0.05 weight_decay: 1e-5 optimizer: paged_adamw_8bit global_batch_size: 1 gradient_accumulation: 64 # effective batch = 64 gradient_checkpointing: true num_shards_per_epoch: 2000 shard_size: 512 episode_sampling_rate: 0.1 color_jitter: brightness=0.3, contrast=0.4, saturation=0.5, hue=0.08 tune_llm: false tune_visual: false tune_projector: true tune_diffusion_model: true ``` ### Loss Curve | Step | Loss | LR | |------|------|----| | 10 | 1.230 | 9e-7 | | 1000 | 0.0942 | 9.99e-5 | | 2000 | 0.0676 | 9.93e-5 | | 4000 | 0.0559 | 9.40e-5 | | 6000 | 0.0431 | 8.39e-5 | | 8000 | 0.0328 | 7.01e-5 | | 10000 | 0.0406 | 5.41e-5 | | 14000 | 0.0280 | 2.27e-5 | | 18000 | 0.0302 | 2.71e-6 | | **20000** | **0.0200** | ~0 | --- ## Modality Configuration ```python # g1_brainco_config_pick_ego.py config = { "video": ModalityConfig( delta_indices=[0], modality_keys=["observation.images.head_stereo_left"], # ego-centric only ), "state": ModalityConfig( delta_indices=[0], modality_keys=[ "observation.state.arm_q", # 14D arm joint positions (rad) "observation.state.hand_state", # 12D BrainCo finger state ], ), "action": ModalityConfig( delta_indices=list(range(16)), # 16-step prediction horizon modality_keys=[ "action.arm_q", # 14D relative arm joint targets "action.hand_cmd", # 12D absolute hand commands ], action_configs=[ ActionConfig(rep=ActionRepresentation.RELATIVE, type=ActionType.NON_EEF, state_key="observation.state.arm_q"), ActionConfig(rep=ActionRepresentation.ABSOLUTE, type=ActionType.NON_EEF), ], ), "language": ModalityConfig( delta_indices=[0], modality_keys=["annotation.human.task_description"], ), } ``` --- ## Quick Start ### Install ```bash git clone https://github.com/NVIDIA/Isaac-GR00T cd Isaac-GR00T uv sync --all-extras # or: pip install -e ".[dev]" ``` ### Download the checkpoint ```python from huggingface_hub import snapshot_download model_path = snapshot_download("JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick") ``` ### Run inference server ```bash python gr00t/eval/run_gr00t_server.py \ --model-path JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick \ --embodiment-tag NEW_EMBODIMENT \ --modality-config-path modality_config.py \ --denoising-steps 4 ``` ### Open-loop evaluation ```bash python gr00t/eval/open_loop_eval.py \ --dataset-path /path/to/brainco_pick/G1_Brainco_PickApple_Dataset \ --embodiment-tag NEW_EMBODIMENT \ --model-path JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick \ --modality-config-path modality_config.py ``` ### Python inference ```python import numpy as np from gr00t.model.gr00t_n1d7.setup import Gr00tN1d7Pipeline pipeline = Gr00tN1d7Pipeline.from_pretrained( "JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick", modality_config_path="modality_config.py", ) # Single inference call output = pipeline.get_action( observation={ # Head camera frame: (H, W, 3) uint8 "observation.images.head_stereo_left": head_frame, # 14D arm joint positions (radians) "observation.state.arm_q": arm_q, # 12D BrainCo finger state "observation.state.hand_state": hand_state, }, language_instruction="Pick up the apple and place it in the bowl", ) # output["action.arm_q"] → (16, 14) relative arm delta targets # output["action.hand_cmd"] → (16, 12) absolute hand commands ``` --- ## Deploying on Real Unitree G1 + BrainCo Hands ### Hardware requirements - Unitree G1 humanoid robot - BrainCo dexterous hands (6-DOF per hand) - Head stereo-left camera (640×480 RGB, 30 fps) - Inference host: ≥16 GB VRAM GPU (RTX 4090 / H100 / Jetson Thor) ### Action space mapping ``` output["action.arm_q"] (16, 14) — relative delta from current arm_q [0:7] left arm : shoulder_pitch, shoulder_roll, shoulder_yaw, elbow_pitch, elbow_roll, wrist_pitch, wrist_yaw [7:14] right arm : (same order; right arm stationary in training data) output["action.hand_cmd"] (16, 12) — absolute finger positions [0, 1] [0:6] left hand : thumb, index, middle, ring, pinky + spread [6:12] right hand: (same order) ``` ### Control loop (30 Hz skeleton) ```python import numpy as np from gr00t.model.gr00t_n1d7.setup import Gr00tN1d7Pipeline pipeline = Gr00tN1d7Pipeline.from_pretrained( "JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick", modality_config_path="modality_config.py", ) action_buffer = None action_step = 0 HORIZON = 16 while True: # 1. Read sensors head_img = robot.get_camera("head_stereo_left") # (480, 640, 3) uint8 arm_q = robot.get_arm_joint_positions() # (14,) radians hand_state = robot.get_hand_state() # (12,) [0,1] # 2. Infer (every HORIZON steps to amortise diffusion cost) if action_step == 0 or action_buffer is None: output = pipeline.get_action( observation={ "observation.images.head_stereo_left": head_img, "observation.state.arm_q": arm_q, "observation.state.hand_state": hand_state, }, language_instruction="Pick up the apple", ) action_buffer = output action_step = 0 # 3. Execute current step delta_arm = action_buffer["action.arm_q"][action_step] # (14,) hand_cmd = action_buffer["action.hand_cmd"][action_step] # (12,) target_arm = arm_q + delta_arm target_arm = np.clip(target_arm, -np.pi, np.pi) robot.set_arm_joints(target_arm, kp=15, kd=0.5) robot.set_hand_joints(hand_cmd) action_step = (action_step + 1) % HORIZON time.sleep(1 / 30.0) ``` --- ## Files in this Repository | File | Description | |------|-------------| | `config.json` | Model architecture config (GR00T N1.7) | | `model-0000*-of-00003.safetensors` | Fine-tuned weights (sharded, ~12 GB total) | | `model.safetensors.index.json` | Shard index | | `statistics.json` | Per-embodiment normalization stats (mean/std for state, action) | | `embodiment_id.json` | Embodiment ID table used during training | | `modality_config.py` | Modality config (ego-only camera, 26D state/action) — **required for inference** | | `trainer_state.json` | Full training log (loss per step) | --- ## Reproducing Training 1. **Clone the repo** ```bash git clone https://github.com/NVIDIA/Isaac-GR00T cd Isaac-GR00T uv sync --all-extras ``` 2. **Download and preprocess the datasets** ```bash python examples/G1_Brainco/preprocess_brainco_pick.py # Downloads 8 datasets from unitreerobotics HF org → dataset_g1/brainco_pick/ ``` 3. **Launch fine-tuning** ```bash bash examples/G1_Brainco/finetune_g1_brainco_pick.sh ``` Hardware notes for 32 GB VRAM (RTX 5090): ``` PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True --gradient_checkpointing --optim paged_adamw_8bit --global_batch_size 1 --gradient_accumulation_steps 64 # Peak ~26 GB, ~5.8 s/step, 20 000 steps ≈ 32 h ``` --- ## Limitations - Trained on **left-arm only** demonstrations; right-arm commands are zero throughout. - Single ego-centric head camera — no wrist-camera feedback. Fine grasping of very small objects may be less precise than a 3-camera setup. - Data collected in a specific lab tabletop environment; real-world lighting and clutter variation may affect performance. - Not trained for whole-body or locomotion control. --- ## Citation If you use this checkpoint, please cite the base GR00T N1.7 model: ```bibtex @misc{gr00tn17_2025, title = {Isaac GR00T N1.7: Open Foundation Model for Generalized Humanoid Control}, author = {NVIDIA}, year = {2025}, url = {https://huggingface.co/nvidia/GR00T-N1.7-3B} } ``` And the Unitree G1 BrainCo datasets: ```bibtex @misc{unitree_brainco_2025, title = {Unitree G1 BrainCo Manipulation Dataset}, author = {Unitree Robotics}, year = {2025}, url = {https://huggingface.co/unitreerobotics} } ``` --- ## License Weights are released under the same [Apache 2.0](LICENSE) license as the base GR00T N1.7 model.