nielsr HF Staff commited on
Commit
1a33950
·
verified ·
1 Parent(s): 6b37ba4

Add model card with metadata and usage example

Browse files

This PR adds a model card with essential metadata and a usage example based on the information provided in the paper and the GitHub README. The `pipeline_tag` is set to `robotics`, reflecting the model's application in robotics. The `library_name` is set to `transformers` based on the provided code snippet. A permissive MIT license is assumed, pending confirmation from the authors. The usage example provides a minimal working example for generating an action chunk.

Files changed (1) hide show
  1. README.md +76 -0
README.md ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: robotics
3
+ library_name: transformers
4
+ license: mit
5
+ ---
6
+
7
+ # Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
8
+
9
+ This repository contains the OpenVLA-OFT model, as described in [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://arxiv.org/abs/2502.19645). OpenVLA-OFT significantly improves upon the base OpenVLA model by incorporating optimized fine-tuning techniques.
10
+
11
+ Project Page: https://openvla-oft.github.io/
12
+
13
+ Code: https://github.com/openvla-oft/openvla-oft
14
+
15
+
16
+ ## Quick Start
17
+
18
+ This example demonstrates generating an action chunk using a pretrained OpenVLA-OFT checkpoint. Ensure you have set up the conda environment as described in the Github README.
19
+
20
+ ```python
21
+ import pickle
22
+ from experiments.robot.libero.run_libero_eval import GenerateConfig
23
+ from experiments.robot.openvla_utils import get_action_head, get_processor, get_proprio_projector, get_vla, get_vla_action
24
+ from prismatic.vla.constants import NUM_ACTIONS_CHUNK, PROPRIO_DIM
25
+
26
+ # Instantiate config (see class GenerateConfig in experiments/robot/libero/run_libero_eval.py for definitions)
27
+ cfg = GenerateConfig(
28
+ pretrained_checkpoint = "moojink/openvla-7b-oft-finetuned-libero-spatial",
29
+ use_l1_regression = True,
30
+ use_diffusion = False,
31
+ use_film = False,
32
+ num_images_in_input = 2,
33
+ use_proprio = True,
34
+ load_in_8bit = False,
35
+ load_in_4bit = False,
36
+ center_crop = True,
37
+ num_open_loop_steps = NUM_ACTIONS_CHUNK,
38
+ unnorm_key = "libero_spatial_no_noops",
39
+ )
40
+
41
+ # Load OpenVLA-OFT policy and inputs processor
42
+ vla = get_vla(cfg)
43
+ processor = get_processor(cfg)
44
+
45
+ # Load MLP action head to generate continuous actions (via L1 regression)
46
+ action_head = get_action_head(cfg, llm_dim=vla.llm_dim)
47
+
48
+ # Load proprio projector to map proprio to language embedding space
49
+ proprio_projector = get_proprio_projector(cfg, llm_dim=vla.llm_dim, proprio_dim=PROPRIO_DIM)
50
+
51
+ # Load sample observation:
52
+ # observation (dict): {
53
+ # "full_image": primary third-person image,
54
+ # "wrist_image": wrist-mounted camera image,
55
+ # "state": robot proprioceptive state,
56
+ # "task_description": task description,
57
+ # }
58
+ with open("experiments/robot/libero/sample_libero_spatial_observation.pkl", "rb") as file:
59
+ observation = pickle.load(file)
60
+
61
+ # Generate robot action chunk (sequence of future actions)
62
+ actions = get_vla_action(cfg, vla, processor, observation, observation["task_description"], action_head, proprio_projector)
63
+ print("Generated action chunk:")
64
+ for act in actions:
65
+ print(act)
66
+ ```
67
+
68
+ ## Citation
69
+
70
+ ```bibtex
71
+ @article{kim25finetuning,
72
+ title={Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success},
73
+ author={{Moo Jin} Kim and Chelsea Finn and Percy Liang},
74
+ journal = {arXiv preprint arXiv:2502.19645},
75
+ year={2025},}
76
+ ```