anjali209/openvla-7b-aloha-coffee-lora

OpenVLA-7B — ALOHA coffee (LoRA adapter)

LoRA adapter (r=32, alpha=64) for bimanual coffee-making. Load on top of the base model.

Fine-tuned from openvla/openvla-7b on a curated subset of lerobot/aloha_static_coffee — a bimanual ALOHA arm making coffee.

Read this before loading the weights

action_stats.json in this repo is required to decode the model's output. The action head emits bin indices, not joint values. Turning a bin back into a joint target needs the exact q01/q99 statistics the model was trained against, which are in that file. Weights without it are not a usable policy.

from transformers import AutoModelForVision2Seq, AutoProcessor
from peft import PeftModel
import json

REPO = "anjali209/openvla-7b-aloha-coffee-lora"
processor = AutoProcessor.from_pretrained("openvla/openvla-7b", trust_remote_code=True)
model = AutoModelForVision2Seq.from_pretrained(
    "openvla/openvla-7b", dtype="bfloat16", trust_remote_code=True)
model = PeftModel.from_pretrained(model, REPO)

# required to decode bins -> joint targets
from huggingface_hub import hf_hub_download
stats = json.load(open(hf_hub_download(REPO, "action_stats.json")))

Training

Base model	openvla/openvla-7b (7.54B params)
Trainable	79,953,920 (1.06%) — LoRA r=32, alpha=64, dropout 0.05
Data	42,006 train frames (48 episodes) / 1,831 val (2 held-out episodes)
Camera	cam_left_wrist — a single wrist (egocentric) view
Steps	5,250 = 1.00 epoch
Precision	bf16, gradient checkpointing on
Optimizer	AdamW, OneCycleLR (pct_start=0.03), grad clip 1.0
Learning rate	2e-4
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Peak GPU memory	9.4 GB
Wall time	70.4 min
Checkpoint selected on	val_action_mae (best 0.0532 @ step 3500)

Actions are 14-D joint targets, normalised per dimension with q01/q99 to [-1, 1], discretised into 256 uniform bins, and mapped onto the least-used token ids in the vocabulary (31744-31999). Loss is cross-entropy over the 14 action tokens with the prompt masked.

Splits are by episode, not by frame — 65-75% of wrist frames are visually near-static, so a frame-level split would put near-duplicates on both sides.

Limitations

Please read these before using the model for anything.

• Not validated on a robot or in simulation. All evaluation was offline and teacher-forced: the model saw a real observation every frame and its predictions drove nothing, so compounding error is not measured at all.
• It does not complete the task on held-out episodes. Scored on a milestone criterion (pod grasped -> placed -> basket grasped -> seated, in order), it reaches all four milestones but fails the ordering/hold check on both held-out episodes: the predicted gripper closes on the basket and reopens 1-2 frames later.
• The gripper command is temporally inconsistent. Each action is an independent per-frame argmax over 256 bins, with nothing tying consecutive frames together. The predicted gripper moves up to 0.48 in a single frame against a demonstration maximum of 0.036. On hardware that is a command to snap the gripper open and shut at 50 Hz.
• Large train/held-out gap. The same criterion scores 85-92% on the 48 training episodes and 0/2 on the 2 held-out ones — the model fits the demonstrations it saw and does not transfer.
• One camera of four. Only cam_left_wrist is used. Notably, the left arm handles the first sub-task while the right arm performs the one the model fails on.
• Action MAE is not a good proxy for competence here. A "repeat the previous frame's action" baseline beats this model 26x on that metric.

The practical next step this points at is constraining the gripper channel temporally at decode time (action chunking, hysteresis, or a loss term penalising frame-to-frame gripper jumps).

Provenance

Trained as part of a data-pipeline exercise covering dataset audit, labeling design, curation, evaluation design, and VLA adaptation. Training data came from a curated build: post-completion motion trimmed (~20% of frames) and a measured 5-frame video/state offset corrected before training.