---
license: apache-2.0
library_name: pytorch
tags:
  - robotics
  - vision-language-action
  - flow-matching
  - so100
  - from-scratch
---

# BeginVLA-12M

BeginVLA-12M is the project-owned model from
[Begin with VLA](https://github.com/tlysanhuo/begin-with-vla). Its vision encoder,
byte-level language encoder, multimodal fusion stack, and flow-matching action decoder are
implemented in the repository and trained end to end from random initialization.

The release is designed for learning: every model input, tensor shape, loss term, checkpoint field,
and evaluation boundary has a corresponding executable lesson in the source repository.

## Released variants

| Directory | Training stage | Intended use |
| --- | --- | --- |
| `pretrained/` | 6,072 updates on the audited 29-dataset SO-100 mix | General course, evaluation, and further post-training |
| `stacking/` | Pretrained model followed by 652 updates on 50 stacking episodes | Target-task prediction and protected replay lesson |

Each directory is a self-contained inference checkpoint with a portable real robot sample. The
approximately 49MB model weights are published; optimizer and RNG state are intentionally omitted.
Every file is covered by SHA-256 in `release_manifest.json`.

## Ten-minute quickstart

```bash
git clone https://github.com/tlysanhuo/begin-with-vla.git
cd begin-with-vla
python -m venv .venv
source .venv/bin/activate
pip install -e '.[mainline]'

begin-vla quickstart
```

The command downloads and verifies `pretrained/`, runs CPU or CUDA inference on the included real
SO-100 observation, and writes `prediction.json` plus `prediction.png` under
`outputs/quickstart/pretrained/`. It does not require the 30GB pretraining dataset.

To inspect modality dependence on the same sample:

```bash
begin-vla ablate
```

To download without running inference:

```bash
begin-vla download --variant pretrained
begin-vla download --variant stacking
```

## Architecture

| Module | Parameters | Contract |
| --- | ---: | --- |
| Vision encoder | 2,882,048 | Three steps, up to two 224x224 RGB views |
| Language encoder | 2,725,120 | UTF-8 bytes, at most 96 tokens |
| Multimodal fusion | 2,710,080 | Vision, instruction, and state history |
| Flow action decoder | 3,850,016 | Fifty future actions, up to 32 dimensions |
| Total | 12,167,264 | All parameters trained from random initialization |

The released SO-100 data uses six state and action dimensions. The larger dimensions are padded and
masked to keep the teaching interface explicit.

## Training and measured results

The pretrained variant consumed 97,152 observations in one 6,072-update epoch. On the 10,382-sample
held-out split it obtained normalized MSE 0.4120 and normalized MAE 0.4509 with ten flow integration
steps and seed 0.

For the stacking lesson, both pretrained and random-initialized models received the same 50 episodes,
652 updates, optimization configuration, and evaluation seed. The pretrained initialization reached
normalized MSE 0.6895, compared with 1.6715 from random initialization, a 58.7% reduction.

A six-condition ablation over all 10,382 held-out pretraining observations found that removing vision
increased normalized MSE by 21.6%, while replacing state with its training mean increased it by
86.6%. Keeping only the current observation increased MSE by 0.4%. Emptying the instruction increased
MSE by 4.9%, but replacing it with another real task instruction did not hurt aggregate error and the
correct instruction won on only 50.4% of samples. This checkpoint uses vision and state, but stable
task-specific language grounding is not established.

Machine-readable training contracts and results are versioned in the source repository:

- [`pretraining-result.json`](https://github.com/tlysanhuo/begin-with-vla/blob/main/results/beginvla_12m/pretraining-result.json)
- [`target-posttrain-result.json`](https://github.com/tlysanhuo/begin-with-vla/blob/main/results/beginvla_12m/target-posttrain-result.json)
- [`modality-ablation/summary.json`](https://github.com/tlysanhuo/begin-with-vla/blob/main/results/beginvla_12m/modality-ablation/summary.json)

## Data

Pretraining uses the pinned Apache-2.0
[`HuggingFaceVLA/community_dataset_v1`](https://huggingface.co/datasets/HuggingFaceVLA/community_dataset_v1)
revision documented by the source repository. The portable samples retain their dataset, episode,
frame, task, and checksum provenance.

## Limitations

- Offline action error is not robot task success.
- The stacking result is one target task with one training and sampling seed.
- Released checkpoints do not establish closed-loop simulation or physical robot success.
- The replay command is a dry run. It does not include calibrated joint limits, collision detection,
  watchdog behavior, or a hardware adapter.
- The byte language encoder learns the robot instructions in this dataset; it is not a general
  language model.

Do not send generated actions to physical hardware without robot-specific calibration, limits,
watchdog, emergency stop, and collision protection.