README.md

---
license: apache-2.0
base_model: allenai/Molmo2-4B
pipeline_tag: image-text-to-text
tags:
  - molmo2
  - adacodec
  - video
  - long-video
  - stage1
  - stage2
---

# Molmo2 Codec V6 — Stage-2 step 100

This is the evaluated V6 AdaCodec-on-Molmo2 research checkpoint. It packages
the Stage-2 model, its matching Stage-1 P-tokenizer, the fixed motion-cost
calibration, the exact saved training config, source provenance, and a paired
six-benchmark evaluation summary.

This is **not** a standalone `transformers` checkpoint. It must be loaded with
the Molmo2 Codec code and its `lmms-eval` adapter.

## Checkpoints

| File | Purpose | SHA256 |
|---|---|---|
| `weights/stage2_step100_consolidated.pt` | Consolidated Molmo2 + trained Stage-2 parameters | `141467de9a28a53d6ebb87ef8bcf1dbf191880d39b64dd0bebc3728baaf79677` |
| `weights/stage1_ptokenizer_step20000.pt` | V6+HEX Stage-1 P-tokenizer | `10fc7d0460199581d9a7f1677c79a6a519adc55184c4ba14cd44f50a4bf5eec9` |

## Training contract

- Base: Molmo2-4B-SFT lineage.
- Stage-1: V6 P-tokenizer, RGB HEX motion search, step 20,000.
- Stage-2 checkpoint: step 100 from the formal 32,768-token run.
- Training visual ceiling: 16,384 tokens; text ceiling: 15,872 tokens.
- Candidate timeline: full-span sampling, 2 FPS, at most 2,048 source frames.
- Mixture: 40% long codec caption, 10% normal codec video, 37.5% native
  video, and 12.5% native image replay.
- Stage-2 trains the LLM, P-ViT, and P-connector. The I-frame visual path stays
  on the native Molmo2 representation.
- The durable Stage-2 GOP cache uses the explicit `fast_gray2` motion-search
  contract and its matching gamma artifact. The packaged Stage-1 weights were
  trained with RGB HEX; this run intentionally adapts the P branch in Stage-2.

The complete serialized configuration is in
`config/stage2_step100_config.yaml`.

## Paired evaluation

Each row uses the same frozen 200 examples for Dense and Codec. Evaluation
uses full-span 2 FPS sampling, a 2,048-frame candidate timeline, 8,192 visual
tokens, and a 16,384-token sequence. The checked machine-readable result is
`eval/paired_summary_n200.json`.

| Benchmark | Dense | Codec | Codec − Dense |
|---|---:|---:|---:|
| Video-MME | 70.0 | 69.0 | -1.0 |
| LVBench | 48.0 | 49.5 | +1.5 |
| MVBench | 75.5 | 68.5 | -7.0 |
| TempCompass | 69.0 | 68.5 | -0.5 |
| PerceptionTest | 82.0 | 83.0 | +1.0 |
| NExT-QA | 84.0 | 83.5 | -0.5 |

The checkpoint is close to Dense on five of six paired subsets, but the
MVBench regression is material and remains an open limitation. These are
bounded `n=200` diagnostics, not full benchmark submissions.

## Code and loading

- Main code: <https://github.com/weikaih04/molmo2-codec>, branch `adacodec`.
- Main release commits: `69f20d99`, `63874827`, `ead01f2e`.
- `lmms-eval` adapter commit: `6fda918` in the local Molmo2 evaluation fork.
- Example launcher: `examples/molmo2_codec/run_smoke.sh` in that fork.

The adapter expects paths to both packaged weight files and the gamma artifact.
It invokes the production Molmo2 preprocessing, timestamp, position, packing,
GOP-selection, and codec inference code instead of reimplementing them.

## Provenance

`provenance/SOURCE_SNAPSHOT.txt` records the exact source state captured by the
formal training job, including the pre-commit working-tree digest and the
post-requeue decode fix. `provenance/SOURCE_FILE_SHA256SUMS` records hashes of
the evaluation-critical source files. The three release commits above organize
that working tree and add the final regression tests; they should not be
mistaken for a claim that the model was retrained after code cleanup.

## Limitations

- Research checkpoint after only 100 Stage-2 optimizer steps.
- Requires the custom code path; it is not directly loadable through
  `AutoModel.from_pretrained`.
- The evaluation sample size is 200 per benchmark.
- Motion cache and gamma artifacts are contract-bound. Do not silently mix
  RGB HEX and `fast_gray2` cache namespaces.
- The upstream base-model license and usage restrictions continue to apply.