---
base_model: Qwen/Qwen3-VL-8B-Instruct
library_name: transformers
pipeline_tag: image-text-to-text
license: apache-2.0
language:
  - en
tags:
  - qwen3-vl
  - grpo
  - rlvr
  - verl
  - multimodal
  - math-reasoning
---

# Qwen3-VL-8B · OMR · GRPO baseline (cold-start)

RLVR post-training of **Qwen/Qwen3-VL-8B-Instruct** on the **OMR** (OpenMMReasoner, math/visual-reasoning) data with fully-async **GRPO**, cold-started from the stock 8B-Instruct (no exploration, no warm-start). This is `experiment_name=grpo_omr_4node_full_v1_8b_base_perf`, **`global_step_25`** (the keeper).

**This is the OMR cold-start GRPO baseline. It peaks at OMR overall-6 val accuracy `0.668` @ step 25, then collapses to `0.499` by step 125** — cold-start RL on 8B-OMR is unstable without exploration. The keeper is `global_step_25`, the only checkpoint before degradation began. (The `omr-8b-grpo-ppexplore` sibling fixes this collapse and reaches 0.714.)

## Results

OMR 6-image inline validation (mmmu val, mathvista testmini, mathverse testmini Text-Dominant, wemath testmini, charxiv reasoning-qa, dynamath test). `overall-6` = unweighted mean. Metric = accuracy.

Keeper = **`global_step_25`** (peak):

| metric | overall-6 | mmmu | mathvista | mathverse | wemath | charxiv | dynamath |
|---|---|---|---|---|---|---|---|
| **baseline peak @25** | **0.6681** | 0.6311 | 0.8019 | 0.8404 | 0.7437 | 0.3930 | 0.5986 |
| stock Qwen3-VL-8B ckpt-0 | 0.659 | 0.629 | 0.811 | 0.824 | 0.723 | 0.396 | 0.570 |
| baseline @125 (collapse) | 0.4994 | 0.5021 | 0.6796 | 0.6995 | 0.6184 | 0.2010 | 0.2958 |

Full trajectory (overall-6): **0.668 @25** → 0.644 @50 → 0.620 @75 → 0.620 @100 → 0.499 @125 (monotone decline; the step-125 drop is catastrophic — charxiv halved, dynamath collapsed). Run died to a vLLM deepstack crash after step 125, but the model was already degrading.

## Training

- **Base model:** `Qwen/Qwen3-VL-8B-Instruct` (cold-start, `resume_mode=auto`).
- **Framework:** fork of [volcengine/verl](https://github.com/volcengine/verl) — [`ngquangtrung57/verl@videorl-mods`](https://github.com/ngquangtrung57/verl/tree/videorl-mods). Fully-async **GRPO**: FSDP2 trainer + vLLM rollouter.
- **Warm start:** none (cold-start from stock 8B-Instruct).
- **Reward:** dapo-style `score = 0.8·accuracy + 0.2·format` (`FORMAT_WEIGHT=0.2`, `FORMAT_MIN_THINK_CHARS=100`). No KL penalty.
- **Exploration:** OFF (this is the bitwise baseline against `omr-8b-grpo-ppexplore`).
- **Topology:** 4-node 2+2 — 2 trainer nodes (16-GPU FSDP2, dp=16) + 2 rollout nodes (16 GPU, vLLM TP=2 → 8 replicas). H100×8 per node.
- **Batch:** `ppo_mini_batch_size=16` × `require_batches=4` × `rollout.n=8` = **512 trajectories/step**.
- **Optim / seq:** lr `1e-6`, warmup 25 steps; `total_epochs=2`; clip_ratio 0.2 / 0.3 (clip_c=10.0); `max_prompt_length=2048`, `max_response_length=16384`; `enforce_eager=true`; `gpu_memory_utilization=0.75`; staleness 0.5.
- **Train metrics:** ~178 s/step; final reward 0.721; final response_length ~2194 tok; 125 steps trained.

## W&B

Project `verl_fully_async` (entity `quangtrung5705-nanyang-technological-university-singapore`):
<https://wandb.ai/quangtrung5705-nanyang-technological-university-singapore/verl_fully_async/runs/gkiiopep>

## Intended use / limitations

Research checkpoint — the **cold-start GRPO baseline** in a controlled exploration study. Useful mainly as the A/B reference for `omr-8b-grpo-ppexplore` (the winner). It only marginally beats the zero-shot base (0.668 vs 0.659) at its peak and is **unstable** (collapses to 0.499 if trained past ~step 100); for actual use prefer the ppexplore checkpoint. Math / visual-reasoning image QA, `<think>…</think>` then-answer format. No safety/RLHF alignment beyond the base.

## Usage

```python
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model_id = "ngqtrung/omr-8b-grpo-base"
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": Image.open("problem.png")},
        {"type": "text", "text": "Solve the problem. Think step by step inside <think>...</think>, then give the final answer."},
    ],
}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)
out = model.generate(**inputs, max_new_tokens=2048)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
```

## Citation / lineage

- **Base model:** Qwen3-VL-8B-Instruct (Qwen team). Inherits the **Qwen3-VL license** — review the base model's terms; the Apache-2.0 tag refers to this repo's RLVR training artifacts.
- **Framework:** verl (volcengine/verl), fork `ngquangtrung57/verl@videorl-mods`; fully-async GRPO (FSDP2 + vLLM).
- **Study:** controlled OMR/Video exploration study on Qwen3-VL-8B; this is the no-exploration OMR baseline arm (`docs/experiments_summary_8b.md`).