--- base_model: Qwen/Qwen3-VL-8B-Instruct library_name: transformers pipeline_tag: image-text-to-text license: apache-2.0 language: - en tags: - qwen3-vl - grpo - rlvr - verl - multimodal - math-reasoning - exploration --- # Qwen3-VL-8B · OMR · GRPO + ppexplore (τ0.95) — campaign winner RLVR post-training of **Qwen/Qwen3-VL-8B-Instruct** on the **OMR** (OpenMMReasoner, math/visual-reasoning) data with fully-async **GRPO** and an entropy-aware **token-dropout exploration** ("ppexplore") at the rollout stage. This checkpoint is `experiment_name=grpo_omr_4node_full_v1_8b_base_ppexplore_v1`, **`global_step_250`** — the best 8B OMR checkpoint of the whole controlled-exploration study. **This is the winner of the campaign: exploration (entropy-aware token-dropout, τ=0.95) reaches OMR overall-6 val accuracy `0.714` @ step 250, +4.6 pt over the cold-start GRPO baseline's peak of `0.668`, and — critically — it stabilizes training: the baseline collapsed to `0.499` by step 125, while this run never dropped below `0.659` over 225 steps.** Exploration was warm-started from the baseline's `global_step_50`. ## Results OMR 6-image inline validation (mmmu val, mathvista testmini, mathverse testmini Text-Dominant, wemath testmini, charxiv reasoning-qa, dynamath test). `overall-6` = unweighted mean of the 6 benches. Metric = accuracy (weight-independent). Keeper = **`global_step_250`**: | metric | overall-6 | mmmu | mathvista | mathverse | wemath | charxiv | dynamath | |---|---|---|---|---|---|---|---| | **ppexplore τ0.95 @250** | **0.7138** | 0.6789 | 0.8352 | 0.8967 | 0.8132 | 0.4150 | 0.6436 | | baseline peak @25 | 0.6681 | 0.6311 | 0.8019 | 0.8404 | 0.7437 | 0.3930 | 0.5986 | | stock Qwen3-VL-8B ckpt-0 | 0.659 | 0.629 | 0.811 | 0.824 | 0.723 | 0.396 | 0.570 | | **Δ (explore − baseline peak)** | **+4.6** | +4.8 | +3.3 | +5.7 | +6.9 | +2.2 | +4.5 | Exploration wins **every benchmark**, largest on wemath (+6.9) and mathverse (+5.7). Val trajectory: 0.681 @75 → 0.673 @100 → 0.659 @150 → 0.668 @200 → **0.714 @250** → 0.661 @300 (never below 0.659 over 225 steps; baseline by contrast fell 0.668→0.620→0.499). ## Training - **Base model:** `Qwen/Qwen3-VL-8B-Instruct`. - **Framework:** fork of [volcengine/verl](https://github.com/volcengine/verl) — [`ngquangtrung57/verl@videorl-mods`](https://github.com/ngquangtrung57/verl/tree/videorl-mods). Fully-async **GRPO**: FSDP2 trainer + vLLM rollouter, partial rollout, staleness-bounded off-policy. - **Warm start:** from the cold-start OMR baseline (`grpo_omr_4node_full_v1_8b_base_perf`) `global_step_50` (`resume_mode=resume_path`). - **Reward:** dapo-style `score = 0.8·accuracy + 0.2·format` (`FORMAT_WEIGHT=0.2`, `FORMAT_MIN_THINK_CHARS=100`). No KL penalty (`use_kl_in_reward=false`, `use_kl_loss=false`). - **Topology:** 4-node 2+2 — 2 trainer nodes (16-GPU FSDP2, dp=16) + 2 rollout nodes (16 GPU, vLLM TP=2 → 8 replicas). H100×8 per node. - **Batch:** `ppo_mini_batch_size=16` × `require_batches=4` × `rollout.n=8` = **512 trajectories/step**. - **Optim / seq:** lr `1e-6`, warmup 25 steps; `total_epochs=2`; clip_ratio 0.2 / 0.3 (clip_c=10.0); `max_prompt_length=2048`, `max_response_length=16384`; `enforce_eager=true`; `gpu_memory_utilization=0.75`; staleness 0.5. - **Exploration block** (entropy-aware token-dropout at rollout; byte-identical to the OMR-4B τ0.95 block that won +1.4 pt): | key | value | |---|---| | `enable` | true | | `trigger_mode` | high | | `top_prob_threshold` (τ) | 0.95 | | `k_explore` | 4 (of n=8 rollouts explore; 4 stay clean) | | `prompt_exploration_prob` | 0.5 | | `deterministic` | true | | `perturb_prob` | 1.0 | | `mask_from_loss` | true | | `drop_top_k` | 1 | | `restrict_to_think_region` | true | | `selection_seed` | 42 | - **Train metrics:** ~216 s/step; final reward 0.658 (lower than baseline by design — exploration tokens score below the greedy anchor); final response_length ~1947 tok. Run died at step 335 to a repeatable verl `resume_path` hang (4× confirmed) — keeper step_250 is well before that. ## W&B Project `verl_fully_async` (entity `quangtrung5705-nanyang-technological-university-singapore`). The logical run spans two crash-resume segments: - steps 75–124: - steps 125–335 (main, contains the step-250 keeper): ## Intended use / limitations Research checkpoint from a controlled exploration study (does token-dropout exploration help multimodal RLVR?). On OMR-8B the answer is a clear **yes**: this is the **campaign winner** (+4.6 pt and training-stability rescue). Best for math / visual-reasoning image QA in a `…` then-answer format. Not a general-purpose chat model; not tuned for video (see the `video-8b-grpo-*` siblings, which are a documented dead-heat at ~0.485). No safety/RLHF alignment beyond the base model. ## Usage ```python from transformers import AutoModelForImageTextToText, AutoProcessor from PIL import Image model_id = "ngqtrung/omr-8b-grpo-ppexplore" model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto") processor = AutoProcessor.from_pretrained(model_id) messages = [{ "role": "user", "content": [ {"type": "image", "image": Image.open("problem.png")}, {"type": "text", "text": "Solve the problem. Think step by step inside ..., then give the final answer."}, ], }] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt" ).to(model.device) out = model.generate(**inputs, max_new_tokens=2048) print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]) ``` ## Citation / lineage - **Base model:** Qwen3-VL-8B-Instruct (Qwen team). This checkpoint inherits the **Qwen3-VL license** — review the base model's terms before use; the Apache-2.0 tag refers to this repo's RLVR training artifacts. - **Framework:** verl (volcengine/verl), fork `ngquangtrung57/verl@videorl-mods`; fully-async GRPO (FSDP2 + vLLM). - **Method:** entropy-aware token-dropout exploration ("ppexplore", τ=0.95) at the rollout stage, warm-started from a mid-RL checkpoint. Part of a controlled OMR/Video exploration study on Qwen3-VL-8B (`docs/experiments_summary_8b.md`).