Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,105 @@
+---
+base_model: Qwen/Qwen3-VL-8B-Instruct
+library_name: transformers
+pipeline_tag: image-text-to-text
+license: apache-2.0
+language:
+  - en
+tags:
+  - qwen3-vl
+  - grpo
+  - rlvr
+  - verl
+  - multimodal
+  - math-reasoning
+  - exploration
+---
+
+# Qwen3-VL-8B · OMR · GRPO + ppexplore (τ0.95) — campaign winner
+
+RLVR post-training of **Qwen/Qwen3-VL-8B-Instruct** on the **OMR** (OpenMMReasoner, math/visual-reasoning) data with fully-async **GRPO** and an entropy-aware **token-dropout exploration** ("ppexplore") at the rollout stage. This checkpoint is `experiment_name=grpo_omr_4node_full_v1_8b_base_ppexplore_v1`, **`global_step_250`** — the best 8B OMR checkpoint of the whole controlled-exploration study.
+
+**This is the winner of the campaign: exploration (entropy-aware token-dropout, τ=0.95) reaches OMR overall-6 val accuracy `0.714` @ step 250, +4.6 pt over the cold-start GRPO baseline's peak of `0.668`, and — critically — it stabilizes training: the baseline collapsed to `0.499` by step 125, while this run never dropped below `0.659` over 225 steps.** Exploration was warm-started from the baseline's `global_step_50`.
+
+## Results
+
+OMR 6-image inline validation (mmmu val, mathvista testmini, mathverse testmini Text-Dominant, wemath testmini, charxiv reasoning-qa, dynamath test). `overall-6` = unweighted mean of the 6 benches. Metric = accuracy (weight-independent).
+
+Keeper = **`global_step_250`**:
+
+| metric | overall-6 | mmmu | mathvista | mathverse | wemath | charxiv | dynamath |
+|---|---|---|---|---|---|---|---|
+| **ppexplore τ0.95 @250** | **0.7138** | 0.6789 | 0.8352 | 0.8967 | 0.8132 | 0.4150 | 0.6436 |
+| baseline peak @25 | 0.6681 | 0.6311 | 0.8019 | 0.8404 | 0.7437 | 0.3930 | 0.5986 |
+| stock Qwen3-VL-8B ckpt-0 | 0.659 | 0.629 | 0.811 | 0.824 | 0.723 | 0.396 | 0.570 |
+| **Δ (explore − baseline peak)** | **+4.6** | +4.8 | +3.3 | +5.7 | +6.9 | +2.2 | +4.5 |
+
+Exploration wins **every benchmark**, largest on wemath (+6.9) and mathverse (+5.7). Val trajectory: 0.681 @75 → 0.673 @100 → 0.659 @150 → 0.668 @200 → **0.714 @250** → 0.661 @300 (never below 0.659 over 225 steps; baseline by contrast fell 0.668→0.620→0.499).
+
+## Training
+
+- **Base model:** `Qwen/Qwen3-VL-8B-Instruct`.
+- **Framework:** fork of [volcengine/verl](https://github.com/volcengine/verl) — [`ngquangtrung57/verl@videorl-mods`](https://github.com/ngquangtrung57/verl/tree/videorl-mods). Fully-async **GRPO**: FSDP2 trainer + vLLM rollouter, partial rollout, staleness-bounded off-policy.
+- **Warm start:** from the cold-start OMR baseline (`grpo_omr_4node_full_v1_8b_base_perf`) `global_step_50` (`resume_mode=resume_path`).
+- **Reward:** dapo-style `score = 0.8·accuracy + 0.2·format` (`FORMAT_WEIGHT=0.2`, `FORMAT_MIN_THINK_CHARS=100`). No KL penalty (`use_kl_in_reward=false`, `use_kl_loss=false`).
+- **Topology:** 4-node 2+2 — 2 trainer nodes (16-GPU FSDP2, dp=16) + 2 rollout nodes (16 GPU, vLLM TP=2 → 8 replicas). H100×8 per node.
+- **Batch:** `ppo_mini_batch_size=16` × `require_batches=4` × `rollout.n=8` = **512 trajectories/step**.
+- **Optim / seq:** lr `1e-6`, warmup 25 steps; `total_epochs=2`; clip_ratio 0.2 / 0.3 (clip_c=10.0); `max_prompt_length=2048`, `max_response_length=16384`; `enforce_eager=true`; `gpu_memory_utilization=0.75`; staleness 0.5.
+- **Exploration block** (entropy-aware token-dropout at rollout; byte-identical to the OMR-4B τ0.95 block that won +1.4 pt):
+
+| key | value |
+|---|---|
+| `enable` | true |
+| `trigger_mode` | high |
+| `top_prob_threshold` (τ) | 0.95 |
+| `k_explore` | 4 (of n=8 rollouts explore; 4 stay clean) |
+| `prompt_exploration_prob` | 0.5 |
+| `deterministic` | true |
+| `perturb_prob` | 1.0 |
+| `mask_from_loss` | true |
+| `drop_top_k` | 1 |
+| `restrict_to_think_region` | true |
+| `selection_seed` | 42 |
+
+- **Train metrics:** ~216 s/step; final reward 0.658 (lower than baseline by design — exploration tokens score below the greedy anchor); final response_length ~1947 tok. Run died at step 335 to a repeatable verl `resume_path` hang (4× confirmed) — keeper step_250 is well before that.
+
+## W&B
+
+Project `verl_fully_async` (entity `quangtrung5705-nanyang-technological-university-singapore`). The logical run spans two crash-resume segments:
+- steps 75–124: <https://wandb.ai/quangtrung5705-nanyang-technological-university-singapore/verl_fully_async/runs/i56xzqqo>
+- steps 125–335 (main, contains the step-250 keeper): <https://wandb.ai/quangtrung5705-nanyang-technological-university-singapore/verl_fully_async/runs/hi1ng1nx>
+
+## Intended use / limitations
+
+Research checkpoint from a controlled exploration study (does token-dropout exploration help multimodal RLVR?). On OMR-8B the answer is a clear **yes**: this is the **campaign winner** (+4.6 pt and training-stability rescue). Best for math / visual-reasoning image QA in a `<think>…</think>` then-answer format. Not a general-purpose chat model; not tuned for video (see the `video-8b-grpo-*` siblings, which are a documented dead-heat at ~0.485). No safety/RLHF alignment beyond the base model.
+
+## Usage
+
+```python
+from transformers import AutoModelForImageTextToText, AutoProcessor
+from PIL import Image
+
+model_id = "ngqtrung/omr-8b-grpo-ppexplore"
+model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto")
+processor = AutoProcessor.from_pretrained(model_id)
+
+messages = [{
+    "role": "user",
+    "content": [
+        {"type": "image", "image": Image.open("problem.png")},
+        {"type": "text", "text": "Solve the problem. Think step by step inside <think>...</think>, then give the final answer."},
+    ],
+}]
+inputs = processor.apply_chat_template(
+    messages, add_generation_prompt=True, tokenize=True,
+    return_dict=True, return_tensors="pt"
+).to(model.device)
+out = model.generate(**inputs, max_new_tokens=2048)
+print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
+```
+
+## Citation / lineage
+
+- **Base model:** Qwen3-VL-8B-Instruct (Qwen team). This checkpoint inherits the **Qwen3-VL license** — review the base model's terms before use; the Apache-2.0 tag refers to this repo's RLVR training artifacts.
+- **Framework:** verl (volcengine/verl), fork `ngquangtrung57/verl@videorl-mods`; fully-async GRPO (FSDP2 + vLLM).
+- **Method:** entropy-aware token-dropout exploration ("ppexplore", τ=0.95) at the rollout stage, warm-started from a mid-RL checkpoint. Part of a controlled OMR/Video exploration study on Qwen3-VL-8B (`docs/experiments_summary_8b.md`).