Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,87 @@
+---
+base_model: Qwen/Qwen3-VL-8B-Instruct
+library_name: transformers
+pipeline_tag: image-text-to-text
+license: apache-2.0
+language:
+  - en
+tags:
+  - qwen3-vl
+  - grpo
+  - rlvr
+  - verl
+  - multimodal
+  - math-reasoning
+---
+
+# Qwen3-VL-8B · OMR · GRPO baseline (cold-start)
+
+RLVR post-training of **Qwen/Qwen3-VL-8B-Instruct** on the **OMR** (OpenMMReasoner, math/visual-reasoning) data with fully-async **GRPO**, cold-started from the stock 8B-Instruct (no exploration, no warm-start). This is `experiment_name=grpo_omr_4node_full_v1_8b_base_perf`, **`global_step_25`** (the keeper).
+
+**This is the OMR cold-start GRPO baseline. It peaks at OMR overall-6 val accuracy `0.668` @ step 25, then collapses to `0.499` by step 125** — cold-start RL on 8B-OMR is unstable without exploration. The keeper is `global_step_25`, the only checkpoint before degradation began. (The `omr-8b-grpo-ppexplore` sibling fixes this collapse and reaches 0.714.)
+
+## Results
+
+OMR 6-image inline validation (mmmu val, mathvista testmini, mathverse testmini Text-Dominant, wemath testmini, charxiv reasoning-qa, dynamath test). `overall-6` = unweighted mean. Metric = accuracy.
+
+Keeper = **`global_step_25`** (peak):
+
+| metric | overall-6 | mmmu | mathvista | mathverse | wemath | charxiv | dynamath |
+|---|---|---|---|---|---|---|---|
+| **baseline peak @25** | **0.6681** | 0.6311 | 0.8019 | 0.8404 | 0.7437 | 0.3930 | 0.5986 |
+| stock Qwen3-VL-8B ckpt-0 | 0.659 | 0.629 | 0.811 | 0.824 | 0.723 | 0.396 | 0.570 |
+| baseline @125 (collapse) | 0.4994 | 0.5021 | 0.6796 | 0.6995 | 0.6184 | 0.2010 | 0.2958 |
+
+Full trajectory (overall-6): **0.668 @25** → 0.644 @50 → 0.620 @75 → 0.620 @100 → 0.499 @125 (monotone decline; the step-125 drop is catastrophic — charxiv halved, dynamath collapsed). Run died to a vLLM deepstack crash after step 125, but the model was already degrading.
+
+## Training
+
+- **Base model:** `Qwen/Qwen3-VL-8B-Instruct` (cold-start, `resume_mode=auto`).
+- **Framework:** fork of [volcengine/verl](https://github.com/volcengine/verl) — [`ngquangtrung57/verl@videorl-mods`](https://github.com/ngquangtrung57/verl/tree/videorl-mods). Fully-async **GRPO**: FSDP2 trainer + vLLM rollouter.
+- **Warm start:** none (cold-start from stock 8B-Instruct).
+- **Reward:** dapo-style `score = 0.8·accuracy + 0.2·format` (`FORMAT_WEIGHT=0.2`, `FORMAT_MIN_THINK_CHARS=100`). No KL penalty.
+- **Exploration:** OFF (this is the bitwise baseline against `omr-8b-grpo-ppexplore`).
+- **Topology:** 4-node 2+2 — 2 trainer nodes (16-GPU FSDP2, dp=16) + 2 rollout nodes (16 GPU, vLLM TP=2 → 8 replicas). H100×8 per node.
+- **Batch:** `ppo_mini_batch_size=16` × `require_batches=4` × `rollout.n=8` = **512 trajectories/step**.
+- **Optim / seq:** lr `1e-6`, warmup 25 steps; `total_epochs=2`; clip_ratio 0.2 / 0.3 (clip_c=10.0); `max_prompt_length=2048`, `max_response_length=16384`; `enforce_eager=true`; `gpu_memory_utilization=0.75`; staleness 0.5.
+- **Train metrics:** ~178 s/step; final reward 0.721; final response_length ~2194 tok; 125 steps trained.
+
+## W&B
+
+Project `verl_fully_async` (entity `quangtrung5705-nanyang-technological-university-singapore`):
+<https://wandb.ai/quangtrung5705-nanyang-technological-university-singapore/verl_fully_async/runs/gkiiopep>
+
+## Intended use / limitations
+
+Research checkpoint — the **cold-start GRPO baseline** in a controlled exploration study. Useful mainly as the A/B reference for `omr-8b-grpo-ppexplore` (the winner). It only marginally beats the zero-shot base (0.668 vs 0.659) at its peak and is **unstable** (collapses to 0.499 if trained past ~step 100); for actual use prefer the ppexplore checkpoint. Math / visual-reasoning image QA, `<think>…</think>` then-answer format. No safety/RLHF alignment beyond the base.
+
+## Usage
+
+```python
+from transformers import AutoModelForImageTextToText, AutoProcessor
+from PIL import Image
+
+model_id = "ngqtrung/omr-8b-grpo-base"
+model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto")
+processor = AutoProcessor.from_pretrained(model_id)
+
+messages = [{
+    "role": "user",
+    "content": [
+        {"type": "image", "image": Image.open("problem.png")},
+        {"type": "text", "text": "Solve the problem. Think step by step inside <think>...</think>, then give the final answer."},
+    ],
+}]
+inputs = processor.apply_chat_template(
+    messages, add_generation_prompt=True, tokenize=True,
+    return_dict=True, return_tensors="pt"
+).to(model.device)
+out = model.generate(**inputs, max_new_tokens=2048)
+print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
+```
+
+## Citation / lineage
+
+- **Base model:** Qwen3-VL-8B-Instruct (Qwen team). Inherits the **Qwen3-VL license** — review the base model's terms; the Apache-2.0 tag refers to this repo's RLVR training artifacts.
+- **Framework:** verl (volcengine/verl), fork `ngquangtrung57/verl@videorl-mods`; fully-async GRPO (FSDP2 + vLLM).
+- **Study:** controlled OMR/Video exploration study on Qwen3-VL-8B; this is the no-exploration OMR baseline arm (`docs/experiments_summary_8b.md`).