Instructions to use steven0226/qwen2.5-3b-grpo-gsm8k-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use steven0226/qwen2.5-3b-grpo-gsm8k-lora with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit") model = PeftModel.from_pretrained(base_model, "steven0226/qwen2.5-3b-grpo-gsm8k-lora") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Unsloth Studio
How to use steven0226/qwen2.5-3b-grpo-gsm8k-lora with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for steven0226/qwen2.5-3b-grpo-gsm8k-lora to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for steven0226/qwen2.5-3b-grpo-gsm8k-lora to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for steven0226/qwen2.5-3b-grpo-gsm8k-lora to start chatting
Load model with FastModel
pip install unsloth from unsloth import FastModel model, tokenizer = FastModel.from_pretrained( model_name="steven0226/qwen2.5-3b-grpo-gsm8k-lora", max_seq_length=2048, )
docs: correct length-growth claim with actual 1000-step findings
Browse files
README.md
CHANGED
|
@@ -36,8 +36,13 @@ GRPO 對同一題一次抽 8 個回答,**組內互相比較**算出每個回答
|
|
| 36 |
| number_only_reward | `<answer>` 是純數字 | 0.5 |
|
| 37 |
|
| 38 |
答案對錯是可驗證的 —— 不會被 reward hacking、也省掉訓 reward model 的成本。
|
| 39 |
-
這就是 DeepSeek-R1 帶起的 RLVR 路線
|
| 40 |
-
**completion 長度
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
## 訓練曲線
|
| 43 |
|
|
@@ -179,7 +184,7 @@ Therefore, Weng earned $10.
|
|
| 179 |
| seed | 3407 |
|
| 180 |
|
| 181 |
超參以 Unsloth 官方 GRPO 範例為基準;偏差:LoRA r=32(官方 64)、
|
| 182 |
-
completion 上限 768(官方 200,為
|
| 183 |
strict_format regex 修正了官方版缺 re.DOTALL 導致多行推理永不匹配的問題。
|
| 184 |
|
| 185 |
## 資料與污染聲明
|
|
|
|
| 36 |
| number_only_reward | `<answer>` 是純數字 | 0.5 |
|
| 37 |
|
| 38 |
答案對錯是可驗證的 —— 不會被 reward hacking、也省掉訓 reward model 的成本。
|
| 39 |
+
這就是 DeepSeek-R1 帶起的 RLVR 路線。訓練結果:reward 乾淨爬升(從 ~1.4 穩定到
|
| 40 |
+
3.1~3.3,滿分 3.5),strict format 遵循率從 19.5% 衝到 90%。但 **completion 長度
|
| 41 |
+
並沒有隨訓練淨成長** —— 這點特別驗證過:500 步觀察到後又延長訓練到 1000 步重現
|
| 42 |
+
幾乎一樣的結果,排除了「訓不夠久」的可能。更細緻的分析發現模型學到的是「難題多想、
|
| 43 |
+
易題少想」的效率提升(每個難度區間的輸出都比訓練前短,但難題仍分配更多 token),
|
| 44 |
+
而不是無腦的長度成長。完整分析見
|
| 45 |
+
[GitHub README](https://github.com/kuotunyu/grpo-rlvr-reasoning#訓練結果reward-爬升確實但長度隨訓練成長沒有出現)。
|
| 46 |
|
| 47 |
## 訓練曲線
|
| 48 |
|
|
|
|
| 184 |
| seed | 3407 |
|
| 185 |
|
| 186 |
超參以 Unsloth 官方 GRPO 範例為基準;偏差:LoRA r=32(官方 64)、
|
| 187 |
+
completion 上限 768(官方 200,為推理長度留觀察空間而加大,實測並未觀察到淨成長)、
|
| 188 |
strict_format regex 修正了官方版缺 re.DOTALL 導致多行推理永不匹配的問題。
|
| 189 |
|
| 190 |
## 資料與污染聲明
|