Instructions to use steven0226/qwen2.5-3b-grpo-gsm8k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use steven0226/qwen2.5-3b-grpo-gsm8k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="steven0226/qwen2.5-3b-grpo-gsm8k")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("steven0226/qwen2.5-3b-grpo-gsm8k")
model = AutoModelForCausalLM.from_pretrained("steven0226/qwen2.5-3b-grpo-gsm8k", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use steven0226/qwen2.5-3b-grpo-gsm8k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "steven0226/qwen2.5-3b-grpo-gsm8k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "steven0226/qwen2.5-3b-grpo-gsm8k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/steven0226/qwen2.5-3b-grpo-gsm8k

SGLang

How to use steven0226/qwen2.5-3b-grpo-gsm8k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "steven0226/qwen2.5-3b-grpo-gsm8k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "steven0226/qwen2.5-3b-grpo-gsm8k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "steven0226/qwen2.5-3b-grpo-gsm8k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "steven0226/qwen2.5-3b-grpo-gsm8k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Studio

How to use steven0226/qwen2.5-3b-grpo-gsm8k with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for steven0226/qwen2.5-3b-grpo-gsm8k to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for steven0226/qwen2.5-3b-grpo-gsm8k to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for steven0226/qwen2.5-3b-grpo-gsm8k to start chatting

Load model with FastModel

pip install unsloth
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
    model_name="steven0226/qwen2.5-3b-grpo-gsm8k",
    max_seq_length=2048,
)

Docker Model Runner
How to use steven0226/qwen2.5-3b-grpo-gsm8k with Docker Model Runner:
```
docker model run hf.co/steven0226/qwen2.5-3b-grpo-gsm8k
```

steven0226 commited on 24 days ago

Commit

eafe399

verified ·

1 Parent(s): 530490b

Unsloth Model Card

Browse files

Files changed (1) hide show

README.md +12 -146

README.md CHANGED Viewed

@@ -1,155 +1,21 @@
 ---
 license: apache-2.0
-base_model: Qwen/Qwen2.5-3B-Instruct
-datasets:
-- openai/gsm8k
 language:
 - en
-library_name: transformers
-pipeline_tag: text-generation
-tags:
-- grpo
-- rlvr
-- reasoning
-- unsloth
-- trl
-- qwen2.5
 ---
-# Qwen2.5-3B GRPO(RLVR)on GSM8K — merged 16-bit 完整模型
-以 **GRPO**(Group Relative Policy Optimization)+ **可驗證獎勵**(RLVR)在 GSM8K
-數學題上訓練 `Qwen/Qwen2.5-3B-Instruct` 的推理能力。LoRA adapter 版本:[steven0226/qwen2.5-3b-grpo-gsm8k-lora](https://huggingface.co/steven0226/qwen2.5-3b-grpo-gsm8k-lora)。
-訓練程式與獎勵函數:[GitHub — grpo-rlvr-reasoning](https://github.com/kuotunyu/grpo-rlvr-reasoning)。
-## 方法(白話)
-GRPO 對同一題一次抽 8 個回答,**組內互相比較**算出每個回答的
-相對優勢(advantage),取代 PPO 的 value model;獎勵不是另一個神經網路
-(reward model),而是**可程式驗證的規則**:
-| 獎勵函數 | 條件 | 分數 |
-|---|---|---|
-| correctness_reward | `<answer>` 內數字 == 標準答案 | 2.0 |
-| strict_format_reward | 完整 `<reasoning>...</reasoning><answer>...</answer>` 結構 | 0.5 |
-| soft_format_reward | 兩組 tag 依序出現(部分符合) | 0.5 |
-| number_only_reward | `<answer>` 是純數字 | 0.5 |
-答案對錯是可驗證的 —— 不會被 reward hacking、也省掉訓 reward model 的成本。
-這就是 DeepSeek-R1 帶起的 RLVR 路線;本專案復刻其招牌現象:
-**completion 長度隨訓練成長、reward 同步爬升**(模型自己學會寫更長的推理)。
-## 訓練曲線
-![reward curve](figs/reward_curve.png)
-![completion length curve](figs/completion_length_curve.png)
-## 訓練前後對照
-**題目**:Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
-<details><summary>訓練前(base)</summary>
-```
-<reasoning>
-To find out how many clips Natalia sold in total in April and May, we first need to determine the number of clips she sold in May. According to the problem, Natalia sold half as many clips in May as she did in April. Since she sold 48 clips in April, the number of clips sold in May would be half of 48. We can then add the number of clips sold in April and May to get the total.
-Let's perform the calculation:
-- Number of clips sold in April: 48
-- Number of clips sold in May: \(\frac{1}{2} \times 48 = 24\)
-Total clips sold in April and May: \(48 + 24 = 72\)
-</reasoning>
-<answer>
-Natalia sold a total of 72 clips in April and May.
-</answer>
-```
-</details>
-<details><summary>訓練後(GRPO)</summary>
-```
-<reasoning>
-To find out how many clips Natalia sold altogether in April and May, we need to follow these steps:
-1. Determine how many clips Natalia sold in May.
-2. Add the number of clips sold in April to the number of clips sold in May.
-First, we know that Natalia sold 48 clips in April. In May, she sold half as many clips as she did in April. Therefore, the number of clips sold in May is:
-\[ \text{Clips sold in May} = \frac{48}{2} = 24 \]
-Next, we add the clips sold in April to the clips sold in May:
-\[ \text{Total clips sold} = \text{Clips sold in April} + \text{Clips sold in May} = 48 + 24 = 72 \]
-So, the total number of clips Natalia sold in April and May is 72.
-</reasoning>
-<answer>
-72
-</answer>
-```
-</details>
-**題目**:Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?
-<details><summary>訓練前(base)</summary>
-```
-<reasoning>
-Weng earns $12 per hour for babysitting. Since she babysat for 50 minutes, we first need to convert this time into hours because her pay rate is given per hour. There are 60 minutes in an hour, so 50 minutes is equivalent to 50/60 hours. We can simplify 50/60 to 5/6. Now, we need to calculate 12 * (5/6) to find out how much she earned.
-</reasoning>
-<answer>
-12 * (5/6) = 10
-</answer>
-```
-</details>
-<details><summary>訓練後(GRPO)</summary>
-```
-<reasoning>
-To determine how much Weng earned for babysitting for 50 minutes at a rate of $12 per hour, we first need to convert the time into hours. There are 60 minutes in an hour.
-The conversion is:
-\[ \text{Time in hours} = \frac{\text{Time in minutes}}{60} = \frac{50}{60} = \frac{5}{6} \text{ hours} \]
-Next, we calculate the earnings by multiplying the hourly rate by the time in hours:
-\[ \text{Earnings} = \text{Hourly rate} \times \text{Time in hours} = 12 \times \frac{5}{6} \]
-To simplify the multiplication:
-\[ 12 \times \frac{5}{6} = 12 \times \frac{5}{6} = 2 \times 5 = 10 \]
-Therefore, Weng earned $10 for the babysitting.
-</reasoning>
-<answer>
-10
-</answer>
-```
-</details>
-## 超參數
-| 項目 | 值 |
-|---|---|
-| base model | Qwen/Qwen2.5-3B-Instruct |
-| 演算法 | GRPO(TRL + Unsloth,vLLM rollout) |
-| LoRA rank / alpha | 32 / 32(QKVO + MLP 全模組) |
-| learning rate | 5e-06(cosine,warmup 0.1,adamw_8bit) |
-| num_generations | 8 |
-| max prompt / completion length | 256 / 768 |
-| steps | 500 |
-| 量化 | 4-bit QLoRA(訓練時) |
-| seed | 3407 |
-超參以 Unsloth 官方 GRPO 範例為基準;偏差:LoRA r=32(官方 64)、
-completion 上限 768(官方 200,為觀察長度成長而加大)、
-strict_format regex 修正了官方版缺 re.DOTALL 導致多行推理永不匹配的問題。
-## 資料與污染聲明
-只使用 `openai/gsm8k`(config `main`)的 **train split(7,473 題)**;
-評測用的另一個 split 在整個訓練管線中**零接觸**(notebook 內有 assert 與
-程式級保證),評測結果見 GitHub repo 的 `results/eval_report.md`。
-## License
-Apache-2.0

 ---
+base_model: unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit
+tags:
+- text-generation-inference
+- transformers
+- unsloth
+- qwen2
 license: apache-2.0
 language:
 - en
 ---
+# Uploaded finetuned  model
+- **Developed by:** steven0226
+- **License:** apache-2.0
+- **Finetuned from model :** unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit
+This qwen2 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
+[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)