| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| base_model: Qwen/Qwen3-14B |
| tags: |
| - grpo |
| - reinforcement-learning |
| - s2l-po |
| --- |
| |
| # Qwen3-14B-S2L-PO-4Bexplorer |
|
|
| This repository contains the Qwen3-14B learner model trained using the **S2L-PO (Small-to-Large Policy Optimization)** framework with a **Qwen3-4B** explorer. |
|
|
| S2L-PO is presented in the paper [Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO](https://huggingface.co/papers/2605.30789). |
|
|
| * **Project Page:** [https://qishisuren123.github.io/S2L-PO/](https://qishisuren123.github.io/S2L-PO/) |
| * **Repository:** [https://github.com/qishisuren123/S2L-PO](https://github.com/qishisuren123/S2L-PO) |
| * **Paper:** [arXiv:2605.30789](https://huggingface.co/papers/2605.30789) |
|
|
| ## Introduction |
|
|
| S2L-PO enhances rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs by using a smaller model within the same family as a natural explorer to generate a portion of the rollout groups. This introduces policy-level diversity, providing structured exploration signals for gradient estimation, resulting in faster convergence and higher final performance on reasoning tasks. |
|
|
| For this specific checkpoint, a fixed **Qwen3-4B-Base** explorer is used to guide the training of the **Qwen3-14B-Base** learner. |
|
|
| ## Environment & Usage |
|
|
| Please refer to the [GitHub Repository](https://github.com/qishisuren123/S2L-PO) for installation instructions, environment setup, and evaluation scripts on mathematical reasoning benchmarks. |
|
|
| ## Citation |
|
|
| If you find this work useful, please cite the paper: |
|
|
| ```bibtex |
| @inproceedings{ren2026s2lpo, |
| title = {Smaller Models are Natural Explorers for Policy-Level Diversity in {GRPO}}, |
| author = {Ren, Yiming and Xu, Yiran and Lin, Zicheng and Shi, Chufan and Chen, Yukang and |
| Wang, Dingdong and Wu, Tianhe and Wang, Jujie and Yang, Yujiu and Qiao, Yu and Chu, Ruihang}, |
| booktitle = {International Conference on Machine Learning (ICML)}, |
| year = {2026}, |
| } |
| ``` |
