| license: apache-2.0 | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| # Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO | |
| This repository contains the model checkpoints presented in the paper [Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO](https://huggingface.co/papers/2605.30789) (ICML 2026). | |
| * **Project Page:** [S2L-PO Project Page](https://qishisuren123.github.io/S2L-PO) | |
| * **GitHub Repository:** [GitHub - S2L-PO](https://github.com/qishisuren123/S2L-PO) | |
| ## Introduction | |
| S2L-PO (Small-to-Large Policy Optimization) is a post-training framework designed to enhance rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs. Instead of injecting token-level randomness (which can cause logical inconsistencies over long reasoning chains), S2L-PO leverages a fixed, smaller model from the same family as an explorer to guide a larger target model during the rollout phase. | |
| ## Citation | |
| If you find this work useful, please cite: | |
| ```bibtex | |
| @inproceedings{ren2026s2lpo, | |
| title = {Smaller Models are Natural Explorers for Policy-Level Diversity in {GRPO}}, | |
| author = {Ren, Yiming and Xu, Yiran and Lin, Zicheng and Shi, Chufan and Chen, Yukang and | |
| Wang, Dingdong and Wu, Tianhe and Wang, Jujie and Yang, Yujiu and Qiao, Yu and Chu, Ruihang}, | |
| booktitle = {International Conference on Machine Learning (ICML)}, | |
| year = {2026}, | |
| } | |
| ``` |