metadata
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
This repository contains the model checkpoints presented in the paper Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO (ICML 2026).
- Project Page: S2L-PO Project Page
- GitHub Repository: GitHub - S2L-PO
Introduction
S2L-PO (Small-to-Large Policy Optimization) is a post-training framework designed to enhance rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs. Instead of injecting token-level randomness (which can cause logical inconsistencies over long reasoning chains), S2L-PO leverages a fixed, smaller model from the same family as an explorer to guide a larger target model during the rollout phase.
Citation
If you find this work useful, please cite:
@inproceedings{ren2026s2lpo,
title = {Smaller Models are Natural Explorers for Policy-Level Diversity in {GRPO}},
author = {Ren, Yiming and Xu, Yiran and Lin, Zicheng and Shi, Chufan and Chen, Yukang and
Wang, Dingdong and Wu, Tianhe and Wang, Jujie and Yang, Yujiu and Qiao, Yu and Chu, Ruihang},
booktitle = {International Conference on Machine Learning (ICML)},
year = {2026},
}