---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
---

# Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO

This repository contains the model checkpoints presented in the paper [Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO](https://huggingface.co/papers/2605.30789) (ICML 2026).

* **Project Page:** [S2L-PO Project Page](https://qishisuren123.github.io/S2L-PO)
* **GitHub Repository:** [GitHub - S2L-PO](https://github.com/qishisuren123/S2L-PO)

## Introduction

S2L-PO (Small-to-Large Policy Optimization) is a post-training framework designed to enhance rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs. Instead of injecting token-level randomness (which can cause logical inconsistencies over long reasoning chains), S2L-PO leverages a fixed, smaller model from the same family as an explorer to guide a larger target model during the rollout phase.

## Citation

If you find this work useful, please cite:

```bibtex
@inproceedings{ren2026s2lpo,
  title     = {Smaller Models are Natural Explorers for Policy-Level Diversity in {GRPO}},
  author    = {Ren, Yiming and Xu, Yiran and Lin, Zicheng and Shi, Chufan and Chen, Yukang and
               Wang, Dingdong and Wu, Tianhe and Wang, Jujie and Yang, Yujiu and Qiao, Yu and Chu, Ruihang},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2026},
}
```