license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: Qwen/Qwen3-14B
tags:
- grpo
- reinforcement-learning
- s2l-po
Qwen3-14B-S2L-PO-4Bexplorer
This repository contains the Qwen3-14B learner model trained using the S2L-PO (Small-to-Large Policy Optimization) framework with a Qwen3-4B explorer.
S2L-PO is presented in the paper Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO.
- Project Page: https://qishisuren123.github.io/S2L-PO/
- Repository: https://github.com/qishisuren123/S2L-PO
- Paper: arXiv:2605.30789
Introduction
S2L-PO enhances rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs by using a smaller model within the same family as a natural explorer to generate a portion of the rollout groups. This introduces policy-level diversity, providing structured exploration signals for gradient estimation, resulting in faster convergence and higher final performance on reasoning tasks.
For this specific checkpoint, a fixed Qwen3-4B-Base explorer is used to guide the training of the Qwen3-14B-Base learner.
Environment & Usage
Please refer to the GitHub Repository for installation instructions, environment setup, and evaluation scripts on mathematical reasoning benchmarks.
Citation
If you find this work useful, please cite the paper:
@inproceedings{ren2026s2lpo,
title = {Smaller Models are Natural Explorers for Policy-Level Diversity in {GRPO}},
author = {Ren, Yiming and Xu, Yiran and Lin, Zicheng and Shi, Chufan and Chen, Yukang and
Wang, Dingdong and Wu, Tianhe and Wang, Jujie and Yang, Yujiu and Qiao, Yu and Chu, Ruihang},
booktitle = {International Conference on Machine Learning (ICML)},
year = {2026},
}