---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: Qwen/Qwen3-14B
tags:
- grpo
- reinforcement-learning
- s2l-po
---

# Qwen3-14B-S2L-PO-4Bexplorer

This repository contains the Qwen3-14B learner model trained using the **S2L-PO (Small-to-Large Policy Optimization)** framework with a **Qwen3-4B** explorer. 

S2L-PO is presented in the paper [Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO](https://huggingface.co/papers/2605.30789).

* **Project Page:** [https://qishisuren123.github.io/S2L-PO/](https://qishisuren123.github.io/S2L-PO/)
* **Repository:** [https://github.com/qishisuren123/S2L-PO](https://github.com/qishisuren123/S2L-PO)
* **Paper:** [arXiv:2605.30789](https://huggingface.co/papers/2605.30789)

## Introduction

S2L-PO enhances rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs by using a smaller model within the same family as a natural explorer to generate a portion of the rollout groups. This introduces policy-level diversity, providing structured exploration signals for gradient estimation, resulting in faster convergence and higher final performance on reasoning tasks.

For this specific checkpoint, a fixed **Qwen3-4B-Base** explorer is used to guide the training of the **Qwen3-14B-Base** learner.

## Environment & Usage

Please refer to the [GitHub Repository](https://github.com/qishisuren123/S2L-PO) for installation instructions, environment setup, and evaluation scripts on mathematical reasoning benchmarks.

## Citation

If you find this work useful, please cite the paper:

```bibtex
@inproceedings{ren2026s2lpo,
  title     = {Smaller Models are Natural Explorers for Policy-Level Diversity in {GRPO}},
  author    = {Ren, Yiming and Xu, Yiran and Lin, Zicheng and Shi, Chufan and Chen, Yukang and
               Wang, Dingdong and Wu, Tianhe and Wang, Jujie and Yang, Yujiu and Qiao, Yu and Chu, Ruihang},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2026},
}
```