---
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: transformers
pipeline_tag: text-generation
tags:
- reinforcement-learning
- agentic-rl
- alfworld
- strata
- grpo
- qwen2.5
---

# StraTA / Qwen2.5-1.5B-Instruct / ALFWorld — optimizer step 20

Reproduction of **StraTA** (Strategic Trajectory Abstraction,
[arXiv:2605.06642](https://arxiv.org/abs/2605.06642), code
[xxyQwQ/StraTA](https://github.com/xxyQwQ/StraTA) @ `592238b`) on ALFWorld with
`Qwen/Qwen2.5-1.5B-Instruct`.

**This checkpoint: optimizer step 20 of 150.**

| | |
|---|---|
| Optimizer step | **20** / 150 |
| Validation success rate at this step | **3.57%** (ALFWorld `valid_seen`, 140 tasks) |
| Base model success rate | 0.71% |
| Backbone | `Qwen/Qwen2.5-1.5B-Instruct` (1.54B params, bf16) |
| Environment | ALFWorld (AgentGym HTTP interface) |
| Hardware | 2x NVIDIA A100 80GB |
| Wall-clock (full 150 steps) | ~44 hours |

## Training curve

Validation success rate on the full ALFWorld `valid_seen` split (140 tasks),
measured every 10 optimizer steps during training. Step 0 is the untrained base model.

| optimizer step | success rate (%) |
|---|---|
| 0 | 0.71 |
| 10 | 0.71 |
| 20 | 3.57 |
| 30 | 1.43 |
| 40 | 14.29 |
| 50 | 18.57 |
| 60 | 6.43 |
| 70 | 22.86 |
| 80 | 37.14 |
| 90 | 42.14 |
| 100 | 43.57 |
| 110 | 50.71 |
| 120 | 69.29 |
| 130 | 85.00 |
| 140 | 83.57 |
| 150 | 86.43 |

## Final evaluation (step 150, 2 seeds)

Run with `evaluate-strata-alfworld-text-qwen2.5-1.5b.sh`: `val_only`, 140 tasks,
temperature 0.7 / top_p 0.8 / top_k 20, max 50 env steps.

| subtask | seed 1 | seed 2 | mean ± std |
|---|---|---|---|
| pick | 97.14 | 97.14 | **97.14 ± 0.00** |
| clean | 92.59 | 92.59 | **92.59 ± 0.00** |
| pick2 | 91.67 | 87.50 | **89.58 ± 2.95** |
| heat | 81.25 | 81.25 | **81.25 ± 0.00** |
| cool | 76.00 | 72.00 | **74.00 ± 2.83** |
| look | 76.92 | 69.23 | **73.08 ± 5.44** |
| ALL | 87.86 | 85.71 | **86.79 ± 1.52** |

Auxiliary metrics: action `format_rate` **1.000** (both seeds), mean episode length
**13.2–14.2** steps (base model: 49.7).

## Hyperparameters

All algorithm hyperparameters follow the paper: 150 steps, batch size 16, 4 strategies
per task, 8 rollouts per strategy, oversampling ratio sigma=8, aggregation ratio
delta=0.5, length penalty threshold lambda=0.5, self-judgment reward weight kappa=0.1.

| group | parameter | value |
|---|---|---|
| StraTA | `num_stras_per_run` (strategies per task) | 4 |
| StraTA | `num_trajs_per_stra` (rollouts per strategy) | 8 |
| StraTA | `num_cands_per_stra` (oversampling ratio, sigma) | 8 |
| StraTA | `delta_coef` (aggregation ratio) | 0.5 |
| StraTA | `lambda_coef` (length penalty threshold) | 0.5 |
| StraTA | `kappa_coef` (self-judgment reward weight) | 0.1 |
| StraTA | `use_diverse_stras` / `use_self_judgment` | True / True |
| StraTA | `use_length_reward` / `use_format_reward` | True / True |
| StraTA | strategy embedding model | `sentence-transformers/all-MiniLM-L6-v2` (CPU) |
| RL | `adv_estimator` | grpo |
| RL | `algorithm.gamma` | 0 |
| RL | `kl_ctrl.kl_coef` | 0.001 |
| RL | `actor.use_kl_loss` | False |
| RL | `clip_ratio_high` | 0.28 |
| RL | `loss_agg_mode` | seq-mean-token-sum |
| RL | `entropy_coeff` | 0 |
| RL | `stepwise_advantage` | enabled, `per_step` |
| Optim | learning rate | 1e-6 |
| Optim | `ppo_mini_batch_size` (global) | 1024 |
| Optim | `use_dynamic_bsz` / `ppo_max_token_len_per_gpu` | True / 32768 |
| Data | `train_batch_size` (tasks per step) | 16 |
| Data | episodes per step | 16 x 4 x 8 = **512** |
| Data | `max_prompt_length` / `max_response_length` | 7168 / 1024 |
| Data | env `num_histories` | 2 |
| Data | max env steps per episode | 50 |
| Rollout | engine / temperature | vLLM 0.11.0 (async) / 1.0 |
| Eval | `val_kwargs` | n=1, temperature 0.7, top_p 0.8, top_k 20 |
| Train | total steps | 150 (`total_training_steps=151`) |


### Infrastructure deltas

This run reproduces the upstream 7B recipe on 2x A100 80GB. A full diff of the
resolved Hydra config against `train-strata-alfworld-text-qwen2.5-7b.sh` shows
**6 differing keys out of 533**, all infrastructure -- no algorithm or optimization
parameter was changed:

| key | upstream 7B | this run | effect on results |
|---|---|---|---|
| `trainer.n_gpus_per_node` | 8 | 2 | none -- verl computes `ppo_mini_batch_size *= rollout.n` then `//= world_size`, so the global mini-batch is invariant |
| `actor.fsdp_config.param_offload` | True | False | none -- CPU offload only |
| `actor.fsdp_config.optimizer_offload` | True | False | none |
| `ref.fsdp_config.param_offload` | True | False | none |
| `actor.checkpoint.save_contents` | `[hf_model]` | `[model,optimizer,extra,hf_model]` | none -- what gets written |
| `trainer.save_freq` | 50 | 10 | none -- checkpoint cadence |


## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "chanyoungkim/strata-qwen2.5-1.5b-alfworld-step020"
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained(repo)
```

The model expects the StraTA ALFWorld prompt format: it first emits a plan inside
`<strategy>...</strategy>`, then acts, emitting one action per turn inside
`<action>...</action>`.

## Files

| path | contents |
|---|---|
| `model.safetensors`, `config.json`, tokenizer files | bf16 weights, ready for `from_pretrained`. `lm_head` is omitted because `tie_word_embeddings=true` (same layout as the base Qwen release). |
| `training_state/model_world_size_2_rank_{0,1}.pt` | FSDP-sharded fp32 weights |
| `training_state/optim_world_size_2_rank_{0,1}.pt` | Adam optimizer state |
| `training_state/extra_state_world_size_2_rank_{0,1}.pt` | RNG and scheduler state |
| `training_state/fsdp_config.json` | FSDP layout descriptor |

`training_state/` lets you resume RL training from this exact optimizer step. Note the
shards are written for **`world_size=2`** (2 GPUs, FSDP). Resuming on a different number
of GPUs requires resharding. To resume, place the files back as
`checkpoints/<exp>/models/global_step_20/actor/` and point verl at that directory.

The published `model.safetensors` is a bf16 cast of the fp32 master weights; the
original fp32 tensors are preserved inside `training_state/`.

## All checkpoints from this run

[step 10](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step010) · [step 20](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step020) · [step 30](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step030) · [step 40](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step040) · [step 50](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step050) · [step 60](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step060) · [step 70](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step070) · [step 80](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step080) · [step 90](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step090) · [step 100](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step100) · [step 110](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step110) · [step 120](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step120) · [step 130](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step130) · [step 140](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step140) · [step 150](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step150)

## Caveats

- **Do not use `is_correct` or `pass@1`** from the logs. Both are hardcoded to `1.0`
  in the workflow and are meaningless. The correct metric is
  `val/average/metrics/success_rate`.
- Evaluation uses the StraTA inference protocol (one strategy, then one trajectory
  conditioned on it; no diversity selection, no self-judgment). This is what
  `strata_workflow.run()` does when `rollout_engine.validate` is set, and it matches
  `evaluate-strata-*.sh`.
- Numbers here are **not** directly comparable to models evaluated under a different
  agent scaffold (different prompt format, context window, or sampling temperature).
- The evaluation set is ALFWorld `valid_seen` (140 tasks, all of them -- not a
  subsample). `valid_unseen` was not evaluated.


## Citation

```bibtex
@article{xue2026strata,
  title={StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction},
  author={Xue, Xiangyuan and Zhou, Yifan and Wang, Zidong and Tang, Shengji and Torr, Philip and Ouyang, Wanli and Bai, Lei and Yin, Zhenfei},
  journal={arXiv preprint arXiv:2605.06642},
  year={2026}
}
```