Add model, config, tokenizer and model card

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,192 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-1.5B-Instruct
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- reinforcement-learning
+- agentic-rl
+- alfworld
+- strata
+- grpo
+- qwen2.5
+---
+
+# StraTA / Qwen2.5-1.5B-Instruct / ALFWorld — optimizer step 120
+
+Reproduction of **StraTA** (Strategic Trajectory Abstraction,
+[arXiv:2605.06642](https://arxiv.org/abs/2605.06642), code
+[xxyQwQ/StraTA](https://github.com/xxyQwQ/StraTA) @ `592238b`) on ALFWorld with
+`Qwen/Qwen2.5-1.5B-Instruct`.
+
+**This checkpoint: optimizer step 120 of 150.**
+
+| | |
+|---|---|
+| Optimizer step | **120** / 150 |
+| Validation success rate at this step | **69.29%** (ALFWorld `valid_seen`, 140 tasks) |
+| Base model success rate | 0.71% |
+| Backbone | `Qwen/Qwen2.5-1.5B-Instruct` (1.54B params, bf16) |
+| Environment | ALFWorld (AgentGym HTTP interface) |
+| Hardware | 2x NVIDIA A100 80GB |
+| Wall-clock (full 150 steps) | ~44 hours |
+
+## Training curve
+
+Validation success rate on the full ALFWorld `valid_seen` split (140 tasks),
+measured every 10 optimizer steps during training. Step 0 is the untrained base model.
+
+| optimizer step | success rate (%) |
+|---|---|
+| 0 | 0.71 |
+| 10 | 0.71 |
+| 20 | 3.57 |
+| 30 | 1.43 |
+| 40 | 14.29 |
+| 50 | 18.57 |
+| 60 | 6.43 |
+| 70 | 22.86 |
+| 80 | 37.14 |
+| 90 | 42.14 |
+| 100 | 43.57 |
+| 110 | 50.71 |
+| 120 | 69.29 |
+| 130 | 85.00 |
+| 140 | 83.57 |
+| 150 | 86.43 |
+
+## Final evaluation (step 150, 2 seeds)
+
+Run with `evaluate-strata-alfworld-text-qwen2.5-1.5b.sh`: `val_only`, 140 tasks,
+temperature 0.7 / top_p 0.8 / top_k 20, max 50 env steps.
+
+| subtask | seed 1 | seed 2 | mean ± std |
+|---|---|---|---|
+| pick | 97.14 | 97.14 | **97.14 ± 0.00** |
+| clean | 92.59 | 92.59 | **92.59 ± 0.00** |
+| pick2 | 91.67 | 87.50 | **89.58 ± 2.95** |
+| heat | 81.25 | 81.25 | **81.25 ± 0.00** |
+| cool | 76.00 | 72.00 | **74.00 ± 2.83** |
+| look | 76.92 | 69.23 | **73.08 ± 5.44** |
+| ALL | 87.86 | 85.71 | **86.79 ± 1.52** |
+
+Auxiliary metrics: action `format_rate` **1.000** (both seeds), mean episode length
+**13.2–14.2** steps (base model: 49.7).
+
+## Hyperparameters
+
+All algorithm hyperparameters follow the paper: 150 steps, batch size 16, 4 strategies
+per task, 8 rollouts per strategy, oversampling ratio sigma=8, aggregation ratio
+delta=0.5, length penalty threshold lambda=0.5, self-judgment reward weight kappa=0.1.
+
+| group | parameter | value |
+|---|---|---|
+| StraTA | `num_stras_per_run` (strategies per task) | 4 |
+| StraTA | `num_trajs_per_stra` (rollouts per strategy) | 8 |
+| StraTA | `num_cands_per_stra` (oversampling ratio, sigma) | 8 |
+| StraTA | `delta_coef` (aggregation ratio) | 0.5 |
+| StraTA | `lambda_coef` (length penalty threshold) | 0.5 |
+| StraTA | `kappa_coef` (self-judgment reward weight) | 0.1 |
+| StraTA | `use_diverse_stras` / `use_self_judgment` | True / True |
+| StraTA | `use_length_reward` / `use_format_reward` | True / True |
+| StraTA | strategy embedding model | `sentence-transformers/all-MiniLM-L6-v2` (CPU) |
+| RL | `adv_estimator` | grpo |
+| RL | `algorithm.gamma` | 0 |
+| RL | `kl_ctrl.kl_coef` | 0.001 |
+| RL | `actor.use_kl_loss` | False |
+| RL | `clip_ratio_high` | 0.28 |
+| RL | `loss_agg_mode` | seq-mean-token-sum |
+| RL | `entropy_coeff` | 0 |
+| RL | `stepwise_advantage` | enabled, `per_step` |
+| Optim | learning rate | 1e-6 |
+| Optim | `ppo_mini_batch_size` (global) | 1024 |
+| Optim | `use_dynamic_bsz` / `ppo_max_token_len_per_gpu` | True / 32768 |
+| Data | `train_batch_size` (tasks per step) | 16 |
+| Data | episodes per step | 16 x 4 x 8 = **512** |
+| Data | `max_prompt_length` / `max_response_length` | 7168 / 1024 |
+| Data | env `num_histories` | 2 |
+| Data | max env steps per episode | 50 |
+| Rollout | engine / temperature | vLLM 0.11.0 (async) / 1.0 |
+| Eval | `val_kwargs` | n=1, temperature 0.7, top_p 0.8, top_k 20 |
+| Train | total steps | 150 (`total_training_steps=151`) |
+
+
+### Infrastructure deltas
+
+This run reproduces the upstream 7B recipe on 2x A100 80GB. A full diff of the
+resolved Hydra config against `train-strata-alfworld-text-qwen2.5-7b.sh` shows
+**6 differing keys out of 533**, all infrastructure -- no algorithm or optimization
+parameter was changed:
+
+| key | upstream 7B | this run | effect on results |
+|---|---|---|---|
+| `trainer.n_gpus_per_node` | 8 | 2 | none -- verl computes `ppo_mini_batch_size *= rollout.n` then `//= world_size`, so the global mini-batch is invariant |
+| `actor.fsdp_config.param_offload` | True | False | none -- CPU offload only |
+| `actor.fsdp_config.optimizer_offload` | True | False | none |
+| `ref.fsdp_config.param_offload` | True | False | none |
+| `actor.checkpoint.save_contents` | `[hf_model]` | `[model,optimizer,extra,hf_model]` | none -- what gets written |
+| `trainer.save_freq` | 50 | 10 | none -- checkpoint cadence |
+
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "chanyoungkim/strata-qwen2.5-1.5b-alfworld-step120"
+model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
+tok = AutoTokenizer.from_pretrained(repo)
+```
+
+The model expects the StraTA ALFWorld prompt format: it first emits a plan inside
+`<strategy>...</strategy>`, then acts, emitting one action per turn inside
+`<action>...</action>`.
+
+## Files
+
+| path | contents |
+|---|---|
+| `model.safetensors`, `config.json`, tokenizer files | bf16 weights, ready for `from_pretrained`. `lm_head` is omitted because `tie_word_embeddings=true` (same layout as the base Qwen release). |
+| `training_state/model_world_size_2_rank_{0,1}.pt` | FSDP-sharded fp32 weights |
+| `training_state/optim_world_size_2_rank_{0,1}.pt` | Adam optimizer state |
+| `training_state/extra_state_world_size_2_rank_{0,1}.pt` | RNG and scheduler state |
+| `training_state/fsdp_config.json` | FSDP layout descriptor |
+
+`training_state/` lets you resume RL training from this exact optimizer step. Note the
+shards are written for **`world_size=2`** (2 GPUs, FSDP). Resuming on a different number
+of GPUs requires resharding. To resume, place the files back as
+`checkpoints/<exp>/models/global_step_120/actor/` and point verl at that directory.
+
+The published `model.safetensors` is a bf16 cast of the fp32 master weights; the
+original fp32 tensors are preserved inside `training_state/`.
+
+## All checkpoints from this run
+
+[step 10](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step010) · [step 20](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step020) · [step 30](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step030) · [step 40](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step040) · [step 50](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step050) · [step 60](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step060) · [step 70](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step070) · [step 80](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step080) · [step 90](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step090) · [step 100](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step100) · [step 110](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step110) · [step 120](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step120) · [step 130](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step130) · [step 140](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step140) · [step 150](https://huggingface.co/chanyoungkim/strata-qwen2.5-1.5b-alfworld-step150)
+
+## Caveats
+
+- **Do not use `is_correct` or `pass@1`** from the logs. Both are hardcoded to `1.0`
+  in the workflow and are meaningless. The correct metric is
+  `val/average/metrics/success_rate`.
+- Evaluation uses the StraTA inference protocol (one strategy, then one trajectory
+  conditioned on it; no diversity selection, no self-judgment). This is what
+  `strata_workflow.run()` does when `rollout_engine.validate` is set, and it matches
+  `evaluate-strata-*.sh`.
+- Numbers here are **not** directly comparable to models evaluated under a different
+  agent scaffold (different prompt format, context window, or sampling temperature).
+- The evaluation set is ALFWorld `valid_seen` (140 tasks, all of them -- not a
+  subsample). `valid_unseen` was not evaluated.
+
+
+## Citation
+
+```bibtex
+@article{xue2026strata,
+  title={StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction},
+  author={Xue, Xiangyuan and Zhou, Yifan and Wang, Zidong and Tang, Shengji and Torr, Philip and Ouyang, Wanli and Bai, Lei and Yin, Zhenfei},
+  journal={arXiv preprint arXiv:2605.06642},
+  year={2026}
+}
+```

added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja

@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

config.json

@@ -0,0 +1,59 @@
+{
+  "architectures": [
+    "Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "hidden_act": "silu",
+  "hidden_size": 1536,
+  "initializer_range": 0.02,
+  "intermediate_size": 8960,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "max_window_layers": 21,
+  "model_type": "qwen2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 2,
+  "pad_token_id": 151643,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "4.57.3",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151936,
+  "torch_dtype": "bfloat16"
+}

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "repetition_penalty": 1.1,
+  "temperature": 0.7,
+  "top_k": 20,
+  "top_p": 0.8,
+  "transformers_version": "4.57.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89b0f9aaaa67772e7da25349dc2a48db97e1a8c03e2ca1a1aeb88c8e9fd25fe1
+size 3087467144

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
+size 11421896

tokenizer_config.json

@@ -0,0 +1,207 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}