Instructions to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret")
model = AutoModelForCausalLM.from_pretrained("moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret

SGLang

How to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Studio

How to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret to start chatting

Load model with FastModel

pip install unsloth
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
    model_name="moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret",
    max_seq_length=2048,
)

Docker Model Runner
How to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with Docker Model Runner:
```
docker model run hf.co/moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret
```

aawwssdddwdwd commited on Jun 22, 2025

Commit

176c11b

verified ·

1 Parent(s): 1e8137b

End of training

Browse files

Files changed (6) hide show

adapter_config.json +5 -5
adapter_model.safetensors +1 -1
all_results.json +5 -5
train_results.json +5 -5
trainer_state.json +1370 -135
training_args.bin +1 -1

adapter_config.json CHANGED Viewed

@@ -24,13 +24,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "k_proj",
-    "up_proj",
-    "q_proj",
     "v_proj",
     "o_proj",
-    "down_proj",
-    "gate_proj"
   ],
   "task_type": "CAUSAL_LM",
   "trainable_token_indices": null,

   "rank_pattern": {},
   "revision": null,
   "target_modules": [
+    "gate_proj",
+    "down_proj",
     "v_proj",
+    "q_proj",
     "o_proj",
+    "k_proj",
+    "up_proj"
   ],
   "task_type": "CAUSAL_LM",
   "trainable_token_indices": null,

adapter_model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c7b265f455d6423141dab1486840df696663a53fca62a7e66b3d177c3a43d54a
 size 73911112

 version https://git-lfs.github.com/spec/v1
+oid sha256:025841bf1ac076e4b501a79e977a79ab09ff20b8504cade6de506be0931e287f
 size 73911112

all_results.json CHANGED Viewed

@@ -1,8 +1,8 @@
 {
     "total_flos": 0.0,
-    "train_loss": 9.159998512586754e-07,
-    "train_runtime": 1404.2154,
-    "train_samples": 11,
-    "train_samples_per_second": 0.684,
-    "train_steps_per_second": 0.014
 }

 {
     "total_flos": 0.0,
+    "train_loss": 4.005705436611606e-06,
+    "train_runtime": 2512.4216,
+    "train_samples": 35,
+    "train_samples_per_second": 0.716,
+    "train_steps_per_second": 0.06
 }

train_results.json CHANGED Viewed

@@ -1,8 +1,8 @@
 {
     "total_flos": 0.0,
-    "train_loss": 9.159998512586754e-07,
-    "train_runtime": 1404.2154,
-    "train_samples": 11,
-    "train_samples_per_second": 0.684,
-    "train_steps_per_second": 0.014
 }

 {
     "total_flos": 0.0,
+    "train_loss": 4.005705436611606e-06,
+    "train_runtime": 2512.4216,
+    "train_samples": 35,
+    "train_samples_per_second": 0.716,
+    "train_steps_per_second": 0.06
 }

trainer_state.json CHANGED Viewed

@@ -1,218 +1,1453 @@
 {
   "best_metric": null,
   "best_model_checkpoint": null,
-  "epoch": 20.0,
   "eval_steps": 500,
-  "global_step": 20,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
-      "completion_length": 393.28125762939453,
-      "epoch": 2.0,
-      "grad_norm": 0.2830348014831543,
-      "kl": 0.000824785056465771,
-      "learning_rate": 4.965903258506806e-07,
       "loss": 0.0,
-      "reward": 1.0125551894307137,
-      "reward_std": 1.091110222041607,
-      "rewards/concensus_correctness_reward_func": 0.04804166546091437,
-      "rewards/consensus_reward_func": 0.40625000186264515,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2080864403396845,
-      "rewards/soft_format_reward_func": 0.0052083334885537624,
-      "rewards/strict_format_reward_func": 0.010416666977107525,
-      "rewards/xmlcount_reward_func": 0.33455209620296955,
       "step": 2
     },
     {
-      "completion_length": 369.2447967529297,
-      "epoch": 4.0,
-      "grad_norm": 0.292804479598999,
-      "kl": 0.0008787213737377897,
-      "learning_rate": 4.698684378016222e-07,
       "loss": 0.0,
-      "reward": 1.0233181528747082,
-      "reward_std": 0.8960795998573303,
-      "rewards/concensus_correctness_reward_func": 0.0588750010356307,
-      "rewards/consensus_reward_func": 0.3020833432674408,
       "rewards/cumulative_reward_2": 0.0,
-      "rewards/final_correctness_reward_func": 0.02083333395421505,
-      "rewards/question_recreation_reward_func": 0.2715317187830806,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.013020833488553762,
-      "rewards/xmlcount_reward_func": 0.35697395727038383,
       "step": 4
     },
     {
-      "completion_length": 384.4479217529297,
-      "epoch": 6.0,
-      "grad_norm": 0.2502409517765045,
-      "kl": 0.0008375985125894658,
-      "learning_rate": 4.193203929064353e-07,
       "loss": 0.0,
-      "reward": 1.0599401649087667,
-      "reward_std": 0.821756474673748,
-      "rewards/concensus_correctness_reward_func": 0.09556250250898302,
-      "rewards/consensus_reward_func": 0.2708333395421505,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.21652869414538145,
-      "rewards/soft_format_reward_func": 0.0052083334885537624,
-      "rewards/strict_format_reward_func": 0.0052083334885537624,
-      "rewards/xmlcount_reward_func": 0.4665989736095071,
       "step": 6
     },
     {
-      "completion_length": 386.85938262939453,
-      "epoch": 8.0,
-      "grad_norm": 0.2976052165031433,
-      "kl": 0.0007596394862048328,
-      "learning_rate": 3.5042385616324236e-07,
       "loss": 0.0,
-      "reward": 0.9930243901908398,
-      "reward_std": 0.9850930720567703,
-      "rewards/concensus_correctness_reward_func": 0.03681250149384141,
-      "rewards/consensus_reward_func": 0.31250000186264515,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2174879079684615,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0052083334885537624,
-      "rewards/xmlcount_reward_func": 0.4210156425833702,
       "step": 8
     },
     {
-      "completion_length": 388.5260543823242,
-      "epoch": 10.0,
-      "grad_norm": 0.29186224937438965,
-      "kl": 0.0008365052271983586,
-      "learning_rate": 2.706448363680831e-07,
       "loss": 0.0,
-      "reward": 0.9328028708696365,
-      "reward_std": 1.0771522670984268,
-      "rewards/concensus_correctness_reward_func": 0.0629479163326323,
-      "rewards/consensus_reward_func": 0.38541667349636555,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2666778340935707,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0052083334885537624,
-      "rewards/xmlcount_reward_func": 0.21255208551883698,
       "step": 10
     },
     {
-      "completion_length": 364.8229293823242,
-      "epoch": 12.0,
-      "grad_norm": 0.30848339200019836,
-      "kl": 0.0008186925406334922,
-      "learning_rate": 1.886286282148002e-07,
       "loss": 0.0,
-      "reward": 0.9532259553670883,
-      "reward_std": 0.913148432970047,
-      "rewards/concensus_correctness_reward_func": 0.03939583315514028,
-      "rewards/consensus_reward_func": 0.2500000037252903,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2515748974401504,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0052083334885537624,
-      "rewards/xmlcount_reward_func": 0.40704688616096973,
       "step": 12
     },
     {
-      "completion_length": 358.90625762939453,
-      "epoch": 14.0,
-      "grad_norm": 2.381375789642334,
-      "kl": 0.0010058831103378907,
-      "learning_rate": 1.1326296046939333e-07,
       "loss": 0.0,
-      "reward": 0.8708688616752625,
-      "reward_std": 0.8778228163719177,
-      "rewards/concensus_correctness_reward_func": 0.01347916736267507,
-      "rewards/consensus_reward_func": 0.2708333395421505,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.22638968750834465,
-      "rewards/soft_format_reward_func": 0.0078125,
-      "rewards/strict_format_reward_func": 0.026041667442768812,
-      "rewards/xmlcount_reward_func": 0.3263125028461218,
       "step": 14
     },
     {
-      "completion_length": 395.5364646911621,
-      "epoch": 16.0,
-      "grad_norm": 0.2610781788825989,
-      "kl": 0.0007961297233123332,
-      "learning_rate": 5.271487265090163e-08,
       "loss": 0.0,
-      "reward": 1.0859159231185913,
-      "reward_std": 0.816021628677845,
-      "rewards/concensus_correctness_reward_func": 0.04837500164285302,
-      "rewards/consensus_reward_func": 0.25000000558793545,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.32880652230232954,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0052083334885537624,
-      "rewards/xmlcount_reward_func": 0.45352603495121,
       "step": 16
     },
     {
-      "completion_length": 376.61458587646484,
-      "epoch": 18.0,
-      "grad_norm": 0.2838130593299866,
-      "kl": 0.0008370706782443449,
-      "learning_rate": 1.3545689574841341e-08,
       "loss": 0.0,
-      "reward": 0.9728391878306866,
-      "reward_std": 0.9583801254630089,
-      "rewards/concensus_correctness_reward_func": 0.04129166854545474,
-      "rewards/consensus_reward_func": 0.2291666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2855110433883965,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.015625000465661287,
-      "rewards/xmlcount_reward_func": 0.40124480426311493,
       "step": 18
     },
     {
-      "completion_length": 383.2604217529297,
-      "epoch": 20.0,
-      "grad_norm": 0.3486255407333374,
-      "kl": 0.0009046856794157065,
-      "learning_rate": 0.0,
       "loss": 0.0,
-      "reward": 1.1996122673153877,
-      "reward_std": 1.11057697981596,
-      "rewards/concensus_correctness_reward_func": 0.06595833483152092,
-      "rewards/consensus_reward_func": 0.3958333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.27341954689472914,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.013020833488553762,
-      "rewards/xmlcount_reward_func": 0.4513802193105221,
       "step": 20
     },
     {
       "epoch": 20.0,
-      "step": 20,
       "total_flos": 0.0,
-      "train_loss": 9.159998512586754e-07,
-      "train_runtime": 1404.2154,
-      "train_samples_per_second": 0.684,
-      "train_steps_per_second": 0.014
     }
   ],
   "logging_steps": 2,
-  "max_steps": 20,
   "num_input_tokens_seen": 0,
-  "num_train_epochs": 20,
-  "save_steps": 20,
   "stateful_callbacks": {
     "TrainerControl": {
       "args": {
@@ -226,7 +1461,7 @@
     }
   },
   "total_flos": 0.0,
-  "train_batch_size": 12,
   "trial_name": null,
   "trial_params": null
 }

 {
   "best_metric": null,
   "best_model_checkpoint": null,
+  "epoch": 25.0,
   "eval_steps": 500,
+  "global_step": 150,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
+      "completion_length": 243.9583396911621,
+      "epoch": 0.3333333333333333,
+      "grad_norm": 0.23586256802082062,
+      "kl": 0.001098359382012859,
+      "learning_rate": 2e-07,
       "loss": 0.0,
+      "reward": 0.8912999480962753,
+      "reward_std": 0.2599460817873478,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2947999183088541,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.575666680932045,
       "step": 2
     },
     {
+      "completion_length": 250.33333587646484,
+      "epoch": 0.6666666666666666,
+      "grad_norm": 0.23456373810768127,
+      "kl": 0.001041582232574001,
+      "learning_rate": 4e-07,
       "loss": 0.0,
+      "reward": 0.7571422457695007,
+      "reward_std": 0.3700219392776489,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
       "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.17472553998231888,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5824166685342789,
       "step": 4
     },
     {
+      "completion_length": 251.41666793823242,
+      "epoch": 1.0,
+      "grad_norm": 0.2602273225784302,
+      "kl": 0.0009062251629075035,
+      "learning_rate": 4.99941324504621e-07,
       "loss": 0.0,
+      "reward": 1.3463366031646729,
+      "reward_std": 0.5522305034101009,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.5000000223517418,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3054823987185955,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5408541858196259,
       "step": 6
     },
     {
+      "completion_length": 249.2916717529297,
+      "epoch": 1.3333333333333333,
+      "grad_norm": 0.22653937339782715,
+      "kl": 0.0009439304994884878,
+      "learning_rate": 4.99472085783721e-07,
       "loss": 0.0,
+      "reward": 0.868863582611084,
+      "reward_std": 0.409902460873127,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.21953026205301285,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6493333578109741,
       "step": 8
     },
     {
+      "completion_length": 245.79166793823242,
+      "epoch": 1.6666666666666665,
+      "grad_norm": 0.22448210418224335,
+      "kl": 0.0011687759833876044,
+      "learning_rate": 4.985344892885899e-07,
       "loss": 0.0,
+      "reward": 1.0392787605524063,
+      "reward_std": 0.5154697671532631,
+      "rewards/concensus_correctness_reward_func": 0.07125000655651093,
+      "rewards/consensus_reward_func": 0.1666666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.25507040694355965,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5462916940450668,
       "step": 10
     },
     {
+      "completion_length": 255.75,
+      "epoch": 2.0,
+      "grad_norm": 0.2590133547782898,
+      "kl": 0.0010423234780319035,
+      "learning_rate": 4.971302952586796e-07,
       "loss": 0.0,
+      "reward": 0.6098116040229797,
+      "reward_std": 0.437416672706604,
+      "rewards/concensus_correctness_reward_func": 0.010416666977107525,
+      "rewards/consensus_reward_func": 0.0,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1546240784227848,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.44477084279060364,
       "step": 12
     },
     {
+      "completion_length": 248.20833587646484,
+      "epoch": 2.3333333333333335,
+      "grad_norm": 0.24884550273418427,
+      "kl": 0.0009182853827951476,
+      "learning_rate": 4.952621399215597e-07,
       "loss": 0.0,
+      "reward": 0.940536618232727,
+      "reward_std": 0.2838931605219841,
+      "rewards/concensus_correctness_reward_func": 0.010416666977107525,
+      "rewards/consensus_reward_func": 0.0,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2979949302971363,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.63212501257658,
       "step": 14
     },
     {
+      "completion_length": 245.0833396911621,
+      "epoch": 2.6666666666666665,
+      "grad_norm": 0.2338993400335312,
+      "kl": 0.0010259418486384675,
+      "learning_rate": 4.929335305436764e-07,
       "loss": 0.0,
+      "reward": 0.7932935208082199,
+      "reward_std": 0.4369620755314827,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.20591850765049458,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5665416866540909,
       "step": 16
     },
     {
+      "completion_length": 252.50000381469727,
+      "epoch": 3.0,
+      "grad_norm": 0.24362091720104218,
+      "kl": 0.0009753270860528573,
+      "learning_rate": 4.901488388458247e-07,
       "loss": 0.0,
+      "reward": 1.042687565088272,
+      "reward_std": 0.7606733236461878,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.26568757742643356,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5270000323653221,
       "step": 18
     },
     {
+      "completion_length": 244.54166793823242,
+      "epoch": 3.3333333333333335,
+      "grad_norm": 0.2889133095741272,
+      "kl": 0.0010776016279123724,
+      "learning_rate": 4.869132927957006e-07,
       "loss": 0.0,
+      "reward": 0.9227994382381439,
+      "reward_std": 0.3747098818421364,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0833333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.21304943412542343,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6055833548307419,
       "step": 20
     },
     {
+      "completion_length": 247.25,
+      "epoch": 3.6666666666666665,
+      "grad_norm": 0.23984657227993011,
+      "kl": 0.0009606060775695369,
+      "learning_rate": 4.832329667929376e-07,
+      "loss": 0.0,
+      "reward": 1.008092075586319,
+      "reward_std": 0.4960784874856472,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.21671707555651665,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6872083470225334,
+      "step": 22
+    },
+    {
+      "completion_length": 243.0416717529297,
+      "epoch": 4.0,
+      "grad_norm": 0.30041074752807617,
+      "kl": 0.0009601146884961054,
+      "learning_rate": 4.791147702650565e-07,
+      "loss": 0.0,
+      "reward": 0.9435153305530548,
+      "reward_std": 0.5804080665111542,
+      "rewards/concensus_correctness_reward_func": 0.010416666977107525,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3101403061300516,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.518791675567627,
+      "step": 24
+    },
+    {
+      "completion_length": 253.50000381469727,
+      "epoch": 4.333333333333333,
+      "grad_norm": 0.21704967319965363,
+      "kl": 0.000969748682109639,
+      "learning_rate": 4.745664346957361e-07,
+      "loss": 0.0,
+      "reward": 0.9164519011974335,
+      "reward_std": 0.5521265864372253,
+      "rewards/concensus_correctness_reward_func": 0.010416666977107525,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.21370190009474754,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5256666839122772,
+      "step": 26
+    },
+    {
+      "completion_length": 239.12500381469727,
+      "epoch": 4.666666666666667,
+      "grad_norm": 0.30419400334358215,
+      "kl": 0.001113584148697555,
+      "learning_rate": 4.695964991097616e-07,
+      "loss": 0.0,
+      "reward": 0.7989022731781006,
+      "reward_std": 0.25462955236434937,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.22277722135186195,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5761250033974648,
+      "step": 28
+    },
+    {
+      "completion_length": 248.54166793823242,
+      "epoch": 5.0,
+      "grad_norm": 0.2773168683052063,
+      "kl": 0.001008832361549139,
+      "learning_rate": 4.642142940418973e-07,
+      "loss": 0.0,
+      "reward": 1.0309449434280396,
+      "reward_std": 0.5466796979308128,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.2500000074505806,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.30559077858924866,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.4753541797399521,
+      "step": 30
+    },
+    {
+      "completion_length": 232.5833396911621,
+      "epoch": 5.333333333333333,
+      "grad_norm": 0.23351286351680756,
+      "kl": 0.0009582913771737367,
+      "learning_rate": 4.5842992401978256e-07,
+      "loss": 0.0,
+      "reward": 1.0708215832710266,
+      "reward_std": 0.29593051224946976,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2096132254227996,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6945417001843452,
+      "step": 32
+    },
+    {
+      "completion_length": 256.0,
+      "epoch": 5.666666666666667,
+      "grad_norm": 0.2183801531791687,
+      "kl": 0.0009512502729194239,
+      "learning_rate": 4.5225424859373684e-07,
+      "loss": 0.0,
+      "reward": 0.7803274691104889,
+      "reward_std": 0.5050432272255421,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.14595245476812124,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.4677083436399698,
+      "step": 34
+    },
+    {
+      "completion_length": 252.29166793823242,
+      "epoch": 6.0,
+      "grad_norm": 0.30777090787887573,
+      "kl": 0.001077517168596387,
+      "learning_rate": 4.456988619490889e-07,
+      "loss": 0.0,
+      "reward": 0.6541076973080635,
+      "reward_std": 0.23607416450977325,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.20950352400541306,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.44460418447852135,
+      "step": 36
+    },
+    {
+      "completion_length": 252.45833587646484,
+      "epoch": 6.333333333333333,
+      "grad_norm": 0.23970821499824524,
+      "kl": 0.0009026254410855472,
+      "learning_rate": 4.3877607113930516e-07,
+      "loss": 0.0,
+      "reward": 0.7649316936731339,
+      "reward_std": 0.6210936903953552,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.16676498856395483,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.514833353459835,
+      "step": 38
+    },
+    {
+      "completion_length": 249.25000381469727,
+      "epoch": 6.666666666666667,
+      "grad_norm": 0.2450600117444992,
+      "kl": 0.0009458353306399658,
+      "learning_rate": 4.314988729807827e-07,
+      "loss": 0.0,
+      "reward": 1.0382067561149597,
+      "reward_std": 0.6295727044343948,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.5,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.09437341336160898,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.4438333362340927,
+      "step": 40
+    },
+    {
+      "completion_length": 244.8333396911621,
+      "epoch": 7.0,
+      "grad_norm": 0.2953910529613495,
+      "kl": 0.0009223672968801111,
+      "learning_rate": 4.238809296526846e-07,
+      "loss": 0.0,
+      "reward": 1.0418670028448105,
+      "reward_std": 0.5014088153839111,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.22415870800614357,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6510416716337204,
+      "step": 42
+    },
+    {
+      "completion_length": 237.91666793823242,
+      "epoch": 7.333333333333333,
+      "grad_norm": 0.2422342747449875,
+      "kl": 0.0009755256760399789,
+      "learning_rate": 4.159365430476261e-07,
+      "loss": 0.0,
+      "reward": 0.8328549861907959,
+      "reward_std": 0.42648887261748314,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.10818830132484436,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5580000206828117,
+      "step": 44
+    },
+    {
+      "completion_length": 235.5833396911621,
+      "epoch": 7.666666666666667,
+      "grad_norm": 0.29116472601890564,
+      "kl": 0.0009618218464311212,
+      "learning_rate": 4.076806279213655e-07,
+      "loss": 0.0,
+      "reward": 0.8638769537210464,
+      "reward_std": 0.33969054371118546,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.16271025501191616,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6803333610296249,
+      "step": 46
+    },
+    {
+      "completion_length": 255.45833587646484,
+      "epoch": 8.0,
+      "grad_norm": 0.24302349984645844,
+      "kl": 0.0008998525445349514,
+      "learning_rate": 3.991286838919086e-07,
+      "loss": 0.0,
+      "reward": 0.7139967754483223,
+      "reward_std": 0.4492686055600643,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.13420509733259678,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.49645835161209106,
+      "step": 48
+    },
+    {
+      "completion_length": 251.25000381469727,
+      "epoch": 8.333333333333334,
+      "grad_norm": 0.2608531415462494,
+      "kl": 0.0009744187555043027,
+      "learning_rate": 3.902967663405956e-07,
+      "loss": 0.0,
+      "reward": 0.9686365872621536,
+      "reward_std": 0.43999840691685677,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.24667825177311897,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5552916750311852,
+      "step": 50
+    },
+    {
+      "completion_length": 241.20833587646484,
+      "epoch": 8.666666666666666,
+      "grad_norm": 0.30811837315559387,
+      "kl": 0.0010436592565383762,
+      "learning_rate": 3.8120145626980015e-07,
+      "loss": 0.0,
+      "reward": 0.9686652719974518,
+      "reward_std": 0.7263502180576324,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3276236057281494,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.45354165881872177,
+      "step": 52
+    },
+    {
+      "completion_length": 243.54166793823242,
+      "epoch": 9.0,
+      "grad_norm": 0.29206618666648865,
+      "kl": 0.0009239439386874437,
+      "learning_rate": 3.718598291738298e-07,
+      "loss": 0.0,
+      "reward": 1.0735424309968948,
+      "reward_std": 0.7325034886598587,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.125,
+      "rewards/question_recreation_reward_func": 0.19197986833751202,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.6315625160932541,
+      "step": 54
+    },
+    {
+      "completion_length": 237.00000381469727,
+      "epoch": 9.333333333333334,
+      "grad_norm": 0.2532391846179962,
+      "kl": 0.0010295536485500634,
+      "learning_rate": 3.622894229814698e-07,
+      "loss": 0.0,
+      "reward": 1.081029787659645,
+      "reward_std": 0.3188396841287613,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.25,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.21769641619175673,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5925000235438347,
+      "step": 56
+    },
+    {
+      "completion_length": 231.12500381469727,
+      "epoch": 9.666666666666666,
+      "grad_norm": 0.28410038352012634,
+      "kl": 0.0010245468729408458,
+      "learning_rate": 3.52508205130354e-07,
+      "loss": 0.0,
+      "reward": 0.8574095070362091,
+      "reward_std": 0.4760412946343422,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2542011570185423,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.43654169887304306,
+      "step": 58
+    },
+    {
+      "completion_length": 249.87500381469727,
+      "epoch": 10.0,
+      "grad_norm": 0.2931140661239624,
+      "kl": 0.001108055905206129,
+      "learning_rate": 3.4253453883497864e-07,
+      "loss": 0.0,
+      "reward": 0.7035448998212814,
+      "reward_std": 0.463116317987442,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.18133653653785586,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5222083479166031,
+      "step": 60
+    },
+    {
+      "completion_length": 249.4166717529297,
+      "epoch": 10.333333333333334,
+      "grad_norm": 0.2339998483657837,
+      "kl": 0.0010156262433156371,
+      "learning_rate": 3.323871486116851e-07,
+      "loss": 0.0,
+      "reward": 1.0926494300365448,
+      "reward_std": 0.8358982801437378,
+      "rewards/concensus_correctness_reward_func": 0.010416666977107525,
+      "rewards/consensus_reward_func": 0.3333333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2077327612787485,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5411666706204414,
+      "step": 62
+    },
+    {
+      "completion_length": 241.125,
+      "epoch": 10.666666666666666,
+      "grad_norm": 0.2425372451543808,
+      "kl": 0.0010399544844403863,
+      "learning_rate": 3.220850851253377e-07,
+      "loss": 0.0,
+      "reward": 0.6774226725101471,
+      "reward_std": 0.15603318624198437,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.10921429842710495,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5682083368301392,
+      "step": 64
+    },
+    {
+      "completion_length": 251.91666793823242,
+      "epoch": 11.0,
+      "grad_norm": 0.23354342579841614,
+      "kl": 0.0008322933135787025,
+      "learning_rate": 3.1164768942369053e-07,
+      "loss": 0.0,
+      "reward": 1.1510787606239319,
+      "reward_std": 0.351579736918211,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3893912173807621,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5741875097155571,
+      "step": 66
+    },
+    {
+      "completion_length": 250.6666717529297,
+      "epoch": 11.333333333333334,
+      "grad_norm": 0.23890496790409088,
+      "kl": 0.0008501187694491819,
+      "learning_rate": 3.010945566265912e-07,
+      "loss": 0.0,
+      "reward": 1.362295776605606,
+      "reward_std": 0.8761751502752304,
+      "rewards/concensus_correctness_reward_func": 0.04475000128149986,
+      "rewards/consensus_reward_func": 0.3333333432674408,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2828373648226261,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.06250000186264515,
+      "rewards/xmlcount_reward_func": 0.6388750001788139,
+      "step": 68
+    },
+    {
+      "completion_length": 237.7916717529297,
+      "epoch": 11.666666666666666,
+      "grad_norm": 0.32228681445121765,
+      "kl": 0.0010085924150189385,
+      "learning_rate": 2.9044549913819124e-07,
+      "loss": 0.0,
+      "reward": 0.854033350944519,
+      "reward_std": 0.45660512894392014,
+      "rewards/concensus_correctness_reward_func": 0.08166667353361845,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2169082798063755,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.38879168033599854,
+      "step": 70
+    },
+    {
+      "completion_length": 244.2291717529297,
+      "epoch": 12.0,
+      "grad_norm": 0.34781065583229065,
+      "kl": 0.0010221569682471454,
+      "learning_rate": 2.797205094512266e-07,
+      "loss": 0.0,
+      "reward": 0.763754477724433,
+      "reward_std": 0.5146888792514801,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.125,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1701919911429286,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.46856251172721386,
+      "step": 72
+    },
+    {
+      "completion_length": 250.5416717529297,
+      "epoch": 12.333333333333334,
+      "grad_norm": 0.26079338788986206,
+      "kl": 0.0009046219201991335,
+      "learning_rate": 2.6893972261320264e-07,
+      "loss": 0.0,
+      "reward": 1.0650671422481537,
+      "reward_std": 0.72061388194561,
+      "rewards/concensus_correctness_reward_func": 0.04475000128149986,
+      "rewards/consensus_reward_func": 0.2500000074505806,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3010254316031933,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.46929167956113815,
+      "step": 74
+    },
+    {
+      "completion_length": 238.79166793823242,
+      "epoch": 12.666666666666666,
+      "grad_norm": 0.25841179490089417,
+      "kl": 0.0009440975554753095,
+      "learning_rate": 2.5812337842494516e-07,
+      "loss": 0.0,
+      "reward": 0.9689909741282463,
+      "reward_std": 0.5216221660375595,
+      "rewards/concensus_correctness_reward_func": 0.010416666977107525,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.25732426159083843,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.5762500017881393,
+      "step": 76
+    },
+    {
+      "completion_length": 247.08333587646484,
+      "epoch": 13.0,
+      "grad_norm": 0.295762836933136,
+      "kl": 0.0008411942981183529,
+      "learning_rate": 2.4729178344249006e-07,
+      "loss": 0.0,
+      "reward": 0.7745198309421539,
+      "reward_std": 0.40021975338459015,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.17797815427184105,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.03125,
+      "rewards/xmlcount_reward_func": 0.565291665494442,
+      "step": 78
+    },
+    {
+      "completion_length": 241.125,
+      "epoch": 13.333333333333334,
+      "grad_norm": 0.2495446801185608,
+      "kl": 0.0009230001014657319,
+      "learning_rate": 2.3646527285364563e-07,
+      "loss": 0.0,
+      "reward": 0.8283002823591232,
+      "reward_std": 0.3567834421992302,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3017169274389744,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5265833511948586,
+      "step": 80
+    },
+    {
+      "completion_length": 240.75000381469727,
+      "epoch": 13.666666666666666,
+      "grad_norm": 0.23679904639720917,
+      "kl": 0.0009740922250784934,
+      "learning_rate": 2.256641723008026e-07,
+      "loss": 0.0,
+      "reward": 0.9943374246358871,
+      "reward_std": 0.5632965341210365,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.2500000074505806,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.20533735305070877,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.539000004529953,
+      "step": 82
+    },
+    {
+      "completion_length": 245.37500381469727,
+      "epoch": 14.0,
+      "grad_norm": 0.27654239535331726,
+      "kl": 0.0010249131155433133,
+      "learning_rate": 2.1490875972166393e-07,
+      "loss": 0.0,
+      "reward": 0.870885506272316,
+      "reward_std": 0.37735576182603836,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.20246883248910308,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.48091667145490646,
+      "step": 84
+    },
+    {
+      "completion_length": 239.6666717529297,
+      "epoch": 14.333333333333334,
+      "grad_norm": 0.2376907616853714,
+      "kl": 0.0009535926801618189,
+      "learning_rate": 2.0421922727953595e-07,
+      "loss": 0.0,
+      "reward": 0.6692031770944595,
+      "reward_std": 0.3753516748547554,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.11624482274055481,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5321250110864639,
+      "step": 86
+    },
+    {
+      "completion_length": 239.54166793823242,
+      "epoch": 14.666666666666666,
+      "grad_norm": 0.33387479186058044,
+      "kl": 0.001069716177880764,
+      "learning_rate": 1.9361564345465145e-07,
+      "loss": 0.0,
+      "reward": 0.9065132290124893,
+      "reward_std": 0.43948036059737206,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.24497149419039488,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5573750212788582,
+      "step": 88
+    },
+    {
+      "completion_length": 251.87500381469727,
+      "epoch": 15.0,
+      "grad_norm": 10.691461563110352,
+      "kl": 0.0026971843908540905,
+      "learning_rate": 1.8311791536769483e-07,
+      "loss": 0.0,
+      "reward": 0.8780257403850555,
+      "reward_std": 0.35077016800642014,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.15740068443119526,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5539583414793015,
+      "step": 90
+    },
+    {
+      "completion_length": 237.12500381469727,
+      "epoch": 15.333333333333334,
+      "grad_norm": 0.2563934922218323,
+      "kl": 0.001029187420499511,
+      "learning_rate": 1.7274575140626315e-07,
+      "loss": 0.0,
+      "reward": 1.044521152973175,
+      "reward_std": 0.6668648347258568,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.4166666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.18302111141383648,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.44483335316181183,
+      "step": 92
+    },
+    {
+      "completion_length": 250.37500381469727,
+      "epoch": 15.666666666666666,
+      "grad_norm": 0.19243191182613373,
+      "kl": 0.0009831937932176515,
+      "learning_rate": 1.6251862422442788e-07,
+      "loss": 0.0,
+      "reward": 0.9626381397247314,
+      "reward_std": 0.48763278126716614,
+      "rewards/concensus_correctness_reward_func": 0.0663333348929882,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.22267980594187975,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5902916640043259,
+      "step": 94
+    },
+    {
+      "completion_length": 247.62500381469727,
+      "epoch": 16.0,
+      "grad_norm": 0.31039178371429443,
+      "kl": 0.0009879204735625535,
+      "learning_rate": 1.5245573418486135e-07,
+      "loss": 0.0,
+      "reward": 0.9554285407066345,
+      "reward_std": 0.14843177422881126,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2345951907336712,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.7208333611488342,
+      "step": 96
+    },
+    {
+      "completion_length": 246.91666793823242,
+      "epoch": 16.333333333333332,
+      "grad_norm": 0.25648194551467896,
+      "kl": 0.0009462093003094196,
+      "learning_rate": 1.4257597331216208e-07,
+      "loss": 0.0,
+      "reward": 0.8551813066005707,
+      "reward_std": 0.42093783617019653,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1955562625080347,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6387916654348373,
+      "step": 98
+    },
+    {
+      "completion_length": 246.91666793823242,
+      "epoch": 16.666666666666668,
+      "grad_norm": 0.2276257574558258,
+      "kl": 0.0008919332904042676,
+      "learning_rate": 1.328978898250525e-07,
+      "loss": 0.0,
+      "reward": 1.0227466598153114,
+      "reward_std": 0.5311935096979141,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.25,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.26124666444957256,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.4906666874885559,
+      "step": 100
+    },
+    {
+      "completion_length": 230.1666717529297,
+      "epoch": 17.0,
+      "grad_norm": 0.28276175260543823,
+      "kl": 0.0009856357210082933,
+      "learning_rate": 1.234396533140365e-07,
+      "loss": 0.0,
+      "reward": 0.8042420297861099,
+      "reward_std": 0.5668299272656441,
+      "rewards/concensus_correctness_reward_func": 0.010416666977107525,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.04959618765860796,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5567291676998138,
+      "step": 102
+    },
+    {
+      "completion_length": 252.95833587646484,
+      "epoch": 17.333333333333332,
+      "grad_norm": 0.24145229160785675,
+      "kl": 0.0010389972885604948,
+      "learning_rate": 1.1421902062989178e-07,
+      "loss": 0.0,
+      "reward": 0.8800064772367477,
+      "reward_std": 0.4219862353056669,
+      "rewards/concensus_correctness_reward_func": 0.010416666977107525,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.12479815538972616,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.661458358168602,
+      "step": 104
+    },
+    {
+      "completion_length": 238.2083396911621,
+      "epoch": 17.666666666666668,
+      "grad_norm": 0.2690638303756714,
+      "kl": 0.0009289942245231941,
+      "learning_rate": 1.0525330254703788e-07,
+      "loss": 0.0,
+      "reward": 1.0552653521299362,
+      "reward_std": 0.47432762011885643,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.3333333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1660986514762044,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5558333694934845,
+      "step": 106
+    },
+    {
+      "completion_length": 233.02083587646484,
+      "epoch": 18.0,
+      "grad_norm": 2.029247522354126,
+      "kl": 0.0012233417219249532,
+      "learning_rate": 9.655933126436563e-08,
+      "loss": 0.0,
+      "reward": 0.8586884886026382,
+      "reward_std": 0.41075659915804863,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.125,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.17650096770375967,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.515520840883255,
+      "step": 108
+    },
+    {
+      "completion_length": 255.33333587646484,
+      "epoch": 18.333333333333332,
+      "grad_norm": 0.2518994212150574,
+      "kl": 0.0009219197236234322,
+      "learning_rate": 8.81534288045431e-08,
+      "loss": 0.0,
+      "reward": 0.8517690300941467,
+      "reward_std": 0.5377568677067757,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.2500000074505806,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.14772731950506568,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.45404166728258133,
+      "step": 110
+    },
+    {
+      "completion_length": 230.91666793823242,
+      "epoch": 18.666666666666668,
+      "grad_norm": 0.44778749346733093,
+      "kl": 0.0011147863697260618,
+      "learning_rate": 8.005137637112302e-08,
+      "loss": 0.0,
+      "reward": 0.8284278735518456,
+      "reward_std": 0.5220160707831383,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.17305286042392254,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.551208358258009,
+      "step": 112
+    },
+    {
+      "completion_length": 248.52083587646484,
+      "epoch": 19.0,
+      "grad_norm": 0.27239030599594116,
+      "kl": 0.0010450754780322313,
+      "learning_rate": 7.226838472098237e-08,
+      "loss": 0.0,
+      "reward": 1.0394659638404846,
+      "reward_std": 0.39139912091195583,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.27873678505420685,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6773958578705788,
+      "step": 114
+    },
+    {
+      "completion_length": 247.12500381469727,
+      "epoch": 19.333333333333332,
+      "grad_norm": 0.2705827057361603,
+      "kl": 0.000944676372455433,
+      "learning_rate": 6.481906560771524e-08,
+      "loss": 0.0,
+      "reward": 1.1934748440980911,
+      "reward_std": 0.6195695661008358,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.4166666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.22809979785233736,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5278749987483025,
+      "step": 116
+    },
+    {
+      "completion_length": 244.16666793823242,
+      "epoch": 19.666666666666668,
+      "grad_norm": 0.3459849953651428,
+      "kl": 0.001006613762001507,
+      "learning_rate": 5.771740434959277e-08,
+      "loss": 0.0,
+      "reward": 0.9170785397291183,
+      "reward_std": 0.30866917595267296,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.21616187877953053,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6800833642482758,
+      "step": 118
+    },
+    {
+      "completion_length": 248.79166793823242,
       "epoch": 20.0,
+      "grad_norm": 0.28259551525115967,
+      "kl": 0.0010699233243940398,
+      "learning_rate": 5.097673357358906e-08,
+      "loss": 0.0,
+      "reward": 0.7739865928888321,
+      "reward_std": 0.2533000372350216,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1705698948353529,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6034166738390923,
+      "step": 120
+    },
+    {
+      "completion_length": 241.5416717529297,
+      "epoch": 20.333333333333332,
+      "grad_norm": 0.25206458568573,
+      "kl": 0.0010052528668893501,
+      "learning_rate": 4.460970818476717e-08,
+      "loss": 0.0,
+      "reward": 1.283503606915474,
+      "reward_std": 0.740435317158699,
+      "rewards/concensus_correctness_reward_func": 0.010416666977107525,
+      "rewards/consensus_reward_func": 0.3333333432674408,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2532535642385483,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.6448333784937859,
+      "step": 122
+    },
+    {
+      "completion_length": 245.50000762939453,
+      "epoch": 20.666666666666668,
+      "grad_norm": 0.2550675868988037,
+      "kl": 0.0010486233804840595,
+      "learning_rate": 3.8628281608017065e-08,
+      "loss": 0.0,
+      "reward": 0.7773748189210892,
+      "reward_std": 0.3785080909729004,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2198331318795681,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5575416684150696,
+      "step": 124
+    },
+    {
+      "completion_length": 252.125,
+      "epoch": 21.0,
+      "grad_norm": 0.25815054774284363,
+      "kl": 0.0008713365532457829,
+      "learning_rate": 3.304368334674965e-08,
+      "loss": 0.0,
+      "reward": 0.9102632999420166,
+      "reward_std": 0.3166055828332901,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.22522159665822983,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5183750167489052,
+      "step": 126
+    },
+    {
+      "completion_length": 254.0,
+      "epoch": 21.333333333333332,
+      "grad_norm": 0.19974423944950104,
+      "kl": 0.0009994291613111272,
+      "learning_rate": 2.7866397900677185e-08,
+      "loss": 0.0,
+      "reward": 0.9313755556941032,
+      "reward_std": 0.47402798384428024,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0833333358168602,
+      "rewards/question_recreation_reward_func": 0.2305004969239235,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5342083647847176,
+      "step": 128
+    },
+    {
+      "completion_length": 236.75000381469727,
+      "epoch": 21.666666666666668,
+      "grad_norm": 0.37588945031166077,
+      "kl": 0.0010261698334943503,
+      "learning_rate": 2.3106145082260774e-08,
+      "loss": 0.0,
+      "reward": 0.9061668962240219,
+      "reward_std": 0.5161278247833252,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.18883355520665646,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.6756666973233223,
+      "step": 130
+    },
+    {
+      "completion_length": 240.95833587646484,
+      "epoch": 22.0,
+      "grad_norm": 0.33937975764274597,
+      "kl": 0.0009526331414235756,
+      "learning_rate": 1.877186176877779e-08,
+      "loss": 0.0,
+      "reward": 0.7365991473197937,
+      "reward_std": 0.3077675364911556,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3288283236324787,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.4077708423137665,
+      "step": 132
+    },
+    {
+      "completion_length": 241.41666793823242,
+      "epoch": 22.333333333333332,
+      "grad_norm": 0.2997651994228363,
+      "kl": 0.0010459028271725401,
+      "learning_rate": 1.4871685124269007e-08,
+      "loss": 0.0,
+      "reward": 1.064703717827797,
+      "reward_std": 0.7619387805461884,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.3333333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.15341205522418022,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5571250021457672,
+      "step": 134
+    },
+    {
+      "completion_length": 237.54167556762695,
+      "epoch": 22.666666666666668,
+      "grad_norm": 0.2647048532962799,
+      "kl": 0.0010394936834927648,
+      "learning_rate": 1.141293732286297e-08,
+      "loss": 0.0,
+      "reward": 0.7999026626348495,
+      "reward_std": 0.673773281276226,
+      "rewards/concensus_correctness_reward_func": 0.0403333343565464,
+      "rewards/consensus_reward_func": 0.1666666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.06627767626196146,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5266250036656857,
+      "step": 136
+    },
+    {
+      "completion_length": 248.08333587646484,
+      "epoch": 23.0,
+      "grad_norm": 0.24097177386283875,
+      "kl": 0.0007808299997122958,
+      "learning_rate": 8.402111802159412e-09,
+      "loss": 0.0,
+      "reward": 0.8799475729465485,
+      "reward_std": 0.412565840408206,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2921558953821659,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5877916663885117,
+      "step": 138
+    },
+    {
+      "completion_length": 229.5416717529297,
+      "epoch": 23.333333333333332,
+      "grad_norm": 0.21611253917217255,
+      "kl": 0.0010441301274113357,
+      "learning_rate": 5.844861072478335e-09,
+      "loss": 0.0,
+      "reward": 1.2775923311710358,
+      "reward_std": 0.6640586405992508,
+      "rewards/concensus_correctness_reward_func": 0.014833333902060986,
+      "rewards/consensus_reward_func": 0.416666679084301,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.22671727556735277,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5985416918992996,
+      "step": 140
+    },
+    {
+      "completion_length": 252.29166793823242,
+      "epoch": 23.666666666666668,
+      "grad_norm": 0.25495585799217224,
+      "kl": 0.0009842409199336544,
+      "learning_rate": 3.745986104862903e-09,
+      "loss": 0.0,
+      "reward": 1.1931225955486298,
+      "reward_std": 1.0680770799517632,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.4166666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.17362259328365326,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5820000022649765,
+      "step": 142
+    },
+    {
+      "completion_length": 235.29166793823242,
+      "epoch": 24.0,
+      "grad_norm": 0.25490984320640564,
+      "kl": 0.0009585395600879565,
+      "learning_rate": 2.1094273177576505e-09,
+      "loss": 0.0,
+      "reward": 1.3230770230293274,
+      "reward_std": 0.5727710947394371,
+      "rewards/concensus_correctness_reward_func": 0.03125,
+      "rewards/consensus_reward_func": 0.4166666716337204,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.27359780576080084,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5807291641831398,
+      "step": 144
+    },
+    {
+      "completion_length": 245.08333587646484,
+      "epoch": 24.333333333333332,
+      "grad_norm": 0.24701599776744843,
+      "kl": 0.0010237596870865673,
+      "learning_rate": 9.38257179284696e-10,
+      "loss": 0.0,
+      "reward": 1.1041172742843628,
+      "reward_std": 0.3621603213250637,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.2500000074505806,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.20274224504828453,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6513750404119492,
+      "step": 146
+    },
+    {
+      "completion_length": 239.62500381469727,
+      "epoch": 24.666666666666668,
+      "grad_norm": 0.2657899260520935,
+      "kl": 0.0011027540458599105,
+      "learning_rate": 2.3467443900582197e-10,
+      "loss": 0.0,
+      "reward": 0.9250312894582748,
+      "reward_std": 0.40416907519102097,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0833333358168602,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.18794792797416449,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6329167112708092,
+      "step": 148
+    },
+    {
+      "completion_length": 233.75000381469727,
+      "epoch": 25.0,
+      "grad_norm": 0.2480180859565735,
+      "kl": 0.000958686214289628,
+      "learning_rate": 0.0,
+      "loss": 0.0,
+      "reward": 0.9879210442304611,
+      "reward_std": 0.1771667394787073,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0,
+      "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2838585525751114,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.7040625363588333,
+      "step": 150
+    },
+    {
+      "epoch": 25.0,
+      "step": 150,
       "total_flos": 0.0,
+      "train_loss": 4.005705436611606e-06,
+      "train_runtime": 2512.4216,
+      "train_samples_per_second": 0.716,
+      "train_steps_per_second": 0.06
     }
   ],
   "logging_steps": 2,
+  "max_steps": 150,
   "num_input_tokens_seen": 0,
+  "num_train_epochs": 25,
+  "save_steps": 150,
   "stateful_callbacks": {
     "TrainerControl": {
       "args": {
     }
   },
   "total_flos": 0.0,
+  "train_batch_size": 6,
   "trial_name": null,
   "trial_params": null
 }

training_args.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f4526a97de704473a2261977f41e39eddffefeb46c4b4b60154d8a49f9dd2c20
 size 5944

 version https://git-lfs.github.com/spec/v1
+oid sha256:398e7787674623c0106e25a413b941a6dafdb584fe48bbd5293f71aa5461a1c5
 size 5944