Instructions to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret")
model = AutoModelForCausalLM.from_pretrained("moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret

SGLang

How to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Studio

How to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret to start chatting

Load model with FastModel

pip install unsloth
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
    model_name="moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret",
    max_seq_length=2048,
)

Docker Model Runner
How to use moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret with Docker Model Runner:
```
docker model run hf.co/moscowx21/Qwen2.5-1.5B-Instruct-Gensyn-Swarm-giant_pale_ferret
```

aawwssdddwdwd commited on Jun 25, 2025

Commit

8467f5e

verified ·

1 Parent(s): 79227a3

End of training

Browse files

Files changed (4) hide show

adapter_model.safetensors +1 -1
all_results.json +4 -4
train_results.json +4 -4
trainer_state.json +754 -754

adapter_model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:43c488745c1fe5c334addb6e9764ad981737748ea3ef7dbc10ebb4814e78240c
 size 73911112

 version https://git-lfs.github.com/spec/v1
+oid sha256:3e8b3399526b8b8832cd9dc9e756062cb2eabbf9e812a41ff214417266a04f11
 size 73911112

all_results.json CHANGED Viewed

@@ -1,8 +1,8 @@
 {
     "total_flos": 0.0,
-    "train_loss": 0.00018999547231942414,
-    "train_runtime": 2785.5958,
     "train_samples": 35,
-    "train_samples_per_second": 0.646,
-    "train_steps_per_second": 0.054
 }

 {
     "total_flos": 0.0,
+    "train_loss": 0.00020825650698194902,
+    "train_runtime": 2537.33,
     "train_samples": 35,
+    "train_samples_per_second": 0.709,
+    "train_steps_per_second": 0.059
 }

train_results.json CHANGED Viewed

@@ -1,8 +1,8 @@
 {
     "total_flos": 0.0,
-    "train_loss": 0.00018999547231942414,
-    "train_runtime": 2785.5958,
     "train_samples": 35,
-    "train_samples_per_second": 0.646,
-    "train_steps_per_second": 0.054
 }

 {
     "total_flos": 0.0,
+    "train_loss": 0.00020825650698194902,
+    "train_runtime": 2537.33,
     "train_samples": 35,
+    "train_samples_per_second": 0.709,
+    "train_steps_per_second": 0.059
 }

trainer_state.json CHANGED Viewed

@@ -9,1438 +9,1438 @@
   "is_world_process_zero": true,
   "log_history": [
     {
-      "completion_length": 232.41666793823242,
       "epoch": 0.3333333333333333,
-      "grad_norm": 0.3317122459411621,
-      "kl": 0.03968494385480881,
       "learning_rate": 2e-07,
       "loss": 0.0002,
-      "reward": 1.2943192273378372,
-      "reward_std": 0.7113365903496742,
-      "rewards/concensus_correctness_reward_func": 0.09375,
-      "rewards/consensus_reward_func": 0.25,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.31394420750439167,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.5949583500623703,
       "step": 2
     },
     {
-      "completion_length": 243.1666717529297,
       "epoch": 0.6666666666666666,
-      "grad_norm": 0.2442961037158966,
-      "kl": 0.04297630675137043,
       "learning_rate": 4e-07,
-      "loss": 0.0002,
-      "reward": 2.2069587111473083,
-      "reward_std": 1.210971087217331,
-      "rewards/concensus_correctness_reward_func": 0.5042500048875809,
-      "rewards/consensus_reward_func": 0.7500000298023224,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2504170313477516,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.6814583092927933,
       "step": 4
     },
     {
-      "completion_length": 227.75000381469727,
       "epoch": 1.0,
-      "grad_norm": 0.3365265429019928,
-      "kl": 0.04355957265943289,
       "learning_rate": 4.99941324504621e-07,
       "loss": 0.0002,
-      "reward": 1.3837293833494186,
-      "reward_std": 0.6235679462552071,
-      "rewards/concensus_correctness_reward_func": 0.05112500116229057,
-      "rewards/consensus_reward_func": 0.2916666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.32508353143930435,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.6741875112056732,
       "step": 6
     },
     {
-      "completion_length": 237.6666717529297,
       "epoch": 1.3333333333333333,
-      "grad_norm": 0.4599948525428772,
-      "kl": 0.046385033056139946,
       "learning_rate": 4.99472085783721e-07,
       "loss": 0.0002,
-      "reward": 1.737962007522583,
-      "reward_std": 0.8370491340756416,
-      "rewards/concensus_correctness_reward_func": 0.3463333398103714,
-      "rewards/consensus_reward_func": 0.416666679084301,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.24967025592923164,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.6836250275373459,
       "step": 8
     },
     {
-      "completion_length": 220.0833396911621,
       "epoch": 1.6666666666666665,
-      "grad_norm": 0.45267096161842346,
-      "kl": 0.05444894265383482,
       "learning_rate": 4.985344892885899e-07,
       "loss": 0.0002,
-      "reward": 1.4824041426181793,
-      "reward_std": 0.7231063693761826,
-      "rewards/concensus_correctness_reward_func": 0.09416666720062494,
-      "rewards/consensus_reward_func": 0.416666679084301,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.3029874712228775,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.6477500349283218,
       "step": 10
     },
     {
-      "completion_length": 244.29166793823242,
       "epoch": 2.0,
-      "grad_norm": 0.3539784252643585,
-      "kl": 0.0389225366525352,
       "learning_rate": 4.971302952586796e-07,
       "loss": 0.0002,
-      "reward": 1.1549717336893082,
-      "reward_std": 0.6939718946814537,
-      "rewards/concensus_correctness_reward_func": 0.07791666686534882,
-      "rewards/consensus_reward_func": 0.1666666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.22690916247665882,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.6834791749715805,
       "step": 12
     },
     {
-      "completion_length": 244.16666793823242,
       "epoch": 2.3333333333333335,
-      "grad_norm": 0.3303356170654297,
-      "kl": 0.037933843676000834,
       "learning_rate": 4.952621399215597e-07,
       "loss": 0.0002,
-      "reward": 1.5575617253780365,
-      "reward_std": 0.8793091028928757,
-      "rewards/concensus_correctness_reward_func": 0.16466667037457228,
-      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.3069366551935673,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.6692917048931122,
       "step": 14
     },
     {
-      "completion_length": 227.58333587646484,
       "epoch": 2.6666666666666665,
-      "grad_norm": 0.3340577185153961,
-      "kl": 0.04321099817752838,
       "learning_rate": 4.929335305436764e-07,
       "loss": 0.0002,
-      "reward": 1.5347478687763214,
-      "reward_std": 1.0792951248586178,
-      "rewards/concensus_correctness_reward_func": 0.27666667103767395,
-      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.20920615270733833,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.6947083473205566,
       "step": 16
     },
     {
-      "completion_length": 228.87500762939453,
       "epoch": 3.0,
-      "grad_norm": 0.3193712532520294,
-      "kl": 0.05019127577543259,
       "learning_rate": 4.901488388458247e-07,
       "loss": 0.0002,
-      "reward": 2.0944919288158417,
-      "reward_std": 0.825613871216774,
-      "rewards/concensus_correctness_reward_func": 0.24100000225007534,
-      "rewards/consensus_reward_func": 0.6250000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.27528344467282295,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.9115417003631592,
       "step": 18
     },
     {
-      "completion_length": 219.33333587646484,
       "epoch": 3.3333333333333335,
-      "grad_norm": 0.41362255811691284,
-      "kl": 0.05465732142329216,
       "learning_rate": 4.869132927957006e-07,
-      "loss": 0.0002,
-      "reward": 1.842283844947815,
-      "reward_std": 1.4312597960233688,
-      "rewards/concensus_correctness_reward_func": 0.4110833369195461,
-      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.1814087089151144,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.06250000186264515,
-      "rewards/xmlcount_reward_func": 0.7706250250339508,
       "step": 20
     },
     {
-      "completion_length": 241.91666793823242,
       "epoch": 3.6666666666666665,
-      "grad_norm": 0.28995105624198914,
-      "kl": 0.03528741002082825,
       "learning_rate": 4.832329667929376e-07,
-      "loss": 0.0001,
-      "reward": 1.3167436867952347,
-      "reward_std": 0.9163419529795647,
-      "rewards/concensus_correctness_reward_func": 0.14691667072474957,
-      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.15841034753248096,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.6780833750963211,
       "step": 22
     },
     {
-      "completion_length": 225.14583587646484,
       "epoch": 4.0,
-      "grad_norm": 0.2544354498386383,
-      "kl": 0.06209309957921505,
       "learning_rate": 4.791147702650565e-07,
       "loss": 0.0002,
-      "reward": 2.206490620970726,
-      "reward_std": 0.9131987206637859,
-      "rewards/concensus_correctness_reward_func": 0.4676249846816063,
-      "rewards/consensus_reward_func": 0.5416666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.30417799577116966,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.06250000186264515,
-      "rewards/xmlcount_reward_func": 0.8305208384990692,
       "step": 24
     },
     {
-      "completion_length": 230.0833396911621,
       "epoch": 4.333333333333333,
-      "grad_norm": 0.2742597758769989,
-      "kl": 0.05144475866109133,
       "learning_rate": 4.745664346957361e-07,
       "loss": 0.0002,
-      "reward": 1.4815124720335007,
-      "reward_std": 0.7737281620502472,
-      "rewards/concensus_correctness_reward_func": 0.14433333277702332,
-      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.20942904613912106,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.6902500241994858,
       "step": 26
     },
     {
-      "completion_length": 245.4166717529297,
       "epoch": 4.666666666666667,
-      "grad_norm": 0.29059553146362305,
-      "kl": 0.03980268817394972,
       "learning_rate": 4.695964991097616e-07,
       "loss": 0.0002,
-      "reward": 1.1801403760910034,
-      "reward_std": 0.6059232130646706,
-      "rewards/concensus_correctness_reward_func": 0.13983333855867386,
-      "rewards/consensus_reward_func": 0.0833333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.25168194994330406,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.6844583451747894,
       "step": 28
     },
     {
-      "completion_length": 217.89583587646484,
       "epoch": 5.0,
-      "grad_norm": 0.36589425802230835,
-      "kl": 0.05048558861017227,
       "learning_rate": 4.642142940418973e-07,
       "loss": 0.0002,
-      "reward": 1.6932251751422882,
-      "reward_std": 1.0846444517374039,
-      "rewards/concensus_correctness_reward_func": 0.18604166340082884,
-      "rewards/consensus_reward_func": 0.3750000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2988292761147022,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.791687548160553,
       "step": 30
     },
     {
-      "completion_length": 241.75000381469727,
       "epoch": 5.333333333333333,
-      "grad_norm": 0.3060760200023651,
-      "kl": 0.04335446655750275,
       "learning_rate": 4.5842992401978256e-07,
-      "loss": 0.0002,
-      "reward": 1.8897629380226135,
-      "reward_std": 1.5923044234514236,
-      "rewards/concensus_correctness_reward_func": 0.387750007212162,
-      "rewards/consensus_reward_func": 0.5000000074505806,
       "rewards/cumulative_reward_2": 0.0,
-      "rewards/final_correctness_reward_func": 0.0833333358168602,
-      "rewards/question_recreation_reward_func": 0.16076288698241115,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.716250017285347,
       "step": 32
     },
     {
-      "completion_length": 230.66666793823242,
       "epoch": 5.666666666666667,
-      "grad_norm": 0.34271323680877686,
-      "kl": 0.0460484242066741,
       "learning_rate": 4.5225424859373684e-07,
       "loss": 0.0002,
-      "reward": 1.4525684118270874,
-      "reward_std": 0.5029263757169247,
-      "rewards/concensus_correctness_reward_func": 0.15141667425632477,
-      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2570683788508177,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.7732500582933426,
       "step": 34
     },
     {
-      "completion_length": 213.41666793823242,
       "epoch": 6.0,
-      "grad_norm": 0.4838489592075348,
-      "kl": 0.05409040488302708,
       "learning_rate": 4.456988619490889e-07,
       "loss": 0.0002,
-      "reward": 1.2900775969028473,
-      "reward_std": 0.576139360666275,
-      "rewards/concensus_correctness_reward_func": 0.110083332285285,
-      "rewards/consensus_reward_func": 0.1666666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.27393175289034843,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.718562513589859,
       "step": 36
     },
     {
-      "completion_length": 238.8333396911621,
       "epoch": 6.333333333333333,
-      "grad_norm": 0.3219103217124939,
-      "kl": 0.042892820201814175,
       "learning_rate": 4.3877607113930516e-07,
       "loss": 0.0002,
-      "reward": 1.236423820257187,
-      "reward_std": 0.4186003617942333,
-      "rewards/concensus_correctness_reward_func": 0.048416669480502605,
-      "rewards/consensus_reward_func": 0.25,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.24634044617414474,
-      "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.6916666775941849,
       "step": 38
     },
     {
-      "completion_length": 239.00000762939453,
       "epoch": 6.666666666666667,
-      "grad_norm": 0.34468311071395874,
-      "kl": 0.04395328648388386,
       "learning_rate": 4.314988729807827e-07,
       "loss": 0.0002,
-      "reward": 2.180278718471527,
-      "reward_std": 1.3632176369428635,
-      "rewards/concensus_correctness_reward_func": 0.4166666818782687,
-      "rewards/consensus_reward_func": 0.6666666865348816,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.15390371531248093,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.06250000186264515,
-      "rewards/xmlcount_reward_func": 0.8805417120456696,
       "step": 40
     },
     {
-      "completion_length": 236.9791717529297,
       "epoch": 7.0,
-      "grad_norm": 0.3818952143192291,
-      "kl": 0.04233813192695379,
       "learning_rate": 4.238809296526846e-07,
-      "loss": 0.0002,
-      "reward": 2.0933502912521362,
-      "reward_std": 0.6345569714903831,
-      "rewards/concensus_correctness_reward_func": 0.5274166613817215,
-      "rewards/consensus_reward_func": 0.6666666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.20128780230879784,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.6979791820049286,
       "step": 42
     },
     {
-      "completion_length": 218.4583396911621,
       "epoch": 7.333333333333333,
-      "grad_norm": 0.5095331072807312,
-      "kl": 0.057227155193686485,
       "learning_rate": 4.159365430476261e-07,
       "loss": 0.0002,
-      "reward": 1.3975305408239365,
-      "reward_std": 0.5227701887488365,
-      "rewards/concensus_correctness_reward_func": 0.11116667091846466,
-      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.18219718150794506,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.8125000149011612,
       "step": 44
     },
     {
-      "completion_length": 210.70833587646484,
       "epoch": 7.666666666666667,
-      "grad_norm": 0.4629160463809967,
-      "kl": 0.06075780279934406,
       "learning_rate": 4.076806279213655e-07,
       "loss": 0.0002,
-      "reward": 1.4839744865894318,
-      "reward_std": 1.1272654682397842,
-      "rewards/concensus_correctness_reward_func": 0.2153333341702819,
       "rewards/consensus_reward_func": 0.416666679084301,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.20697450265288353,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.6241666749119759,
       "step": 46
     },
     {
-      "completion_length": 239.83333587646484,
       "epoch": 8.0,
-      "grad_norm": 0.3576819598674774,
-      "kl": 0.03891618363559246,
       "learning_rate": 3.991286838919086e-07,
       "loss": 0.0002,
-      "reward": 1.2269162833690643,
-      "reward_std": 0.6302230060100555,
-      "rewards/concensus_correctness_reward_func": 0.14491667225956917,
-      "rewards/consensus_reward_func": 0.1666666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.11966623738408089,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.7748333513736725,
       "step": 48
     },
     {
-      "completion_length": 228.37500381469727,
       "epoch": 8.333333333333334,
-      "grad_norm": 0.28766632080078125,
-      "kl": 0.04578912723809481,
       "learning_rate": 3.902967663405956e-07,
       "loss": 0.0002,
-      "reward": 1.6250251531600952,
-      "reward_std": 1.1099622249603271,
-      "rewards/concensus_correctness_reward_func": 0.1595000047236681,
-      "rewards/consensus_reward_func": 0.416666679084301,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.21798345260322094,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.06250000186264515,
-      "rewards/xmlcount_reward_func": 0.7683750241994858,
       "step": 50
     },
     {
-      "completion_length": 244.87500762939453,
       "epoch": 8.666666666666666,
-      "grad_norm": 0.30231478810310364,
-      "kl": 0.03796851169317961,
       "learning_rate": 3.8120145626980015e-07,
       "loss": 0.0002,
-      "reward": 2.069254457950592,
-      "reward_std": 1.26461523771286,
-      "rewards/concensus_correctness_reward_func": 0.5482500046491623,
-      "rewards/consensus_reward_func": 0.6666666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.21271273493766785,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.6416250169277191,
       "step": 52
     },
     {
-      "completion_length": 229.7916717529297,
       "epoch": 9.0,
-      "grad_norm": 0.36025601625442505,
-      "kl": 0.04843745008111,
       "learning_rate": 3.718598291738298e-07,
       "loss": 0.0002,
-      "reward": 1.5220298618078232,
-      "reward_std": 0.9439242780208588,
-      "rewards/concensus_correctness_reward_func": 0.25691667199134827,
-      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.304363208822906,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.7107500284910202,
       "step": 54
     },
     {
-      "completion_length": 213.37500762939453,
       "epoch": 9.333333333333334,
-      "grad_norm": 0.527549147605896,
-      "kl": 0.05422765016555786,
       "learning_rate": 3.622894229814698e-07,
       "loss": 0.0002,
-      "reward": 1.6290104389190674,
-      "reward_std": 0.6193504929542542,
-      "rewards/concensus_correctness_reward_func": 0.0787500012665987,
       "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.327135369181633,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.9314583837985992,
       "step": 56
     },
     {
-      "completion_length": 236.5833396911621,
       "epoch": 9.666666666666666,
-      "grad_norm": 0.27220746874809265,
-      "kl": 0.047029558569192886,
       "learning_rate": 3.52508205130354e-07,
       "loss": 0.0002,
-      "reward": 1.9271060526371002,
-      "reward_std": 0.6832540258765221,
-      "rewards/concensus_correctness_reward_func": 0.30508334934711456,
-      "rewards/consensus_reward_func": 0.5000000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.34806425496935844,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.7322916686534882,
       "step": 58
     },
     {
-      "completion_length": 224.68750381469727,
       "epoch": 10.0,
-      "grad_norm": 0.35148441791534424,
-      "kl": 0.053522015921771526,
       "learning_rate": 3.4253453883497864e-07,
       "loss": 0.0002,
-      "reward": 1.8315961062908173,
-      "reward_std": 0.643212903290987,
-      "rewards/concensus_correctness_reward_func": 0.17683333903551102,
-      "rewards/consensus_reward_func": 0.416666679084301,
       "rewards/cumulative_reward_2": 0.0,
-      "rewards/final_correctness_reward_func": 0.0833333358168602,
-      "rewards/question_recreation_reward_func": 0.227846035733819,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.06250000186264515,
-      "rewards/xmlcount_reward_func": 0.8644166886806488,
       "step": 60
     },
     {
-      "completion_length": 222.54166793823242,
       "epoch": 10.333333333333334,
-      "grad_norm": 0.2843964397907257,
-      "kl": 0.052015384659171104,
       "learning_rate": 3.323871486116851e-07,
       "loss": 0.0002,
-      "reward": 1.9143906235694885,
-      "reward_std": 1.2949679270386696,
-      "rewards/concensus_correctness_reward_func": 0.35216666758060455,
-      "rewards/consensus_reward_func": 0.5000000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.22976558655500412,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.8116250336170197,
       "step": 62
     },
     {
-      "completion_length": 211.45833587646484,
       "epoch": 10.666666666666666,
-      "grad_norm": 0.4334377944469452,
-      "kl": 0.05805550795048475,
       "learning_rate": 3.220850851253377e-07,
       "loss": 0.0002,
-      "reward": 1.3707057535648346,
-      "reward_std": 0.7664157077670097,
-      "rewards/concensus_correctness_reward_func": 0.0885833352804184,
-      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.14374739676713943,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.784208357334137,
       "step": 64
     },
     {
-      "completion_length": 240.91666793823242,
       "epoch": 11.0,
-      "grad_norm": 0.27897217869758606,
-      "kl": 0.04044825863093138,
       "learning_rate": 3.1164768942369053e-07,
       "loss": 0.0002,
-      "reward": 1.8431612253189087,
-      "reward_std": 1.3769638314843178,
-      "rewards/concensus_correctness_reward_func": 0.2984999977052212,
-      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.38643206655979156,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.720729187130928,
       "step": 66
     },
     {
-      "completion_length": 229.79166793823242,
       "epoch": 11.333333333333334,
-      "grad_norm": 0.3036816120147705,
-      "kl": 0.044112094677984715,
       "learning_rate": 3.010945566265912e-07,
       "loss": 0.0002,
-      "reward": 2.113689601421356,
-      "reward_std": 1.096156120300293,
-      "rewards/concensus_correctness_reward_func": 0.37566667329519987,
-      "rewards/consensus_reward_func": 0.6666666865348816,
       "rewards/cumulative_reward_2": 0.0,
-      "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2770228972658515,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.7526666820049286,
       "step": 68
     },
     {
-      "completion_length": 235.75000762939453,
       "epoch": 11.666666666666666,
-      "grad_norm": 0.29061147570610046,
-      "kl": 0.0439354982227087,
       "learning_rate": 2.9044549913819124e-07,
       "loss": 0.0002,
-      "reward": 2.1150560677051544,
-      "reward_std": 1.0947167873382568,
-      "rewards/concensus_correctness_reward_func": 0.3095000088214874,
-      "rewards/consensus_reward_func": 0.5833333507180214,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.30793093889951706,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.8726250529289246,
       "step": 70
     },
     {
-      "completion_length": 240.08333587646484,
       "epoch": 12.0,
-      "grad_norm": 0.28262025117874146,
-      "kl": 0.04367496073246002,
       "learning_rate": 2.797205094512266e-07,
       "loss": 0.0002,
-      "reward": 1.086464375257492,
-      "reward_std": 0.5841841058572754,
-      "rewards/concensus_correctness_reward_func": 0.07587499916553497,
       "rewards/consensus_reward_func": 0.125,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.22606852278113365,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.6595208421349525,
       "step": 72
     },
     {
-      "completion_length": 224.62500381469727,
       "epoch": 12.333333333333334,
-      "grad_norm": 0.26583132147789,
-      "kl": 0.042624068446457386,
       "learning_rate": 2.6893972261320264e-07,
       "loss": 0.0002,
-      "reward": 1.8349280655384064,
-      "reward_std": 0.8481295928359032,
-      "rewards/concensus_correctness_reward_func": 0.304583340883255,
       "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.3018029108643532,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.7702083438634872,
       "step": 74
     },
     {
-      "completion_length": 252.50000381469727,
       "epoch": 12.666666666666666,
-      "grad_norm": 0.2328660637140274,
-      "kl": 0.03829667530953884,
       "learning_rate": 2.5812337842494516e-07,
       "loss": 0.0002,
-      "reward": 1.3585190325975418,
-      "reward_std": 0.8468546122312546,
-      "rewards/concensus_correctness_reward_func": 0.12975000590085983,
       "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.3044356517493725,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.6743333339691162,
       "step": 76
     },
     {
-      "completion_length": 244.70833587646484,
       "epoch": 13.0,
-      "grad_norm": 0.3013342618942261,
-      "kl": 0.04578291438519955,
       "learning_rate": 2.4729178344249006e-07,
       "loss": 0.0002,
-      "reward": 1.8034937977790833,
-      "reward_std": 0.6810333989560604,
-      "rewards/concensus_correctness_reward_func": 0.34833332896232605,
-      "rewards/consensus_reward_func": 0.5000000149011612,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.271368820220232,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.6837916821241379,
       "step": 78
     },
     {
-      "completion_length": 235.1666717529297,
       "epoch": 13.333333333333334,
-      "grad_norm": 0.2825220227241516,
-      "kl": 0.04979126527905464,
       "learning_rate": 2.3646527285364563e-07,
       "loss": 0.0002,
-      "reward": 1.5074428021907806,
-      "reward_std": 0.8244890198111534,
-      "rewards/concensus_correctness_reward_func": 0.1145833358168602,
-      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
-      "rewards/final_correctness_reward_func": 0.0833333358168602,
-      "rewards/question_recreation_reward_func": 0.23719280119985342,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.8015000224113464,
       "step": 80
     },
     {
-      "completion_length": 242.41666793823242,
       "epoch": 13.666666666666666,
-      "grad_norm": 0.22803151607513428,
-      "kl": 0.03778956923633814,
       "learning_rate": 2.256641723008026e-07,
       "loss": 0.0002,
-      "reward": 2.0478877425193787,
-      "reward_std": 1.4828827381134033,
-      "rewards/concensus_correctness_reward_func": 0.35108333453536034,
-      "rewards/consensus_reward_func": 0.7500000223517418,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2499709390103817,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.6760000213980675,
       "step": 82
     },
     {
-      "completion_length": 218.7083396911621,
       "epoch": 14.0,
-      "grad_norm": 0.288075715303421,
-      "kl": 0.05219701956957579,
       "learning_rate": 2.1490875972166393e-07,
-      "loss": 0.0002,
-      "reward": 2.024667412042618,
-      "reward_std": 0.9902419969439507,
-      "rewards/concensus_correctness_reward_func": 0.42395834997296333,
-      "rewards/consensus_reward_func": 0.4583333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.20850072987377644,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.06250000186264515,
-      "rewards/xmlcount_reward_func": 0.8713749945163727,
       "step": 84
     },
     {
-      "completion_length": 219.0833396911621,
       "epoch": 14.333333333333334,
-      "grad_norm": 0.45791658759117126,
-      "kl": 0.050105306319892406,
       "learning_rate": 2.0421922727953595e-07,
       "loss": 0.0002,
-      "reward": 1.1238564401865005,
-      "reward_std": 0.32448672130703926,
-      "rewards/concensus_correctness_reward_func": 0.0,
       "rewards/consensus_reward_func": 0.0833333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.13618971407413483,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.8626667261123657,
       "step": 86
     },
     {
-      "completion_length": 243.37500381469727,
       "epoch": 14.666666666666666,
-      "grad_norm": 0.29003942012786865,
-      "kl": 0.04252202343195677,
       "learning_rate": 1.9361564345465145e-07,
       "loss": 0.0002,
-      "reward": 1.5835271179676056,
-      "reward_std": 1.4408081620931625,
-      "rewards/concensus_correctness_reward_func": 0.27816667687147856,
-      "rewards/consensus_reward_func": 0.5000000149011612,
       "rewards/cumulative_reward_2": 0.0,
-      "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.24036041554063559,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.5650000125169754,
       "step": 88
     },
     {
-      "completion_length": 242.7916717529297,
       "epoch": 15.0,
-      "grad_norm": 0.25433099269866943,
-      "kl": 0.04068591818213463,
       "learning_rate": 1.8311791536769483e-07,
       "loss": 0.0002,
-      "reward": 1.7171051800251007,
-      "reward_std": 0.5225639697164297,
-      "rewards/concensus_correctness_reward_func": 0.33641667664051056,
-      "rewards/consensus_reward_func": 0.416666679084301,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.20400103740394115,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.7391875237226486,
       "step": 90
     },
     {
-      "completion_length": 223.00000762939453,
       "epoch": 15.333333333333334,
-      "grad_norm": 0.3473324477672577,
-      "kl": 0.05556494276970625,
       "learning_rate": 1.7274575140626315e-07,
       "loss": 0.0002,
-      "reward": 1.4172859191894531,
-      "reward_std": 0.9905060231685638,
-      "rewards/concensus_correctness_reward_func": 0.09850000124424696,
-      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.17020257096737623,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.7735833823680878,
       "step": 92
     },
     {
-      "completion_length": 235.12500381469727,
       "epoch": 15.666666666666666,
-      "grad_norm": 0.215248242020607,
-      "kl": 0.04321257583796978,
       "learning_rate": 1.6251862422442788e-07,
       "loss": 0.0002,
-      "reward": 1.6363305747509003,
-      "reward_std": 0.5130042061209679,
-      "rewards/concensus_correctness_reward_func": 0.1953333355486393,
-      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.299247108399868,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.8084166944026947,
       "step": 94
     },
     {
-      "completion_length": 235.43750381469727,
       "epoch": 16.0,
-      "grad_norm": 0.3912461996078491,
-      "kl": 0.043383605778217316,
       "learning_rate": 1.5245573418486135e-07,
-      "loss": 0.0002,
-      "reward": 1.6504737734794617,
-      "reward_std": 0.5562736243009567,
-      "rewards/concensus_correctness_reward_func": 0.09133333526551723,
-      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.25524457916617393,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.9288958460092545,
       "step": 96
     },
     {
-      "completion_length": 218.3333396911621,
       "epoch": 16.333333333333332,
-      "grad_norm": 0.4353533387184143,
-      "kl": 0.04901806078851223,
       "learning_rate": 1.4257597331216208e-07,
       "loss": 0.0002,
-      "reward": 1.6328352391719818,
-      "reward_std": 0.5481727533042431,
-      "rewards/concensus_correctness_reward_func": 0.08733333554118872,
       "rewards/consensus_reward_func": 0.5000000149011612,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2137102521955967,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.10416666977107525,
-      "rewards/xmlcount_reward_func": 0.7276250272989273,
       "step": 98
     },
     {
-      "completion_length": 220.3333396911621,
       "epoch": 16.666666666666668,
-      "grad_norm": 0.3302115797996521,
-      "kl": 0.0540414247661829,
       "learning_rate": 1.328978898250525e-07,
       "loss": 0.0002,
-      "reward": 1.7450006306171417,
-      "reward_std": 0.7688679657876492,
-      "rewards/concensus_correctness_reward_func": 0.3136666752398014,
-      "rewards/consensus_reward_func": 0.416666679084301,
       "rewards/cumulative_reward_2": 0.0,
-      "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.30995889753103256,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.7047083526849747,
       "step": 100
     },
     {
-      "completion_length": 221.95833587646484,
       "epoch": 17.0,
-      "grad_norm": 0.43570858240127563,
-      "kl": 0.054108550772070885,
       "learning_rate": 1.234396533140365e-07,
       "loss": 0.0002,
-      "reward": 1.433046594262123,
-      "reward_std": 0.501047745347023,
-      "rewards/concensus_correctness_reward_func": 0.22716667503118515,
-      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.17369242385029793,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.7405208498239517,
       "step": 102
     },
     {
-      "completion_length": 235.33333587646484,
       "epoch": 17.333333333333332,
-      "grad_norm": 0.2823254466056824,
-      "kl": 0.04462088830769062,
       "learning_rate": 1.1421902062989178e-07,
       "loss": 0.0002,
-      "reward": 1.6426206827163696,
-      "reward_std": 1.3593513071537018,
-      "rewards/concensus_correctness_reward_func": 0.22100000828504562,
-      "rewards/consensus_reward_func": 0.5000000149011612,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.14599553542211652,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.7339583486318588,
       "step": 104
     },
     {
-      "completion_length": 225.45833587646484,
       "epoch": 17.666666666666668,
-      "grad_norm": 0.423560231924057,
-      "kl": 0.051327490247786045,
       "learning_rate": 1.0525330254703788e-07,
       "loss": 0.0002,
-      "reward": 1.814395785331726,
-      "reward_std": 1.1752215772867203,
-      "rewards/concensus_correctness_reward_func": 0.33800000324845314,
-      "rewards/consensus_reward_func": 0.5833333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.21422904171049595,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.6580000221729279,
       "step": 106
     },
     {
-      "completion_length": 226.62500381469727,
       "epoch": 18.0,
-      "grad_norm": 0.3786259591579437,
-      "kl": 0.04873098246753216,
       "learning_rate": 9.655933126436563e-08,
       "loss": 0.0002,
-      "reward": 1.7698679268360138,
-      "reward_std": 0.9986891597509384,
-      "rewards/concensus_correctness_reward_func": 0.1616250080987811,
-      "rewards/consensus_reward_func": 0.7083333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.15799284353852272,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.741916686296463,
       "step": 108
     },
     {
-      "completion_length": 236.9166717529297,
       "epoch": 18.333333333333332,
-      "grad_norm": 0.2980271577835083,
-      "kl": 0.04274381324648857,
       "learning_rate": 8.81534288045431e-08,
       "loss": 0.0002,
-      "reward": 1.7083100080490112,
-      "reward_std": 1.369063675403595,
-      "rewards/concensus_correctness_reward_func": 0.33241667598485947,
-      "rewards/consensus_reward_func": 0.5000000149011612,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.18101838137954473,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.674041673541069,
       "step": 110
     },
     {
-      "completion_length": 234.33333587646484,
       "epoch": 18.666666666666668,
-      "grad_norm": 0.3028627336025238,
-      "kl": 0.04447379894554615,
       "learning_rate": 8.005137637112302e-08,
       "loss": 0.0002,
-      "reward": 1.6027878820896149,
-      "reward_std": 0.6922035813331604,
-      "rewards/concensus_correctness_reward_func": 0.18299999833106995,
-      "rewards/consensus_reward_func": 0.5,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.22003786265850067,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.6789166927337646,
       "step": 112
     },
     {
-      "completion_length": 214.1458396911621,
       "epoch": 19.0,
-      "grad_norm": 0.4206281900405884,
-      "kl": 0.06770230270922184,
       "learning_rate": 7.226838472098237e-08,
-      "loss": 0.0003,
-      "reward": 2.0530657172203064,
-      "reward_std": 1.1123262345790863,
-      "rewards/concensus_correctness_reward_func": 0.28820834308862686,
-      "rewards/consensus_reward_func": 0.541666679084301,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.27941976487636566,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.1458333358168602,
-      "rewards/xmlcount_reward_func": 0.7979374825954437,
       "step": 114
     },
     {
-      "completion_length": 231.50000381469727,
       "epoch": 19.333333333333332,
-      "grad_norm": 0.413167804479599,
-      "kl": 0.05072194617241621,
       "learning_rate": 6.481906560771524e-08,
       "loss": 0.0002,
-      "reward": 2.3032450079917908,
-      "reward_std": 1.2325157672166824,
-      "rewards/concensus_correctness_reward_func": 0.43541667982935905,
-      "rewards/consensus_reward_func": 0.5833333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.3058698810636997,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0833333358168602,
-      "rewards/xmlcount_reward_func": 0.8952916860580444,
       "step": 116
     },
     {
-      "completion_length": 230.83333587646484,
       "epoch": 19.666666666666668,
-      "grad_norm": 0.3174293041229248,
-      "kl": 0.05088344682008028,
       "learning_rate": 5.771740434959277e-08,
       "loss": 0.0002,
-      "reward": 1.7904408872127533,
-      "reward_std": 0.9557172805070877,
-      "rewards/concensus_correctness_reward_func": 0.2647499991580844,
-      "rewards/consensus_reward_func": 0.5000000149011612,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2976074516773224,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.7280833870172501,
       "step": 118
     },
     {
-      "completion_length": 228.08333587646484,
       "epoch": 20.0,
-      "grad_norm": 0.3763580322265625,
-      "kl": 0.04254571441560984,
       "learning_rate": 5.097673357358906e-08,
-      "loss": 0.0002,
-      "reward": 1.7011156678199768,
-      "reward_std": 0.751253291964531,
-      "rewards/concensus_correctness_reward_func": 0.2535000145435333,
-      "rewards/consensus_reward_func": 0.4583333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.21674058679491282,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.7725417166948318,
       "step": 120
     },
     {
-      "completion_length": 228.75000381469727,
       "epoch": 20.333333333333332,
-      "grad_norm": 0.3015306890010834,
-      "kl": 0.047702303156256676,
       "learning_rate": 4.460970818476717e-08,
       "loss": 0.0002,
-      "reward": 1.9937624335289001,
-      "reward_std": 0.560178704559803,
-      "rewards/concensus_correctness_reward_func": 0.3349999990314245,
-      "rewards/consensus_reward_func": 0.5833333507180214,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.33501239120960236,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.7195833772420883,
       "step": 122
     },
     {
-      "completion_length": 226.0416717529297,
       "epoch": 20.666666666666668,
-      "grad_norm": 0.32871049642562866,
-      "kl": 0.042758834548294544,
       "learning_rate": 3.8628281608017065e-08,
       "loss": 0.0002,
-      "reward": 1.278107762336731,
-      "reward_std": 0.9112474080175161,
-      "rewards/concensus_correctness_reward_func": 0.14350000768899918,
       "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2738994061946869,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.6107083261013031,
       "step": 124
     },
     {
-      "completion_length": 236.9791717529297,
       "epoch": 21.0,
-      "grad_norm": 0.34103289246559143,
-      "kl": 0.048127926886081696,
       "learning_rate": 3.304368334674965e-08,
       "loss": 0.0002,
-      "reward": 1.4082773625850677,
-      "reward_std": 0.620393842458725,
-      "rewards/concensus_correctness_reward_func": 0.13066667318344116,
-      "rewards/consensus_reward_func": 0.1666666716337204,
       "rewards/cumulative_reward_2": 0.0,
-      "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.3291940316557884,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.7817500233650208,
       "step": 126
     },
     {
-      "completion_length": 227.75000381469727,
       "epoch": 21.333333333333332,
-      "grad_norm": 0.29250410199165344,
-      "kl": 0.045962803065776825,
       "learning_rate": 2.7866397900677185e-08,
       "loss": 0.0002,
-      "reward": 1.6840953528881073,
-      "reward_std": 0.7961876690387726,
-      "rewards/concensus_correctness_reward_func": 0.24575000628829002,
-      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2565953563898802,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.7442500442266464,
       "step": 128
     },
     {
-      "completion_length": 240.7083396911621,
       "epoch": 21.666666666666668,
-      "grad_norm": 0.28800123929977417,
-      "kl": 0.04085789620876312,
       "learning_rate": 2.3106145082260774e-08,
       "loss": 0.0002,
-      "reward": 1.466057911515236,
-      "reward_std": 1.0127187222242355,
-      "rewards/concensus_correctness_reward_func": 0.21766667068004608,
-      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.16243284940719604,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.7526250034570694,
       "step": 130
     },
     {
-      "completion_length": 224.31250381469727,
       "epoch": 22.0,
-      "grad_norm": 1.3320063352584839,
-      "kl": 0.06310583744198084,
       "learning_rate": 1.877186176877779e-08,
-      "loss": 0.0003,
-      "reward": 1.6232208609580994,
-      "reward_std": 0.4738241471350193,
-      "rewards/concensus_correctness_reward_func": 0.18699999898672104,
-      "rewards/consensus_reward_func": 0.3333333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.3201374337077141,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.7827500104904175,
       "step": 132
     },
     {
-      "completion_length": 235.00000381469727,
       "epoch": 22.333333333333332,
-      "grad_norm": 0.4193820059299469,
-      "kl": 0.044954728335142136,
       "learning_rate": 1.4871685124269007e-08,
       "loss": 0.0002,
-      "reward": 1.4239540100097656,
-      "reward_std": 0.5730173215270042,
-      "rewards/concensus_correctness_reward_func": 0.12691666930913925,
-      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.24432901293039322,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.636041671037674,
       "step": 134
     },
     {
-      "completion_length": 230.66666793823242,
       "epoch": 22.666666666666668,
-      "grad_norm": 0.332435667514801,
-      "kl": 0.04397624731063843,
       "learning_rate": 1.141293732286297e-08,
-      "loss": 0.0002,
-      "reward": 1.4200390577316284,
-      "reward_std": 0.5456952154636383,
-      "rewards/concensus_correctness_reward_func": 0.1510000079870224,
-      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
-      "rewards/final_correctness_reward_func": 0.0833333358168602,
-      "rewards/question_recreation_reward_func": 0.1249973401427269,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0416666679084301,
-      "rewards/xmlcount_reward_func": 0.7690416872501373,
       "step": 136
     },
     {
-      "completion_length": 249.7916717529297,
       "epoch": 23.0,
-      "grad_norm": 0.2746550142765045,
-      "kl": 0.04453123267740011,
       "learning_rate": 8.402111802159412e-09,
       "loss": 0.0002,
-      "reward": 1.917917937040329,
-      "reward_std": 1.0155989080667496,
-      "rewards/concensus_correctness_reward_func": 0.25612500961869955,
-      "rewards/consensus_reward_func": 0.541666679084301,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.3599179908633232,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.0,
-      "rewards/xmlcount_reward_func": 0.7602083459496498,
       "step": 138
     },
     {
-      "completion_length": 230.2083396911621,
       "epoch": 23.333333333333332,
-      "grad_norm": 0.311666339635849,
-      "kl": 0.04547077417373657,
       "learning_rate": 5.844861072478335e-09,
       "loss": 0.0002,
-      "reward": 1.3695115596055984,
-      "reward_std": 0.7494064196944237,
-      "rewards/concensus_correctness_reward_func": 0.17908333614468575,
-      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.22401154413819313,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.6955833435058594,
       "step": 140
     },
     {
-      "completion_length": 239.25000381469727,
       "epoch": 23.666666666666668,
-      "grad_norm": 0.36921557784080505,
-      "kl": 0.04659950826317072,
       "learning_rate": 3.745986104862903e-09,
       "loss": 0.0002,
-      "reward": 2.4078575372695923,
-      "reward_std": 0.9934381544589996,
-      "rewards/concensus_correctness_reward_func": 0.5574166774749756,
-      "rewards/consensus_reward_func": 0.916666679084301,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2016907762736082,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.7112500220537186,
       "step": 142
     },
     {
-      "completion_length": 246.5208396911621,
       "epoch": 24.0,
-      "grad_norm": 0.30261996388435364,
-      "kl": 0.04283002857118845,
       "learning_rate": 2.1094273177576505e-09,
       "loss": 0.0002,
-      "reward": 1.537266582250595,
-      "reward_std": 0.3170964866876602,
-      "rewards/concensus_correctness_reward_func": 0.10383334010839462,
-      "rewards/consensus_reward_func": 0.1666666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.4318498745560646,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.02083333395421505,
-      "rewards/xmlcount_reward_func": 0.8140833675861359,
       "step": 144
     },
     {
-      "completion_length": 216.2916717529297,
       "epoch": 24.333333333333332,
-      "grad_norm": 0.6070147156715393,
-      "kl": 0.058314370922744274,
       "learning_rate": 9.38257179284696e-10,
       "loss": 0.0002,
-      "reward": 1.786061018705368,
-      "reward_std": 1.4069084376096725,
-      "rewards/concensus_correctness_reward_func": 0.18958334159106016,
-      "rewards/consensus_reward_func": 0.5000000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.2662693187594414,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.06250000186264515,
-      "rewards/xmlcount_reward_func": 0.7677083313465118,
       "step": 146
     },
     {
-      "completion_length": 217.25000762939453,
       "epoch": 24.666666666666668,
-      "grad_norm": 0.28437700867652893,
-      "kl": 0.057665291242301464,
       "learning_rate": 2.3467443900582197e-10,
       "loss": 0.0002,
-      "reward": 1.7139559984207153,
-      "reward_std": 0.6555914841592312,
-      "rewards/concensus_correctness_reward_func": 0.19833334535360336,
-      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.19328934885561466,
       "rewards/soft_format_reward_func": 0.0,
-      "rewards/strict_format_reward_func": 0.10416666977107525,
-      "rewards/xmlcount_reward_func": 0.8848333954811096,
       "step": 148
     },
     {
-      "completion_length": 217.00000381469727,
       "epoch": 25.0,
-      "grad_norm": 0.37572333216667175,
-      "kl": 0.052414169535040855,
       "learning_rate": 0.0,
       "loss": 0.0002,
-      "reward": 1.5510069131851196,
-      "reward_std": 0.8791492730379105,
-      "rewards/concensus_correctness_reward_func": 0.18766666948795319,
-      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
-      "rewards/final_correctness_reward_func": 0.0,
-      "rewards/question_recreation_reward_func": 0.24656930193305016,
-      "rewards/soft_format_reward_func": 0.03125,
-      "rewards/strict_format_reward_func": 0.03125,
-      "rewards/xmlcount_reward_func": 0.6376041695475578,
       "step": 150
     },
     {
       "epoch": 25.0,
       "step": 150,
       "total_flos": 0.0,
-      "train_loss": 0.00018999547231942414,
-      "train_runtime": 2785.5958,
-      "train_samples_per_second": 0.646,
-      "train_steps_per_second": 0.054
     }
   ],
   "logging_steps": 2,

   "is_world_process_zero": true,
   "log_history": [
     {
+      "completion_length": 227.70833587646484,
       "epoch": 0.3333333333333333,
+      "grad_norm": 0.33890047669410706,
+      "kl": 0.05720572918653488,
       "learning_rate": 2e-07,
       "loss": 0.0002,
+      "reward": 1.8000114858150482,
+      "reward_std": 0.9447137117385864,
+      "rewards/concensus_correctness_reward_func": 0.21050000190734863,
+      "rewards/consensus_reward_func": 0.5833333507180214,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3152197636663914,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6701250225305557,
       "step": 2
     },
     {
+      "completion_length": 219.1666717529297,
       "epoch": 0.6666666666666666,
+      "grad_norm": 0.3916985094547272,
+      "kl": 0.07264125626534224,
       "learning_rate": 4e-07,
+      "loss": 0.0003,
+      "reward": 1.8105705082416534,
+      "reward_std": 0.8106473684310913,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.7500000149011612,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.20457049366086721,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.8351666927337646,
       "step": 4
     },
     {
+      "completion_length": 236.37500381469727,
       "epoch": 1.0,
+      "grad_norm": 0.3033287227153778,
+      "kl": 0.04683956317603588,
       "learning_rate": 4.99941324504621e-07,
       "loss": 0.0002,
+      "reward": 1.5050671696662903,
+      "reward_std": 0.9151060022413731,
+      "rewards/concensus_correctness_reward_func": 0.20875000953674316,
+      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.260817164555192,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.7855000346899033,
       "step": 6
     },
     {
+      "completion_length": 236.5833396911621,
       "epoch": 1.3333333333333333,
+      "grad_norm": 0.2626227140426636,
+      "kl": 0.04971806053072214,
       "learning_rate": 4.99472085783721e-07,
       "loss": 0.0002,
+      "reward": 1.492462694644928,
+      "reward_std": 1.0241488218307495,
+      "rewards/concensus_correctness_reward_func": 0.040166668593883514,
+      "rewards/consensus_reward_func": 0.5833333507180214,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2397960852831602,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6083333343267441,
       "step": 8
     },
     {
+      "completion_length": 231.7083396911621,
       "epoch": 1.6666666666666665,
+      "grad_norm": 0.2794240415096283,
+      "kl": 0.04943300876766443,
       "learning_rate": 4.985344892885899e-07,
       "loss": 0.0002,
+      "reward": 1.436886966228485,
+      "reward_std": 0.47921693325042725,
+      "rewards/concensus_correctness_reward_func": 0.06283333897590637,
+      "rewards/consensus_reward_func": 0.3333333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.26051197201013565,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.7385417073965073,
       "step": 10
     },
     {
+      "completion_length": 244.20833587646484,
       "epoch": 2.0,
+      "grad_norm": 0.3874855041503906,
+      "kl": 0.043467676267027855,
       "learning_rate": 4.971302952586796e-07,
       "loss": 0.0002,
+      "reward": 1.175136260688305,
+      "reward_std": 0.6766310762614012,
+      "rewards/concensus_correctness_reward_func": 0.0416666679084301,
+      "rewards/consensus_reward_func": 0.25,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.27315706200897694,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6103125214576721,
       "step": 12
     },
     {
+      "completion_length": 230.58333587646484,
       "epoch": 2.3333333333333335,
+      "grad_norm": 0.30124643445014954,
+      "kl": 0.04529039096087217,
       "learning_rate": 4.952621399215597e-07,
       "loss": 0.0002,
+      "reward": 1.195864737033844,
+      "reward_std": 0.8673334270715714,
+      "rewards/concensus_correctness_reward_func": 0.03824999928474426,
+      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.29573966190218925,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5910416692495346,
       "step": 14
     },
     {
+      "completion_length": 227.50000381469727,
       "epoch": 2.6666666666666665,
+      "grad_norm": 0.30079737305641174,
+      "kl": 0.050349836237728596,
       "learning_rate": 4.929335305436764e-07,
       "loss": 0.0002,
+      "reward": 1.6915239095687866,
+      "reward_std": 0.7203124165534973,
+      "rewards/concensus_correctness_reward_func": 0.05166666954755783,
+      "rewards/consensus_reward_func": 0.5833333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.22785719856619835,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.7870000153779984,
       "step": 16
     },
     {
+      "completion_length": 250.1875,
       "epoch": 3.0,
+      "grad_norm": 0.32253557443618774,
+      "kl": 0.04314618185162544,
       "learning_rate": 4.901488388458247e-07,
       "loss": 0.0002,
+      "reward": 1.4165514409542084,
+      "reward_std": 0.7523854952305555,
+      "rewards/concensus_correctness_reward_func": 0.13858333975076675,
+      "rewards/consensus_reward_func": 0.2916666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.22509302338585258,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.7612083554267883,
       "step": 18
     },
     {
+      "completion_length": 214.00000381469727,
       "epoch": 3.3333333333333335,
+      "grad_norm": 0.7095449566841125,
+      "kl": 0.06387152150273323,
       "learning_rate": 4.869132927957006e-07,
+      "loss": 0.0003,
+      "reward": 1.4457966983318329,
+      "reward_std": 0.9242068529129028,
+      "rewards/concensus_correctness_reward_func": 0.0481666699051857,
+      "rewards/consensus_reward_func": 0.5000000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.15229658968746662,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.7036667019128799,
       "step": 20
     },
     {
+      "completion_length": 239.95833587646484,
       "epoch": 3.6666666666666665,
+      "grad_norm": 0.3170989751815796,
+      "kl": 0.050088039599359035,
       "learning_rate": 4.832329667929376e-07,
+      "loss": 0.0002,
+      "reward": 1.331322818994522,
+      "reward_std": 0.7925830259919167,
+      "rewards/concensus_correctness_reward_func": 0.13858333975076675,
+      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2100728079676628,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.691000010818243,
       "step": 22
     },
     {
+      "completion_length": 234.08333587646484,
       "epoch": 4.0,
+      "grad_norm": 0.336065411567688,
+      "kl": 0.06093376409262419,
       "learning_rate": 4.791147702650565e-07,
       "loss": 0.0002,
+      "reward": 1.5727748572826385,
+      "reward_std": 0.7074941545724869,
+      "rewards/concensus_correctness_reward_func": 0.0416666679084301,
+      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3307124115526676,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.8462291657924652,
       "step": 24
     },
     {
+      "completion_length": 250.91666793823242,
       "epoch": 4.333333333333333,
+      "grad_norm": 0.26310837268829346,
+      "kl": 0.04688491579145193,
       "learning_rate": 4.745664346957361e-07,
       "loss": 0.0002,
+      "reward": 1.4385896027088165,
+      "reward_std": 0.741107240319252,
+      "rewards/concensus_correctness_reward_func": 0.09316666796803474,
+      "rewards/consensus_reward_func": 0.5000000149011612,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1978812012821436,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6267083436250687,
       "step": 26
     },
     {
+      "completion_length": 244.8333396911621,
       "epoch": 4.666666666666667,
+      "grad_norm": 0.3028097450733185,
+      "kl": 0.04158833995461464,
       "learning_rate": 4.695964991097616e-07,
       "loss": 0.0002,
+      "reward": 1.7363463044166565,
+      "reward_std": 0.3860256150364876,
+      "rewards/concensus_correctness_reward_func": 0.057333335280418396,
+      "rewards/consensus_reward_func": 0.6666666865348816,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3005962036550045,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.7117500007152557,
       "step": 28
     },
     {
+      "completion_length": 229.62500381469727,
       "epoch": 5.0,
+      "grad_norm": 0.30192285776138306,
+      "kl": 0.055508922785520554,
       "learning_rate": 4.642142940418973e-07,
       "loss": 0.0002,
+      "reward": 1.4712807536125183,
+      "reward_std": 0.7601758241653442,
+      "rewards/concensus_correctness_reward_func": 0.15825000777840614,
+      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.30686406418681145,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.7145000398159027,
       "step": 30
     },
     {
+      "completion_length": 230.87500381469727,
       "epoch": 5.333333333333333,
+      "grad_norm": 4.099803447723389,
+      "kl": 0.0760243758559227,
       "learning_rate": 4.5842992401978256e-07,
+      "loss": 0.0003,
+      "reward": 1.4511585533618927,
+      "reward_std": 0.6573517695069313,
+      "rewards/concensus_correctness_reward_func": 0.04241666756570339,
+      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.23803354986011982,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.7957083582878113,
       "step": 32
     },
     {
+      "completion_length": 235.5416717529297,
       "epoch": 5.666666666666667,
+      "grad_norm": 0.34264251589775085,
+      "kl": 0.05061831418424845,
       "learning_rate": 4.5225424859373684e-07,
       "loss": 0.0002,
+      "reward": 1.452418863773346,
+      "reward_std": 0.5857485830783844,
+      "rewards/concensus_correctness_reward_func": 0.07324999943375587,
+      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.23775223270058632,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7039166986942291,
       "step": 34
     },
     {
+      "completion_length": 240.66666793823242,
       "epoch": 6.0,
+      "grad_norm": 0.42476335167884827,
+      "kl": 0.04584663826972246,
       "learning_rate": 4.456988619490889e-07,
       "loss": 0.0002,
+      "reward": 0.9470100253820419,
+      "reward_std": 0.5773016959428787,
+      "rewards/concensus_correctness_reward_func": 0.034833334386348724,
+      "rewards/consensus_reward_func": 0.0833333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3073641713708639,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5214791595935822,
       "step": 36
     },
     {
+      "completion_length": 240.62500381469727,
       "epoch": 6.333333333333333,
+      "grad_norm": 0.26901736855506897,
+      "kl": 0.04825030732899904,
       "learning_rate": 4.3877607113930516e-07,
       "loss": 0.0002,
+      "reward": 1.6666896641254425,
+      "reward_std": 0.9019213914871216,
+      "rewards/concensus_correctness_reward_func": 0.1275000087916851,
+      "rewards/consensus_reward_func": 0.6666666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.22656460665166378,
+      "rewards/soft_format_reward_func": 0.02083333395421505,
       "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6251250058412552,
       "step": 38
     },
     {
+      "completion_length": 241.62500762939453,
       "epoch": 6.666666666666667,
+      "grad_norm": 0.30646660923957825,
+      "kl": 0.049314847216010094,
       "learning_rate": 4.314988729807827e-07,
       "loss": 0.0002,
+      "reward": 1.865004450082779,
+      "reward_std": 0.5881587564945221,
+      "rewards/concensus_correctness_reward_func": 0.10433333367109299,
+      "rewards/consensus_reward_func": 0.7500000298023224,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1724627036601305,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.7965416759252548,
       "step": 40
     },
     {
+      "completion_length": 224.95833587646484,
       "epoch": 7.0,
+      "grad_norm": 0.38283172249794006,
+      "kl": 0.06825974117964506,
       "learning_rate": 4.238809296526846e-07,
+      "loss": 0.0003,
+      "reward": 2.3348300457000732,
+      "reward_std": 1.0311907231807709,
+      "rewards/concensus_correctness_reward_func": 0.32250000536441803,
+      "rewards/consensus_reward_func": 0.833333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.19339236430823803,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0833333358168602,
+      "rewards/xmlcount_reward_func": 0.9022708535194397,
       "step": 42
     },
     {
+      "completion_length": 229.12500762939453,
       "epoch": 7.333333333333333,
+      "grad_norm": 0.3858100473880768,
+      "kl": 0.05412571597844362,
       "learning_rate": 4.159365430476261e-07,
       "loss": 0.0002,
+      "reward": 1.8162244409322739,
+      "reward_std": 0.8844669237732887,
+      "rewards/concensus_correctness_reward_func": 0.06599999964237213,
+      "rewards/consensus_reward_func": 0.8333333656191826,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1488076914101839,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7472500205039978,
       "step": 44
     },
     {
+      "completion_length": 242.1666717529297,
       "epoch": 7.666666666666667,
+      "grad_norm": 0.32038140296936035,
+      "kl": 0.04718218743801117,
       "learning_rate": 4.076806279213655e-07,
       "loss": 0.0002,
+      "reward": 1.4529284834861755,
+      "reward_std": 0.9867856577038765,
+      "rewards/concensus_correctness_reward_func": 0.09458333440124989,
       "rewards/consensus_reward_func": 0.416666679084301,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.13534517213702202,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.7646666765213013,
       "step": 46
     },
     {
+      "completion_length": 233.85416793823242,
       "epoch": 8.0,
+      "grad_norm": 0.5537049770355225,
+      "kl": 0.057070521637797356,
       "learning_rate": 3.991286838919086e-07,
       "loss": 0.0002,
+      "reward": 1.5067541599273682,
+      "reward_std": 0.7801018357276917,
+      "rewards/concensus_correctness_reward_func": 0.07991666719317436,
+      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2330041080713272,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.03125,
+      "rewards/xmlcount_reward_func": 0.8292500227689743,
       "step": 48
     },
     {
+      "completion_length": 231.41667556762695,
       "epoch": 8.333333333333334,
+      "grad_norm": 0.2945857346057892,
+      "kl": 0.058525013737380505,
       "learning_rate": 3.902967663405956e-07,
       "loss": 0.0002,
+      "reward": 1.1730442941188812,
+      "reward_std": 0.4971357434988022,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.0833333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.24308593943715096,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.8049583435058594,
       "step": 50
     },
     {
+      "completion_length": 218.08333587646484,
       "epoch": 8.666666666666666,
+      "grad_norm": 0.3458411693572998,
+      "kl": 0.053861903958022594,
       "learning_rate": 3.8120145626980015e-07,
       "loss": 0.0002,
+      "reward": 1.4622103124856949,
+      "reward_std": 0.5742723122239113,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.24300195463001728,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.1041666679084301,
+      "rewards/xmlcount_reward_func": 0.7817083597183228,
       "step": 52
     },
     {
+      "completion_length": 235.68750381469727,
       "epoch": 9.0,
+      "grad_norm": 0.2695240080356598,
+      "kl": 0.054968020878732204,
       "learning_rate": 3.718598291738298e-07,
       "loss": 0.0002,
+      "reward": 1.494756355881691,
+      "reward_std": 0.6502897441387177,
+      "rewards/concensus_correctness_reward_func": 0.1977500170469284,
+      "rewards/consensus_reward_func": 0.2083333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.28436052799224854,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.8043125122785568,
       "step": 54
     },
     {
+      "completion_length": 241.2916717529297,
       "epoch": 9.333333333333334,
+      "grad_norm": 0.2965337336063385,
+      "kl": 0.04629590827971697,
       "learning_rate": 3.622894229814698e-07,
       "loss": 0.0002,
+      "reward": 1.3549380600452423,
+      "reward_std": 0.35284742526710033,
+      "rewards/concensus_correctness_reward_func": 0.13183334469795227,
       "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3231046348810196,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6500000208616257,
       "step": 56
     },
     {
+      "completion_length": 237.12500381469727,
       "epoch": 9.666666666666666,
+      "grad_norm": 0.39377561211586,
+      "kl": 0.049407465383410454,
       "learning_rate": 3.52508205130354e-07,
       "loss": 0.0002,
+      "reward": 1.58523827791214,
+      "reward_std": 0.4869128167629242,
+      "rewards/concensus_correctness_reward_func": 0.0625,
+      "rewards/consensus_reward_func": 0.3333333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.33502987399697304,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.812708392739296,
       "step": 58
     },
     {
+      "completion_length": 230.9166717529297,
       "epoch": 10.0,
+      "grad_norm": 0.33048373460769653,
+      "kl": 0.057509299367666245,
       "learning_rate": 3.4253453883497864e-07,
       "loss": 0.0002,
+      "reward": 1.3047290593385696,
+      "reward_std": 0.6186777204275131,
+      "rewards/concensus_correctness_reward_func": 0.0988750010728836,
+      "rewards/consensus_reward_func": 0.1666666716337204,
       "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2581872884184122,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.7393333464860916,
       "step": 60
     },
     {
+      "completion_length": 243.87500381469727,
       "epoch": 10.333333333333334,
+      "grad_norm": 0.27913355827331543,
+      "kl": 0.04607606399804354,
       "learning_rate": 3.323871486116851e-07,
       "loss": 0.0002,
+      "reward": 1.61500483751297,
+      "reward_std": 0.8177201747894287,
+      "rewards/concensus_correctness_reward_func": 0.1132500022649765,
+      "rewards/consensus_reward_func": 0.5833333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1745881037786603,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.7021666914224625,
       "step": 62
     },
     {
+      "completion_length": 242.70833587646484,
       "epoch": 10.666666666666666,
+      "grad_norm": 0.30817922949790955,
+      "kl": 0.04235330317169428,
       "learning_rate": 3.220850851253377e-07,
       "loss": 0.0002,
+      "reward": 1.2049356400966644,
+      "reward_std": 1.0750536024570465,
+      "rewards/concensus_correctness_reward_func": 0.19475000351667404,
+      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.16143561899662018,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.5987500250339508,
       "step": 64
     },
     {
+      "completion_length": 231.10416793823242,
       "epoch": 11.0,
+      "grad_norm": 0.32778000831604004,
+      "kl": 0.056444195099174976,
       "learning_rate": 3.1164768942369053e-07,
       "loss": 0.0002,
+      "reward": 1.6779266893863678,
+      "reward_std": 0.9572022035717964,
+      "rewards/concensus_correctness_reward_func": 0.176916666328907,
+      "rewards/consensus_reward_func": 0.2916666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.3696349188685417,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0729166679084301,
+      "rewards/xmlcount_reward_func": 0.7667916864156723,
       "step": 66
     },
     {
+      "completion_length": 227.2916717529297,
       "epoch": 11.333333333333334,
+      "grad_norm": 0.35803529620170593,
+      "kl": 0.057758173905313015,
       "learning_rate": 3.010945566265912e-07,
       "loss": 0.0002,
+      "reward": 1.7297503054141998,
+      "reward_std": 0.8079974018037319,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.6666666716337204,
       "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0833333358168602,
+      "rewards/question_recreation_reward_func": 0.2813752740621567,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6775416731834412,
       "step": 68
     },
     {
+      "completion_length": 234.50000381469727,
       "epoch": 11.666666666666666,
+      "grad_norm": 0.361043781042099,
+      "kl": 0.05352189112454653,
       "learning_rate": 2.9044549913819124e-07,
       "loss": 0.0002,
+      "reward": 1.3361421823501587,
+      "reward_std": 0.7610064297914505,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.5000000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.31085047870874405,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.5044583640992641,
       "step": 70
     },
     {
+      "completion_length": 239.18750762939453,
       "epoch": 12.0,
+      "grad_norm": 0.4037138521671295,
+      "kl": 0.04751924052834511,
       "learning_rate": 2.797205094512266e-07,
       "loss": 0.0002,
+      "reward": 1.4057151973247528,
+      "reward_std": 0.7661054208874702,
+      "rewards/concensus_correctness_reward_func": 0.2408333420753479,
       "rewards/consensus_reward_func": 0.125,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.23336100485175848,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7856874912977219,
       "step": 72
     },
     {
+      "completion_length": 238.87500762939453,
       "epoch": 12.333333333333334,
+      "grad_norm": 0.3457435071468353,
+      "kl": 0.042915891855955124,
       "learning_rate": 2.6893972261320264e-07,
       "loss": 0.0002,
+      "reward": 1.3980638980865479,
+      "reward_std": 0.8063503727316856,
+      "rewards/concensus_correctness_reward_func": 0.0,
       "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.30481382086873055,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6765833348035812,
       "step": 74
     },
     {
+      "completion_length": 219.5416717529297,
       "epoch": 12.666666666666666,
+      "grad_norm": 0.40163183212280273,
+      "kl": 0.05886824615299702,
       "learning_rate": 2.5812337842494516e-07,
       "loss": 0.0002,
+      "reward": 1.3007263541221619,
+      "reward_std": 0.5792420618236065,
+      "rewards/concensus_correctness_reward_func": 0.06599999964237213,
       "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2327263280749321,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7311666905879974,
       "step": 76
     },
     {
+      "completion_length": 249.20833587646484,
       "epoch": 13.0,
+      "grad_norm": 0.27837109565734863,
+      "kl": 0.047816917300224304,
       "learning_rate": 2.4729178344249006e-07,
       "loss": 0.0002,
+      "reward": 1.5985942333936691,
+      "reward_std": 0.5690072141587734,
+      "rewards/concensus_correctness_reward_func": 0.07266666740179062,
+      "rewards/consensus_reward_func": 0.4583333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2542608454823494,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.8133333474397659,
       "step": 78
     },
     {
+      "completion_length": 241.62500762939453,
       "epoch": 13.333333333333334,
+      "grad_norm": 0.3208363950252533,
+      "kl": 0.04787137173116207,
       "learning_rate": 2.3646527285364563e-07,
       "loss": 0.0002,
+      "reward": 1.2686220705509186,
+      "reward_std": 0.5177956447005272,
+      "rewards/concensus_correctness_reward_func": 0.09983333572745323,
+      "rewards/consensus_reward_func": 0.1666666716337204,
       "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2596219964325428,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.7424999922513962,
       "step": 80
     },
     {
+      "completion_length": 225.62500381469727,
       "epoch": 13.666666666666666,
+      "grad_norm": 0.2911342680454254,
+      "kl": 0.053801815025508404,
       "learning_rate": 2.256641723008026e-07,
       "loss": 0.0002,
+      "reward": 1.550098717212677,
+      "reward_std": 0.5509862564504147,
+      "rewards/concensus_correctness_reward_func": 0.17783334478735924,
+      "rewards/consensus_reward_func": 0.1666666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2142653577029705,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0833333358168602,
+      "rewards/xmlcount_reward_func": 0.908000037074089,
       "step": 82
     },
     {
+      "completion_length": 229.06250381469727,
       "epoch": 14.0,
+      "grad_norm": 0.46740520000457764,
+      "kl": 0.06748753227293491,
       "learning_rate": 2.1490875972166393e-07,
+      "loss": 0.0003,
+      "reward": 1.1959194540977478,
+      "reward_std": 0.5019870288670063,
+      "rewards/concensus_correctness_reward_func": 0.19500000402331352,
+      "rewards/consensus_reward_func": 0.0833333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.17558605503290892,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7211667150259018,
       "step": 84
     },
     {
+      "completion_length": 229.2916717529297,
       "epoch": 14.333333333333334,
+      "grad_norm": 0.312028706073761,
+      "kl": 0.052867443300783634,
       "learning_rate": 2.0421922727953595e-07,
       "loss": 0.0002,
+      "reward": 1.0239089727401733,
+      "reward_std": 0.4360574968159199,
+      "rewards/concensus_correctness_reward_func": 0.03500000014901161,
       "rewards/consensus_reward_func": 0.0833333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.19924231618642807,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6855000406503677,
       "step": 86
     },
     {
+      "completion_length": 225.50000381469727,
       "epoch": 14.666666666666666,
+      "grad_norm": 0.34441980719566345,
+      "kl": 0.06209190376102924,
       "learning_rate": 1.9361564345465145e-07,
       "loss": 0.0002,
+      "reward": 2.05111762881279,
+      "reward_std": 0.8670734912157059,
+      "rewards/concensus_correctness_reward_func": 0.0898333378136158,
+      "rewards/consensus_reward_func": 0.5833333358168602,
       "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.1666666716337204,
+      "rewards/question_recreation_reward_func": 0.28711760602891445,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.06250000186264515,
+      "rewards/xmlcount_reward_func": 0.861666664481163,
       "step": 88
     },
     {
+      "completion_length": 239.54166793823242,
       "epoch": 15.0,
+      "grad_norm": 0.2994779944419861,
+      "kl": 0.04788489267230034,
       "learning_rate": 1.8311791536769483e-07,
       "loss": 0.0002,
+      "reward": 1.3930490911006927,
+      "reward_std": 0.571447841823101,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.4166666865348816,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.27179905865341425,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.6629166603088379,
       "step": 90
     },
     {
+      "completion_length": 246.25000381469727,
       "epoch": 15.333333333333334,
+      "grad_norm": 0.26075470447540283,
+      "kl": 0.043299716897308826,
       "learning_rate": 1.7274575140626315e-07,
       "loss": 0.0002,
+      "reward": 1.2123132348060608,
+      "reward_std": 0.5058312565088272,
+      "rewards/concensus_correctness_reward_func": 0.1158333346247673,
+      "rewards/consensus_reward_func": 0.0833333358168602,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.17077156156301498,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.8423750400543213,
       "step": 92
     },
     {
+      "completion_length": 242.83333587646484,
       "epoch": 15.666666666666666,
+      "grad_norm": 0.257622092962265,
+      "kl": 0.04368113446980715,
       "learning_rate": 1.6251862422442788e-07,
       "loss": 0.0002,
+      "reward": 1.387978509068489,
+      "reward_std": 0.8142535351216793,
+      "rewards/concensus_correctness_reward_func": 0.03824999928474426,
+      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.18227007426321507,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7299583554267883,
       "step": 94
     },
     {
+      "completion_length": 236.7916717529297,
       "epoch": 16.0,
+      "grad_norm": 0.3843899965286255,
+      "kl": 0.06746228598058224,
       "learning_rate": 1.5245573418486135e-07,
+      "loss": 0.0003,
+      "reward": 1.4802160412073135,
+      "reward_std": 0.6605739668011665,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.4583333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.15907016210258007,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.06250000186264515,
+      "rewards/xmlcount_reward_func": 0.8003125041723251,
       "step": 96
     },
     {
+      "completion_length": 227.0416717529297,
       "epoch": 16.333333333333332,
+      "grad_norm": 0.313445121049881,
+      "kl": 0.05225266329944134,
       "learning_rate": 1.4257597331216208e-07,
       "loss": 0.0002,
+      "reward": 1.6126384288072586,
+      "reward_std": 0.5593572035431862,
+      "rewards/concensus_correctness_reward_func": 0.08424999937415123,
       "rewards/consensus_reward_func": 0.5000000149011612,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.27855511754751205,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.7498333603143692,
       "step": 98
     },
     {
+      "completion_length": 243.45833587646484,
       "epoch": 16.666666666666668,
+      "grad_norm": 0.286333829164505,
+      "kl": 0.04129187762737274,
       "learning_rate": 1.328978898250525e-07,
       "loss": 0.0002,
+      "reward": 1.0129260122776031,
+      "reward_std": 0.799390435218811,
+      "rewards/concensus_correctness_reward_func": 0.040166668593883514,
+      "rewards/consensus_reward_func": 0.0833333358168602,
       "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0833333358168602,
+      "rewards/question_recreation_reward_func": 0.3159259632229805,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.49016668647527695,
       "step": 100
     },
     {
+      "completion_length": 231.5416717529297,
       "epoch": 17.0,
+      "grad_norm": 0.35134121775627136,
+      "kl": 0.05213775299489498,
       "learning_rate": 1.234396533140365e-07,
       "loss": 0.0002,
+      "reward": 1.8263617157936096,
+      "reward_std": 0.7876124978065491,
+      "rewards/concensus_correctness_reward_func": 0.15541666746139526,
+      "rewards/consensus_reward_func": 0.8333333507180214,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1288825012743473,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.7087291777133942,
       "step": 102
     },
     {
+      "completion_length": 233.4583396911621,
       "epoch": 17.333333333333332,
+      "grad_norm": 0.3022390902042389,
+      "kl": 0.048062763176858425,
       "learning_rate": 1.1421902062989178e-07,
       "loss": 0.0002,
+      "reward": 1.0451362580060959,
+      "reward_std": 0.6692814938724041,
+      "rewards/concensus_correctness_reward_func": 0.06591667234897614,
+      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.12380289193242788,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6054166853427887,
       "step": 104
     },
     {
+      "completion_length": 240.75,
       "epoch": 17.666666666666668,
+      "grad_norm": 0.3042658865451813,
+      "kl": 0.05203111656010151,
       "learning_rate": 1.0525330254703788e-07,
       "loss": 0.0002,
+      "reward": 1.6611795723438263,
+      "reward_std": 0.590112742036581,
+      "rewards/concensus_correctness_reward_func": 0.13641666620969772,
+      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2227628417313099,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.8645000010728836,
       "step": 106
     },
     {
+      "completion_length": 238.16666793823242,
       "epoch": 18.0,
+      "grad_norm": 0.3192152678966522,
+      "kl": 0.051314856857061386,
       "learning_rate": 9.655933126436563e-08,
       "loss": 0.0002,
+      "reward": 1.5194370597600937,
+      "reward_std": 0.7573564946651459,
+      "rewards/concensus_correctness_reward_func": 0.13850000500679016,
+      "rewards/consensus_reward_func": 0.3333333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2693120799958706,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7574583441019058,
       "step": 108
     },
     {
+      "completion_length": 240.70833587646484,
       "epoch": 18.333333333333332,
+      "grad_norm": 0.3217252790927887,
+      "kl": 0.04907871410250664,
       "learning_rate": 8.81534288045431e-08,
       "loss": 0.0002,
+      "reward": 1.3132466971874237,
+      "reward_std": 0.7200035415589809,
+      "rewards/concensus_correctness_reward_func": 0.079500000923872,
+      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.19924663566052914,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.06250000186264515,
+      "rewards/xmlcount_reward_func": 0.7220000326633453,
       "step": 110
     },
     {
+      "completion_length": 232.1666717529297,
       "epoch": 18.666666666666668,
+      "grad_norm": 0.3155098855495453,
+      "kl": 0.05641421768814325,
       "learning_rate": 8.005137637112302e-08,
       "loss": 0.0002,
+      "reward": 1.509556770324707,
+      "reward_std": 1.2360821068286896,
+      "rewards/concensus_correctness_reward_func": 0.16583333536982536,
+      "rewards/consensus_reward_func": 0.416666679084301,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.13014000188559294,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.06250000186264515,
+      "rewards/xmlcount_reward_func": 0.7344166934490204,
       "step": 112
     },
     {
+      "completion_length": 232.9166717529297,
       "epoch": 19.0,
+      "grad_norm": 0.33141234517097473,
+      "kl": 0.05206760857254267,
       "learning_rate": 7.226838472098237e-08,
+      "loss": 0.0002,
+      "reward": 1.9219300746917725,
+      "reward_std": 1.1108430549502373,
+      "rewards/concensus_correctness_reward_func": 0.15133334323763847,
+      "rewards/consensus_reward_func": 0.6250000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2631174586713314,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.06250000186264515,
+      "rewards/xmlcount_reward_func": 0.8199791759252548,
       "step": 114
     },
     {
+      "completion_length": 238.91666793823242,
       "epoch": 19.333333333333332,
+      "grad_norm": 0.2539539039134979,
+      "kl": 0.051578775979578495,
       "learning_rate": 6.481906560771524e-08,
       "loss": 0.0002,
+      "reward": 1.655208259820938,
+      "reward_std": 1.009856939315796,
+      "rewards/concensus_correctness_reward_func": 0.09341667126864195,
+      "rewards/consensus_reward_func": 0.5000000149011612,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2639997862279415,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7769583761692047,
       "step": 116
     },
     {
+      "completion_length": 250.62500381469727,
       "epoch": 19.666666666666668,
+      "grad_norm": 0.30547869205474854,
+      "kl": 0.04439305979758501,
       "learning_rate": 5.771740434959277e-08,
       "loss": 0.0002,
+      "reward": 1.5269443988800049,
+      "reward_std": 0.9310744404792786,
+      "rewards/concensus_correctness_reward_func": 0.12908333912491798,
+      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.31090278178453445,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.7952917069196701,
       "step": 118
     },
     {
+      "completion_length": 214.95833587646484,
       "epoch": 20.0,
+      "grad_norm": 0.4013596475124359,
+      "kl": 0.06496140733361244,
       "learning_rate": 5.097673357358906e-08,
+      "loss": 0.0003,
+      "reward": 1.5405447483062744,
+      "reward_std": 0.7922499552369118,
+      "rewards/concensus_correctness_reward_func": 0.11241666972637177,
+      "rewards/consensus_reward_func": 0.3750000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.20348212867975235,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.09375000186264515,
+      "rewards/xmlcount_reward_func": 0.7558958530426025,
       "step": 120
     },
     {
+      "completion_length": 241.3333396911621,
       "epoch": 20.333333333333332,
+      "grad_norm": 0.29899001121520996,
+      "kl": 0.04371773358434439,
       "learning_rate": 4.460970818476717e-08,
       "loss": 0.0002,
+      "reward": 1.4422058463096619,
+      "reward_std": 0.6626204028725624,
+      "rewards/concensus_correctness_reward_func": 0.24758334457874298,
+      "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2500807549804449,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.6737083569169044,
       "step": 122
     },
     {
+      "completion_length": 233.4166717529297,
       "epoch": 20.666666666666668,
+      "grad_norm": 0.30645209550857544,
+      "kl": 0.05884226877242327,
       "learning_rate": 3.8628281608017065e-08,
       "loss": 0.0002,
+      "reward": 1.3813151270151138,
+      "reward_std": 0.8012359291315079,
+      "rewards/concensus_correctness_reward_func": 0.062166668474674225,
       "rewards/consensus_reward_func": 0.2500000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2932317517697811,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7550833523273468,
       "step": 124
     },
     {
+      "completion_length": 239.47916793823242,
       "epoch": 21.0,
+      "grad_norm": 0.2772859036922455,
+      "kl": 0.05142911709845066,
       "learning_rate": 3.304368334674965e-08,
       "loss": 0.0002,
+      "reward": 2.326990634202957,
+      "reward_std": 0.6766933500766754,
+      "rewards/concensus_correctness_reward_func": 0.16600000858306885,
+      "rewards/consensus_reward_func": 0.9166666716337204,
       "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0833333358168602,
+      "rewards/question_recreation_reward_func": 0.3391989953815937,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.03125,
+      "rewards/xmlcount_reward_func": 0.7905416935682297,
       "step": 126
     },
     {
+      "completion_length": 237.5416717529297,
       "epoch": 21.333333333333332,
+      "grad_norm": 0.2806893289089203,
+      "kl": 0.05129498243331909,
       "learning_rate": 2.7866397900677185e-08,
       "loss": 0.0002,
+      "reward": 1.6936083137989044,
+      "reward_std": 0.7103723436594009,
+      "rewards/concensus_correctness_reward_func": 0.0273333340883255,
+      "rewards/consensus_reward_func": 0.5000000074505806,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.29106658324599266,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.833541676402092,
       "step": 128
     },
     {
+      "completion_length": 230.4166717529297,
       "epoch": 21.666666666666668,
+      "grad_norm": 0.3453538715839386,
+      "kl": 0.05201288964599371,
       "learning_rate": 2.3106145082260774e-08,
       "loss": 0.0002,
+      "reward": 1.6965271532535553,
+      "reward_std": 0.8494137600064278,
+      "rewards/concensus_correctness_reward_func": 0.19891666620969772,
+      "rewards/consensus_reward_func": 0.5833333432674408,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.15011047292500734,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.7225000187754631,
       "step": 130
     },
     {
+      "completion_length": 242.35416793823242,
       "epoch": 22.0,
+      "grad_norm": 0.3385789692401886,
+      "kl": 0.05280694644898176,
       "learning_rate": 1.877186176877779e-08,
+      "loss": 0.0002,
+      "reward": 1.931239366531372,
+      "reward_std": 0.8559084534645081,
+      "rewards/concensus_correctness_reward_func": 0.17491666972637177,
+      "rewards/consensus_reward_func": 0.541666679084301,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.37569759786129,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.05208333395421505,
+      "rewards/xmlcount_reward_func": 0.7868750244379044,
       "step": 132
     },
     {
+      "completion_length": 228.8333396911621,
       "epoch": 22.333333333333332,
+      "grad_norm": 0.3556283116340637,
+      "kl": 0.05452739167958498,
       "learning_rate": 1.4871685124269007e-08,
       "loss": 0.0002,
+      "reward": 1.1983706653118134,
+      "reward_std": 0.42030148580670357,
+      "rewards/concensus_correctness_reward_func": 0.057333335280418396,
+      "rewards/consensus_reward_func": 0.1666666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.2380373403429985,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7155000269412994,
       "step": 134
     },
     {
+      "completion_length": 226.16666793823242,
       "epoch": 22.666666666666668,
+      "grad_norm": 0.38454777002334595,
+      "kl": 0.0568731427192688,
       "learning_rate": 1.141293732286297e-08,
+      "loss": 0.0001,
+      "reward": 1.9853862524032593,
+      "reward_std": 1.331713080406189,
+      "rewards/concensus_correctness_reward_func": 0.15325000416487455,
+      "rewards/consensus_reward_func": 0.8333333432674408,
       "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.14426118321716785,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.08333333395421505,
+      "rewards/xmlcount_reward_func": 0.7712083458900452,
       "step": 136
     },
     {
+      "completion_length": 240.00000381469727,
       "epoch": 23.0,
+      "grad_norm": 0.36095941066741943,
+      "kl": 0.04767625965178013,
       "learning_rate": 8.402111802159412e-09,
       "loss": 0.0002,
+      "reward": 1.6724743098020554,
+      "reward_std": 1.116004541516304,
+      "rewards/concensus_correctness_reward_func": 0.2174583412706852,
+      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.30618254095315933,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7113333642482758,
       "step": 138
     },
     {
+      "completion_length": 224.2916717529297,
       "epoch": 23.333333333333332,
+      "grad_norm": 0.49767544865608215,
+      "kl": 0.060226147063076496,
       "learning_rate": 5.844861072478335e-09,
       "loss": 0.0002,
+      "reward": 1.4785583317279816,
+      "reward_std": 0.9657530337572098,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1823915969580412,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0833333358168602,
+      "rewards/xmlcount_reward_func": 0.7961666882038116,
       "step": 140
     },
     {
+      "completion_length": 230.9583396911621,
       "epoch": 23.666666666666668,
+      "grad_norm": 0.31328701972961426,
+      "kl": 0.05263958778232336,
       "learning_rate": 3.745986104862903e-09,
       "loss": 0.0002,
+      "reward": 1.2850330173969269,
+      "reward_std": 0.9222555011510849,
+      "rewards/concensus_correctness_reward_func": 0.05791666731238365,
+      "rewards/consensus_reward_func": 0.4166666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1542829778045416,
       "rewards/soft_format_reward_func": 0.0,
       "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.635333351790905,
       "step": 142
     },
     {
+      "completion_length": 246.16666793823242,
       "epoch": 24.0,
+      "grad_norm": 0.2712779939174652,
+      "kl": 0.04399109352380037,
       "learning_rate": 2.1094273177576505e-09,
       "loss": 0.0002,
+      "reward": 1.8127109110355377,
+      "reward_std": 0.556302860379219,
+      "rewards/concensus_correctness_reward_func": 0.034833334386348724,
+      "rewards/consensus_reward_func": 0.6666666716337204,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.4046483635902405,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.7065625041723251,
       "step": 144
     },
     {
+      "completion_length": 236.2083396911621,
       "epoch": 24.333333333333332,
+      "grad_norm": 0.3137350380420685,
+      "kl": 0.0501671489328146,
       "learning_rate": 9.38257179284696e-10,
       "loss": 0.0002,
+      "reward": 1.3607908338308334,
+      "reward_std": 0.4233456924557686,
+      "rewards/concensus_correctness_reward_func": 0.0,
+      "rewards/consensus_reward_func": 0.5000000149011612,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.1967908414080739,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0416666679084301,
+      "rewards/xmlcount_reward_func": 0.6223333328962326,
       "step": 146
     },
     {
+      "completion_length": 223.58333587646484,
       "epoch": 24.666666666666668,
+      "grad_norm": 0.29890674352645874,
+      "kl": 0.060095793567597866,
       "learning_rate": 2.3467443900582197e-10,
       "loss": 0.0002,
+      "reward": 1.317613661289215,
+      "reward_std": 0.6358807981014252,
+      "rewards/concensus_correctness_reward_func": 0.04325000196695328,
+      "rewards/consensus_reward_func": 0.25,
       "rewards/cumulative_reward_2": 0.0,
       "rewards/final_correctness_reward_func": 0.0,
+      "rewards/question_recreation_reward_func": 0.27653032541275024,
       "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.02083333395421505,
+      "rewards/xmlcount_reward_func": 0.7270000278949738,
       "step": 148
     },
     {
+      "completion_length": 233.62500381469727,
       "epoch": 25.0,
+      "grad_norm": 0.3637758195400238,
+      "kl": 0.042237890884280205,
       "learning_rate": 0.0,
       "loss": 0.0002,
+      "reward": 1.663491129875183,
+      "reward_std": 0.8305407166481018,
+      "rewards/concensus_correctness_reward_func": 0.06158333644270897,
+      "rewards/consensus_reward_func": 0.416666679084301,
       "rewards/cumulative_reward_2": 0.0,
+      "rewards/final_correctness_reward_func": 0.25,
+      "rewards/question_recreation_reward_func": 0.29997023195028305,
+      "rewards/soft_format_reward_func": 0.0,
+      "rewards/strict_format_reward_func": 0.0,
+      "rewards/xmlcount_reward_func": 0.6352708414196968,
       "step": 150
     },
     {
       "epoch": 25.0,
       "step": 150,
       "total_flos": 0.0,
+      "train_loss": 0.00020825650698194902,
+      "train_runtime": 2537.33,
+      "train_samples_per_second": 0.709,
+      "train_steps_per_second": 0.059
     }
   ],
   "logging_steps": 2,