Instructions to use jz666/gemma-2-9b-it-dpo-train_filtered_full with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jz666/gemma-2-9b-it-dpo-train_filtered_full with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jz666/gemma-2-9b-it-dpo-train_filtered_full")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jz666/gemma-2-9b-it-dpo-train_filtered_full")
model = AutoModelForCausalLM.from_pretrained("jz666/gemma-2-9b-it-dpo-train_filtered_full", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jz666/gemma-2-9b-it-dpo-train_filtered_full with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jz666/gemma-2-9b-it-dpo-train_filtered_full"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jz666/gemma-2-9b-it-dpo-train_filtered_full",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jz666/gemma-2-9b-it-dpo-train_filtered_full

SGLang

How to use jz666/gemma-2-9b-it-dpo-train_filtered_full with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jz666/gemma-2-9b-it-dpo-train_filtered_full" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jz666/gemma-2-9b-it-dpo-train_filtered_full",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jz666/gemma-2-9b-it-dpo-train_filtered_full" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jz666/gemma-2-9b-it-dpo-train_filtered_full",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use jz666/gemma-2-9b-it-dpo-train_filtered_full with Docker Model Runner:
```
docker model run hf.co/jz666/gemma-2-9b-it-dpo-train_filtered_full
```

jz666 commited on Oct 20, 2025

Commit

ef15816

verified ·

1 Parent(s): 712466e

Model save

Browse files

Files changed (5) hide show

README.md +74 -0
all_results.json +9 -0
generation_config.json +8 -0
train_results.json +9 -0
trainer_state.json +1423 -0

README.md ADDED Viewed

	@@ -0,0 +1,74 @@

+---
+library_name: transformers
+license: gemma
+base_model: google/gemma-2-9b-it
+tags:
+- trl
+- simpo
+- generated_from_trainer
+model-index:
+- name: gemma-2-9b-it-dpo-train_filtered_full
+  results: []
+---
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+# gemma-2-9b-it-dpo-train_filtered_full
+This model is a fine-tuned version of [google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.6509
+- Rewards/chosen: -0.1749
+- Rewards/rejected: -0.2671
+- Rewards/accuracies: 0.7561
+- Rewards/margins: 0.0922
+- Logps/rejected: -26.7055
+- Logps/chosen: -17.4891
+- Logits/rejected: -24.9311
+- Logits/chosen: -23.9323
+## Model description
+More information needed
+## Intended uses & limitations
+More information needed
+## Training and evaluation data
+More information needed
+## Training procedure
+### Training hyperparameters
+The following hyperparameters were used during training:
+- learning_rate: 5e-07
+- train_batch_size: 2
+- eval_batch_size: 4
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 4
+- gradient_accumulation_steps: 16
+- total_train_batch_size: 128
+- total_eval_batch_size: 16
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 1
+### Training results
+| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
+|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
+| 0.6503        | 0.8724 | 400  | 0.6509          | -0.1749        | -0.2671          | 0.7561             | 0.0922          | -26.7055       | -17.4891     | -24.9311        | -23.9323      |
+### Framework versions
+- Transformers 4.44.2
+- Pytorch 2.7.0+cu128
+- Datasets 2.18.0
+- Tokenizers 0.19.1

all_results.json ADDED Viewed

	@@ -0,0 +1,9 @@

+{
+    "epoch": 0.9989094874591058,
+    "total_flos": 0.0,
+    "train_loss": 0.6719389883191305,
+    "train_runtime": 8958.0836,
+    "train_samples": 58684,
+    "train_samples_per_second": 6.551,
+    "train_steps_per_second": 0.051
+}

generation_config.json ADDED Viewed

	@@ -0,0 +1,8 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "transformers_version": "4.44.2"
+}

train_results.json ADDED Viewed

	@@ -0,0 +1,9 @@

+{
+    "epoch": 0.9989094874591058,
+    "total_flos": 0.0,
+    "train_loss": 0.6719389883191305,
+    "train_runtime": 8958.0836,
+    "train_samples": 58684,
+    "train_samples_per_second": 6.551,
+    "train_steps_per_second": 0.051
+}

trainer_state.json ADDED Viewed

	@@ -0,0 +1,1423 @@

+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9989094874591058,
+  "eval_steps": 400,
+  "global_step": 458,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.010905125408942203,
+      "grad_norm": 0.030929296339124078,
+      "learning_rate": 5.434782608695652e-08,
+      "logits/chosen": -10.946500778198242,
+      "logits/rejected": -10.932920455932617,
+      "logps/chosen": -0.8995386362075806,
+      "logps/rejected": -0.859021008014679,
+      "loss": 0.6946,
+      "rewards/accuracies": 0.4937500059604645,
+      "rewards/chosen": -0.008995385840535164,
+      "rewards/margins": -0.0004051766009069979,
+      "rewards/rejected": -0.008590209297835827,
+      "step": 5
+    },
+    {
+      "epoch": 0.021810250817884406,
+      "grad_norm": 0.056975571795654635,
+      "learning_rate": 1.0869565217391303e-07,
+      "logits/chosen": -9.881834030151367,
+      "logits/rejected": -9.909406661987305,
+      "logps/chosen": -1.1416651010513306,
+      "logps/rejected": -1.1611769199371338,
+      "loss": 0.6945,
+      "rewards/accuracies": 0.512499988079071,
+      "rewards/chosen": -0.011416650377213955,
+      "rewards/margins": 0.00019511766731739044,
+      "rewards/rejected": -0.01161176897585392,
+      "step": 10
+    },
+    {
+      "epoch": 0.03271537622682661,
+      "grad_norm": 0.06260194504125143,
+      "learning_rate": 1.6304347826086955e-07,
+      "logits/chosen": -10.483625411987305,
+      "logits/rejected": -10.386244773864746,
+      "logps/chosen": -0.888972282409668,
+      "logps/rejected": -0.8580414056777954,
+      "loss": 0.6941,
+      "rewards/accuracies": 0.5,
+      "rewards/chosen": -0.008889722637832165,
+      "rewards/margins": -0.00030930957291275263,
+      "rewards/rejected": -0.008580413646996021,
+      "step": 15
+    },
+    {
+      "epoch": 0.04362050163576881,
+      "grad_norm": 0.04432449010316115,
+      "learning_rate": 2.1739130434782607e-07,
+      "logits/chosen": -10.143729209899902,
+      "logits/rejected": -10.027105331420898,
+      "logps/chosen": -0.968403697013855,
+      "logps/rejected": -1.055953860282898,
+      "loss": 0.6943,
+      "rewards/accuracies": 0.5687500238418579,
+      "rewards/chosen": -0.009684036485850811,
+      "rewards/margins": 0.0008755017188377678,
+      "rewards/rejected": -0.010559538379311562,
+      "step": 20
+    },
+    {
+      "epoch": 0.05452562704471101,
+      "grad_norm": 0.03633191693144241,
+      "learning_rate": 2.717391304347826e-07,
+      "logits/chosen": -10.541749954223633,
+      "logits/rejected": -10.645447731018066,
+      "logps/chosen": -0.8666504621505737,
+      "logps/rejected": -0.7926900386810303,
+      "loss": 0.6944,
+      "rewards/accuracies": 0.518750011920929,
+      "rewards/chosen": -0.008666505105793476,
+      "rewards/margins": -0.0007396038272418082,
+      "rewards/rejected": -0.007926899939775467,
+      "step": 25
+    },
+    {
+      "epoch": 0.06543075245365322,
+      "grad_norm": 0.03704281683817436,
+      "learning_rate": 3.260869565217391e-07,
+      "logits/chosen": -10.218794822692871,
+      "logits/rejected": -10.175888061523438,
+      "logps/chosen": -0.8812982439994812,
+      "logps/rejected": -1.040617823600769,
+      "loss": 0.6942,
+      "rewards/accuracies": 0.5562499761581421,
+      "rewards/chosen": -0.00881298165768385,
+      "rewards/margins": 0.001593196764588356,
+      "rewards/rejected": -0.010406178422272205,
+      "step": 30
+    },
+    {
+      "epoch": 0.07633587786259542,
+      "grad_norm": 0.04627201231568104,
+      "learning_rate": 3.8043478260869567e-07,
+      "logits/chosen": -10.650358200073242,
+      "logits/rejected": -10.50601863861084,
+      "logps/chosen": -0.8724371194839478,
+      "logps/rejected": -0.8634036779403687,
+      "loss": 0.6944,
+      "rewards/accuracies": 0.53125,
+      "rewards/chosen": -0.00872437097132206,
+      "rewards/margins": -9.033350215759128e-05,
+      "rewards/rejected": -0.008634037338197231,
+      "step": 35
+    },
+    {
+      "epoch": 0.08724100327153762,
+      "grad_norm": 0.08463024653133087,
+      "learning_rate": 4.3478260869565214e-07,
+      "logits/chosen": -10.552633285522461,
+      "logits/rejected": -10.402359008789062,
+      "logps/chosen": -0.8455907702445984,
+      "logps/rejected": -0.8345605134963989,
+      "loss": 0.6944,
+      "rewards/accuracies": 0.5625,
+      "rewards/chosen": -0.008455907925963402,
+      "rewards/margins": -0.00011030271707568318,
+      "rewards/rejected": -0.008345604874193668,
+      "step": 40
+    },
+    {
+      "epoch": 0.09814612868047982,
+      "grad_norm": 0.07287385137819305,
+      "learning_rate": 4.891304347826087e-07,
+      "logits/chosen": -10.243791580200195,
+      "logits/rejected": -10.251897811889648,
+      "logps/chosen": -0.8864496350288391,
+      "logps/rejected": -0.8499637842178345,
+      "loss": 0.6943,
+      "rewards/accuracies": 0.512499988079071,
+      "rewards/chosen": -0.008864495903253555,
+      "rewards/margins": -0.00036485903547145426,
+      "rewards/rejected": -0.008499637246131897,
+      "step": 45
+    },
+    {
+      "epoch": 0.10905125408942203,
+      "grad_norm": 0.03143058985457917,
+      "learning_rate": 4.998837209058379e-07,
+      "logits/chosen": -10.024384498596191,
+      "logits/rejected": -9.826332092285156,
+      "logps/chosen": -0.965563178062439,
+      "logps/rejected": -1.0626901388168335,
+      "loss": 0.694,
+      "rewards/accuracies": 0.5562499761581421,
+      "rewards/chosen": -0.009655632078647614,
+      "rewards/margins": 0.000971272005699575,
+      "rewards/rejected": -0.010626902803778648,
+      "step": 50
+    },
+    {
+      "epoch": 0.11995637949836423,
+      "grad_norm": 0.03937305345117487,
+      "learning_rate": 4.994115224629204e-07,
+      "logits/chosen": -11.061460494995117,
+      "logits/rejected": -10.943079948425293,
+      "logps/chosen": -0.8814244270324707,
+      "logps/rejected": -0.9907912015914917,
+      "loss": 0.6939,
+      "rewards/accuracies": 0.5562499761581421,
+      "rewards/chosen": -0.008814243599772453,
+      "rewards/margins": 0.0010936675826087594,
+      "rewards/rejected": -0.009907910600304604,
+      "step": 55
+    },
+    {
+      "epoch": 0.13086150490730644,
+      "grad_norm": 0.11654523697246219,
+      "learning_rate": 4.98576823004801e-07,
+      "logits/chosen": -10.834707260131836,
+      "logits/rejected": -10.95356559753418,
+      "logps/chosen": -0.9678617715835571,
+      "logps/rejected": -1.1425087451934814,
+      "loss": 0.6939,
+      "rewards/accuracies": 0.5687500238418579,
+      "rewards/chosen": -0.009678617119789124,
+      "rewards/margins": 0.0017464684788137674,
+      "rewards/rejected": -0.011425087228417397,
+      "step": 60
+    },
+    {
+      "epoch": 0.14176663031624864,
+      "grad_norm": 0.07036248559926651,
+      "learning_rate": 4.973808357047844e-07,
+      "logits/chosen": -11.805063247680664,
+      "logits/rejected": -12.005667686462402,
+      "logps/chosen": -1.0525237321853638,
+      "logps/rejected": -1.302790880203247,
+      "loss": 0.6932,
+      "rewards/accuracies": 0.5874999761581421,
+      "rewards/chosen": -0.010525235906243324,
+      "rewards/margins": 0.0025026723742485046,
+      "rewards/rejected": -0.013027909211814404,
+      "step": 65
+    },
+    {
+      "epoch": 0.15267175572519084,
+      "grad_norm": 0.10218773629789948,
+      "learning_rate": 4.95825298841063e-07,
+      "logits/chosen": -12.53321647644043,
+      "logits/rejected": -12.769686698913574,
+      "logps/chosen": -1.4837205410003662,
+      "logps/rejected": -1.564917802810669,
+      "loss": 0.6937,
+      "rewards/accuracies": 0.6000000238418579,
+      "rewards/chosen": -0.014837203547358513,
+      "rewards/margins": 0.0008119717240333557,
+      "rewards/rejected": -0.01564917527139187,
+      "step": 70
+    },
+    {
+      "epoch": 0.16357688113413305,
+      "grad_norm": 0.0847097031908177,
+      "learning_rate": 4.939124732702595e-07,
+      "logits/chosen": -13.13440227508545,
+      "logits/rejected": -13.338907241821289,
+      "logps/chosen": -1.2520761489868164,
+      "logps/rejected": -1.7030487060546875,
+      "loss": 0.6933,
+      "rewards/accuracies": 0.581250011920929,
+      "rewards/chosen": -0.012520760297775269,
+      "rewards/margins": 0.004509727004915476,
+      "rewards/rejected": -0.017030486837029457,
+      "step": 75
+    },
+    {
+      "epoch": 0.17448200654307525,
+      "grad_norm": 0.06786395465454413,
+      "learning_rate": 4.9164513914144e-07,
+      "logits/chosen": -13.549310684204102,
+      "logits/rejected": -13.727215766906738,
+      "logps/chosen": -1.2481013536453247,
+      "logps/rejected": -1.5681723356246948,
+      "loss": 0.6929,
+      "rewards/accuracies": 0.6000000238418579,
+      "rewards/chosen": -0.01248101331293583,
+      "rewards/margins": 0.00320070912130177,
+      "rewards/rejected": -0.01568172499537468,
+      "step": 80
+    },
+    {
+      "epoch": 0.18538713195201745,
+      "grad_norm": 0.14405979791376544,
+      "learning_rate": 4.890265918553759e-07,
+      "logits/chosen": -13.3426513671875,
+      "logits/rejected": -13.706072807312012,
+      "logps/chosen": -1.1179548501968384,
+      "logps/rejected": -1.5296298265457153,
+      "loss": 0.6928,
+      "rewards/accuracies": 0.606249988079071,
+      "rewards/chosen": -0.011179549619555473,
+      "rewards/margins": 0.004116747993975878,
+      "rewards/rejected": -0.015296298079192638,
+      "step": 85
+    },
+    {
+      "epoch": 0.19629225736095965,
+      "grad_norm": 0.11143154894940574,
+      "learning_rate": 4.860606372749246e-07,
+      "logits/chosen": -13.619897842407227,
+      "logits/rejected": -14.0485258102417,
+      "logps/chosen": -1.3001683950424194,
+      "logps/rejected": -1.6298154592514038,
+      "loss": 0.6925,
+      "rewards/accuracies": 0.59375,
+      "rewards/chosen": -0.013001683168113232,
+      "rewards/margins": 0.003296471433714032,
+      "rewards/rejected": -0.01629815436899662,
+      "step": 90
+    },
+    {
+      "epoch": 0.20719738276990185,
+      "grad_norm": 0.1370023325254743,
+      "learning_rate": 4.827515861934924e-07,
+      "logits/chosen": -14.583166122436523,
+      "logits/rejected": -14.910693168640137,
+      "logps/chosen": -1.7190353870391846,
+      "logps/rejected": -2.3775477409362793,
+      "loss": 0.692,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.0171903558075428,
+      "rewards/margins": 0.00658512394875288,
+      "rewards/rejected": -0.023775478824973106,
+      "step": 95
+    },
+    {
+      "epoch": 0.21810250817884405,
+      "grad_norm": 0.16816122755954477,
+      "learning_rate": 4.791042480696179e-07,
+      "logits/chosen": -15.31629467010498,
+      "logits/rejected": -15.362884521484375,
+      "logps/chosen": -1.7697193622589111,
+      "logps/rejected": -2.579834461212158,
+      "loss": 0.6915,
+      "rewards/accuracies": 0.574999988079071,
+      "rewards/chosen": -0.01769719272851944,
+      "rewards/margins": 0.008101152256131172,
+      "rewards/rejected": -0.025798344984650612,
+      "step": 100
+    },
+    {
+      "epoch": 0.22900763358778625,
+      "grad_norm": 0.10536216668445386,
+      "learning_rate": 4.751239240367838e-07,
+      "logits/chosen": -16.13490867614746,
+      "logits/rejected": -16.28664779663086,
+      "logps/chosen": -1.6209475994110107,
+      "logps/rejected": -1.7623802423477173,
+      "loss": 0.692,
+      "rewards/accuracies": 0.5625,
+      "rewards/chosen": -0.016209475696086884,
+      "rewards/margins": 0.0014143256703391671,
+      "rewards/rejected": -0.01762380264699459,
+      "step": 105
+    },
+    {
+      "epoch": 0.23991275899672845,
+      "grad_norm": 0.07946442768084311,
+      "learning_rate": 4.7081639919861513e-07,
+      "logits/chosen": -16.778369903564453,
+      "logits/rejected": -17.438077926635742,
+      "logps/chosen": -1.8414093255996704,
+      "logps/rejected": -2.527489185333252,
+      "loss": 0.691,
+      "rewards/accuracies": 0.5687500238418579,
+      "rewards/chosen": -0.01841409131884575,
+      "rewards/margins": 0.006860800087451935,
+      "rewards/rejected": -0.025274891406297684,
+      "step": 110
+    },
+    {
+      "epoch": 0.25081788440567065,
+      "grad_norm": 0.19920298946639162,
+      "learning_rate": 4.661879342206636e-07,
+      "logits/chosen": -17.369365692138672,
+      "logits/rejected": -17.71587371826172,
+      "logps/chosen": -2.287747621536255,
+      "logps/rejected": -3.306758165359497,
+      "loss": 0.6906,
+      "rewards/accuracies": 0.6187499761581421,
+      "rewards/chosen": -0.02287747897207737,
+      "rewards/margins": 0.010190103203058243,
+      "rewards/rejected": -0.033067576587200165,
+      "step": 115
+    },
+    {
+      "epoch": 0.2617230098146129,
+      "grad_norm": 0.13511852371641964,
+      "learning_rate": 4.6124525623099745e-07,
+      "logits/chosen": -17.924758911132812,
+      "logits/rejected": -18.15680694580078,
+      "logps/chosen": -2.1572861671447754,
+      "logps/rejected": -2.6086325645446777,
+      "loss": 0.6909,
+      "rewards/accuracies": 0.6312500238418579,
+      "rewards/chosen": -0.021572861820459366,
+      "rewards/margins": 0.004513464402407408,
+      "rewards/rejected": -0.02608632482588291,
+      "step": 120
+    },
+    {
+      "epoch": 0.27262813522355506,
+      "grad_norm": 0.14616514763497487,
+      "learning_rate": 4.5599554904282377e-07,
+      "logits/chosen": -18.488866806030273,
+      "logits/rejected": -18.58377456665039,
+      "logps/chosen": -2.484189987182617,
+      "logps/rejected": -3.0170865058898926,
+      "loss": 0.69,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.024841900914907455,
+      "rewards/margins": 0.00532896351069212,
+      "rewards/rejected": -0.0301708672195673,
+      "step": 125
+    },
+    {
+      "epoch": 0.2835332606324973,
+      "grad_norm": 0.12127246177781695,
+      "learning_rate": 4.5044644271335264e-07,
+      "logits/chosen": -18.254194259643555,
+      "logits/rejected": -18.820096969604492,
+      "logps/chosen": -3.067932605743408,
+      "logps/rejected": -4.546811103820801,
+      "loss": 0.688,
+      "rewards/accuracies": 0.581250011920929,
+      "rewards/chosen": -0.03067932464182377,
+      "rewards/margins": 0.014788785949349403,
+      "rewards/rejected": -0.04546811059117317,
+      "step": 130
+    },
+    {
+      "epoch": 0.29443838604143946,
+      "grad_norm": 0.12658327392034777,
+      "learning_rate": 4.4460600245407876e-07,
+      "logits/chosen": -18.94732093811035,
+      "logits/rejected": -19.414775848388672,
+      "logps/chosen": -3.276075839996338,
+      "logps/rejected": -4.676281452178955,
+      "loss": 0.6891,
+      "rewards/accuracies": 0.6499999761581421,
+      "rewards/chosen": -0.03276075795292854,
+      "rewards/margins": 0.014002054929733276,
+      "rewards/rejected": -0.04676281288266182,
+      "step": 135
+    },
+    {
+      "epoch": 0.3053435114503817,
+      "grad_norm": 0.10760144841394713,
+      "learning_rate": 4.3848271690859927e-07,
+      "logits/chosen": -18.926616668701172,
+      "logits/rejected": -19.358631134033203,
+      "logps/chosen": -3.4336490631103516,
+      "logps/rejected": -4.3059892654418945,
+      "loss": 0.6895,
+      "rewards/accuracies": 0.574999988079071,
+      "rewards/chosen": -0.03433648869395256,
+      "rewards/margins": 0.008723398670554161,
+      "rewards/rejected": -0.04305989295244217,
+      "step": 140
+    },
+    {
+      "epoch": 0.31624863685932386,
+      "grad_norm": 0.1382830404713516,
+      "learning_rate": 4.3208548581500455e-07,
+      "logits/chosen": -19.934051513671875,
+      "logits/rejected": -19.995485305786133,
+      "logps/chosen": -3.247889995574951,
+      "logps/rejected": -4.867912769317627,
+      "loss": 0.6869,
+      "rewards/accuracies": 0.6875,
+      "rewards/chosen": -0.032478902488946915,
+      "rewards/margins": 0.016200218349695206,
+      "rewards/rejected": -0.04867911711335182,
+      "step": 145
+    },
+    {
+      "epoch": 0.3271537622682661,
+      "grad_norm": 0.17855337171133584,
+      "learning_rate": 4.2542360707077327e-07,
+      "logits/chosen": -20.74686050415039,
+      "logits/rejected": -20.895763397216797,
+      "logps/chosen": -3.7406821250915527,
+      "logps/rejected": -4.3149309158325195,
+      "loss": 0.6888,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.03740682452917099,
+      "rewards/margins": 0.005742488894611597,
+      "rewards/rejected": -0.04314931482076645,
+      "step": 150
+    },
+    {
+      "epoch": 0.33805888767720826,
+      "grad_norm": 0.412241203723386,
+      "learning_rate": 4.185067632189737e-07,
+      "logits/chosen": -20.4768009185791,
+      "logits/rejected": -20.78377914428711,
+      "logps/chosen": -4.34366512298584,
+      "logps/rejected": -5.832276821136475,
+      "loss": 0.6878,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.04343665391206741,
+      "rewards/margins": 0.014886116608977318,
+      "rewards/rejected": -0.05832276865839958,
+      "step": 155
+    },
+    {
+      "epoch": 0.3489640130861505,
+      "grad_norm": 0.18153876299727773,
+      "learning_rate": 4.113450073754102e-07,
+      "logits/chosen": -20.895898818969727,
+      "logits/rejected": -21.19687843322754,
+      "logps/chosen": -5.16976261138916,
+      "logps/rejected": -6.861837863922119,
+      "loss": 0.6872,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.05169763043522835,
+      "rewards/margins": 0.016920754685997963,
+      "rewards/rejected": -0.06861837953329086,
+      "step": 160
+    },
+    {
+      "epoch": 0.35986913849509267,
+      "grad_norm": 0.21342125732127162,
+      "learning_rate": 4.039487486171707e-07,
+      "logits/chosen": -21.601755142211914,
+      "logits/rejected": -21.642393112182617,
+      "logps/chosen": -4.909188270568848,
+      "logps/rejected": -6.523903846740723,
+      "loss": 0.6864,
+      "rewards/accuracies": 0.606249988079071,
+      "rewards/chosen": -0.0490918830037117,
+      "rewards/margins": 0.016147149726748466,
+      "rewards/rejected": -0.06523902714252472,
+      "step": 165
+    },
+    {
+      "epoch": 0.3707742639040349,
+      "grad_norm": 0.22893947928190392,
+      "learning_rate": 3.963287368538105e-07,
+      "logits/chosen": -22.623598098754883,
+      "logits/rejected": -22.650497436523438,
+      "logps/chosen": -6.888545036315918,
+      "logps/rejected": -8.499242782592773,
+      "loss": 0.6853,
+      "rewards/accuracies": 0.6000000238418579,
+      "rewards/chosen": -0.06888546049594879,
+      "rewards/margins": 0.0161069817841053,
+      "rewards/rejected": -0.0849924311041832,
+      "step": 170
+    },
+    {
+      "epoch": 0.3816793893129771,
+      "grad_norm": 0.288117238602039,
+      "learning_rate": 3.884960472031622e-07,
+      "logits/chosen": -23.40947151184082,
+      "logits/rejected": -23.49216079711914,
+      "logps/chosen": -6.957637786865234,
+      "logps/rejected": -8.827032089233398,
+      "loss": 0.683,
+      "rewards/accuracies": 0.706250011920929,
+      "rewards/chosen": -0.06957637518644333,
+      "rewards/margins": 0.018693946301937103,
+      "rewards/rejected": -0.08827032148838043,
+      "step": 175
+    },
+    {
+      "epoch": 0.3925845147219193,
+      "grad_norm": 0.27552264475755045,
+      "learning_rate": 3.804620638944791e-07,
+      "logits/chosen": -22.68706512451172,
+      "logits/rejected": -22.84552764892578,
+      "logps/chosen": -9.296040534973145,
+      "logps/rejected": -11.636777877807617,
+      "loss": 0.6843,
+      "rewards/accuracies": 0.6937500238418579,
+      "rewards/chosen": -0.0929604023694992,
+      "rewards/margins": 0.023407381027936935,
+      "rewards/rejected": -0.11636777967214584,
+      "step": 180
+    },
+    {
+      "epoch": 0.4034896401308615,
+      "grad_norm": 0.27805736438529915,
+      "learning_rate": 3.722384637223084e-07,
+      "logits/chosen": -24.04327392578125,
+      "logits/rejected": -24.12721824645996,
+      "logps/chosen": -9.727739334106445,
+      "logps/rejected": -11.088327407836914,
+      "loss": 0.6848,
+      "rewards/accuracies": 0.6625000238418579,
+      "rewards/chosen": -0.09727739542722702,
+      "rewards/margins": 0.013605867512524128,
+      "rewards/rejected": -0.11088325828313828,
+      "step": 185
+    },
+    {
+      "epoch": 0.4143947655398037,
+      "grad_norm": 0.27448426639927703,
+      "learning_rate": 3.6383719907514274e-07,
+      "logits/chosen": -23.48944091796875,
+      "logits/rejected": -23.55702781677246,
+      "logps/chosen": -11.73414134979248,
+      "logps/rejected": -14.337844848632812,
+      "loss": 0.6804,
+      "rewards/accuracies": 0.643750011920929,
+      "rewards/chosen": -0.11734139919281006,
+      "rewards/margins": 0.026037033647298813,
+      "rewards/rejected": -0.14337845146656036,
+      "step": 190
+    },
+    {
+      "epoch": 0.42529989094874593,
+      "grad_norm": 0.30500743869421215,
+      "learning_rate": 3.5527048056351654e-07,
+      "logits/chosen": -24.235092163085938,
+      "logits/rejected": -24.290729522705078,
+      "logps/chosen": -10.834362030029297,
+      "logps/rejected": -13.723051071166992,
+      "loss": 0.6817,
+      "rewards/accuracies": 0.6499999761581421,
+      "rewards/chosen": -0.10834361612796783,
+      "rewards/margins": 0.028886884450912476,
+      "rewards/rejected": -0.1372305005788803,
+      "step": 195
+    },
+    {
+      "epoch": 0.4362050163576881,
+      "grad_norm": 0.37377098173604945,
+      "learning_rate": 3.465507592727957e-07,
+      "logits/chosen": -23.979053497314453,
+      "logits/rejected": -24.071990966796875,
+      "logps/chosen": -12.171896934509277,
+      "logps/rejected": -16.258481979370117,
+      "loss": 0.6799,
+      "rewards/accuracies": 0.731249988079071,
+      "rewards/chosen": -0.12171895802021027,
+      "rewards/margins": 0.04086584970355034,
+      "rewards/rejected": -0.1625848114490509,
+      "step": 200
+    },
+    {
+      "epoch": 0.44711014176663033,
+      "grad_norm": 0.37089627410325865,
+      "learning_rate": 3.37690708666455e-07,
+      "logits/chosen": -24.075651168823242,
+      "logits/rejected": -24.11351203918457,
+      "logps/chosen": -12.68486499786377,
+      "logps/rejected": -14.939569473266602,
+      "loss": 0.6807,
+      "rewards/accuracies": 0.668749988079071,
+      "rewards/chosen": -0.12684863805770874,
+      "rewards/margins": 0.022547055035829544,
+      "rewards/rejected": -0.14939570426940918,
+      "step": 205
+    },
+    {
+      "epoch": 0.4580152671755725,
+      "grad_norm": 0.37053905747928084,
+      "learning_rate": 3.287032061661462e-07,
+      "logits/chosen": -24.205493927001953,
+      "logits/rejected": -24.298234939575195,
+      "logps/chosen": -13.609357833862305,
+      "logps/rejected": -18.634946823120117,
+      "loss": 0.6757,
+      "rewards/accuracies": 0.6937500238418579,
+      "rewards/chosen": -0.13609358668327332,
+      "rewards/margins": 0.050255876034498215,
+      "rewards/rejected": -0.18634946644306183,
+      "step": 210
+    },
+    {
+      "epoch": 0.46892039258451473,
+      "grad_norm": 0.3633800041091051,
+      "learning_rate": 3.196013144353274e-07,
+      "logits/chosen": -24.66036033630371,
+      "logits/rejected": -24.763050079345703,
+      "logps/chosen": -13.911908149719238,
+      "logps/rejected": -17.66451644897461,
+      "loss": 0.6782,
+      "rewards/accuracies": 0.706250011920929,
+      "rewards/chosen": -0.13911907374858856,
+      "rewards/margins": 0.03752611204981804,
+      "rewards/rejected": -0.1766451895236969,
+      "step": 215
+    },
+    {
+      "epoch": 0.4798255179934569,
+      "grad_norm": 1.0948303510787734,
+      "learning_rate": 3.103982623936575e-07,
+      "logits/chosen": -24.110694885253906,
+      "logits/rejected": -24.23526954650879,
+      "logps/chosen": -15.881921768188477,
+      "logps/rejected": -20.27142906188965,
+      "loss": 0.6752,
+      "rewards/accuracies": 0.706250011920929,
+      "rewards/chosen": -0.15881919860839844,
+      "rewards/margins": 0.043895088136196136,
+      "rewards/rejected": -0.20271429419517517,
+      "step": 220
+    },
+    {
+      "epoch": 0.49073064340239914,
+      "grad_norm": 0.5197470019914936,
+      "learning_rate": 3.011074259897499e-07,
+      "logits/chosen": -24.41493034362793,
+      "logits/rejected": -24.538925170898438,
+      "logps/chosen": -15.199113845825195,
+      "logps/rejected": -19.166044235229492,
+      "loss": 0.6746,
+      "rewards/accuracies": 0.6499999761581421,
+      "rewards/chosen": -0.1519911140203476,
+      "rewards/margins": 0.03966929763555527,
+      "rewards/rejected": -0.19166040420532227,
+      "step": 225
+    },
+    {
+      "epoch": 0.5016357688113413,
+      "grad_norm": 0.6808943035909418,
+      "learning_rate": 2.9174230876023054e-07,
+      "logits/chosen": -24.559289932250977,
+      "logits/rejected": -24.69730567932129,
+      "logps/chosen": -18.434986114501953,
+      "logps/rejected": -22.413331985473633,
+      "loss": 0.6766,
+      "rewards/accuracies": 0.6812499761581421,
+      "rewards/chosen": -0.18434984982013702,
+      "rewards/margins": 0.03978346660733223,
+      "rewards/rejected": -0.22413332760334015,
+      "step": 230
+    },
+    {
+      "epoch": 0.5125408942202835,
+      "grad_norm": 0.5653763032514039,
+      "learning_rate": 2.8231652220335603e-07,
+      "logits/chosen": -24.52471351623535,
+      "logits/rejected": -24.600597381591797,
+      "logps/chosen": -18.48925018310547,
+      "logps/rejected": -23.984424591064453,
+      "loss": 0.6707,
+      "rewards/accuracies": 0.643750011920929,
+      "rewards/chosen": -0.18489250540733337,
+      "rewards/margins": 0.0549517385661602,
+      "rewards/rejected": -0.23984424769878387,
+      "step": 235
+    },
+    {
+      "epoch": 0.5234460196292258,
+      "grad_norm": 0.6497132806211787,
+      "learning_rate": 2.728437659957177e-07,
+      "logits/chosen": -24.80784797668457,
+      "logits/rejected": -24.986042022705078,
+      "logps/chosen": -19.79264259338379,
+      "logps/rejected": -24.296344757080078,
+      "loss": 0.6715,
+      "rewards/accuracies": 0.637499988079071,
+      "rewards/chosen": -0.19792641699314117,
+      "rewards/margins": 0.045037031173706055,
+      "rewards/rejected": -0.24296343326568604,
+      "step": 240
+    },
+    {
+      "epoch": 0.5343511450381679,
+      "grad_norm": 0.9626814200638923,
+      "learning_rate": 2.6333780808078594e-07,
+      "logits/chosen": -24.93509292602539,
+      "logits/rejected": -25.033912658691406,
+      "logps/chosen": -22.77157211303711,
+      "logps/rejected": -29.090633392333984,
+      "loss": 0.6665,
+      "rewards/accuracies": 0.78125,
+      "rewards/chosen": -0.22771573066711426,
+      "rewards/margins": 0.06319063156843185,
+      "rewards/rejected": -0.2909063696861267,
+      "step": 245
+    },
+    {
+      "epoch": 0.5452562704471101,
+      "grad_norm": 0.8758115163547733,
+      "learning_rate": 2.538124646582315e-07,
+      "logits/chosen": -24.20846176147461,
+      "logits/rejected": -24.555444717407227,
+      "logps/chosen": -13.707853317260742,
+      "logps/rejected": -20.560867309570312,
+      "loss": 0.6683,
+      "rewards/accuracies": 0.7124999761581421,
+      "rewards/chosen": -0.13707853853702545,
+      "rewards/margins": 0.0685301423072815,
+      "rewards/rejected": -0.20560868084430695,
+      "step": 250
+    },
+    {
+      "epoch": 0.5561613958560524,
+      "grad_norm": 1.2031911734843694,
+      "learning_rate": 2.442815801031128e-07,
+      "logits/chosen": -24.35436248779297,
+      "logits/rejected": -24.653484344482422,
+      "logps/chosen": -15.509927749633789,
+      "logps/rejected": -22.37173080444336,
+      "loss": 0.6642,
+      "rewards/accuracies": 0.793749988079071,
+      "rewards/chosen": -0.1550992727279663,
+      "rewards/margins": 0.06861802190542221,
+      "rewards/rejected": -0.22371730208396912,
+      "step": 255
+    },
+    {
+      "epoch": 0.5670665212649946,
+      "grad_norm": 1.26121798704608,
+      "learning_rate": 2.3475900684411025e-07,
+      "logits/chosen": -24.240325927734375,
+      "logits/rejected": -24.665193557739258,
+      "logps/chosen": -14.284322738647461,
+      "logps/rejected": -21.667491912841797,
+      "loss": 0.665,
+      "rewards/accuracies": 0.768750011920929,
+      "rewards/chosen": -0.14284323155879974,
+      "rewards/margins": 0.07383168488740921,
+      "rewards/rejected": -0.21667489409446716,
+      "step": 260
+    },
+    {
+      "epoch": 0.5779716466739367,
+      "grad_norm": 1.4926633522783754,
+      "learning_rate": 2.2525858523005637e-07,
+      "logits/chosen": -24.77151870727539,
+      "logits/rejected": -25.164073944091797,
+      "logps/chosen": -19.22076416015625,
+      "logps/rejected": -25.62625503540039,
+      "loss": 0.664,
+      "rewards/accuracies": 0.6937500238418579,
+      "rewards/chosen": -0.19220763444900513,
+      "rewards/margins": 0.06405490636825562,
+      "rewards/rejected": -0.25626254081726074,
+      "step": 265
+    },
+    {
+      "epoch": 0.5888767720828789,
+      "grad_norm": 1.9007333677964324,
+      "learning_rate": 2.1579412341402248e-07,
+      "logits/chosen": -24.617694854736328,
+      "logits/rejected": -24.862285614013672,
+      "logps/chosen": -16.634843826293945,
+      "logps/rejected": -22.411142349243164,
+      "loss": 0.6635,
+      "rewards/accuracies": 0.7124999761581421,
+      "rewards/chosen": -0.1663484275341034,
+      "rewards/margins": 0.05776297301054001,
+      "rewards/rejected": -0.2241113930940628,
+      "step": 270
+    },
+    {
+      "epoch": 0.5997818974918212,
+      "grad_norm": 1.5526395061631688,
+      "learning_rate": 2.0637937728420008e-07,
+      "logits/chosen": -24.820884704589844,
+      "logits/rejected": -25.266693115234375,
+      "logps/chosen": -19.834583282470703,
+      "logps/rejected": -27.931264877319336,
+      "loss": 0.6597,
+      "rewards/accuracies": 0.737500011920929,
+      "rewards/chosen": -0.1983458548784256,
+      "rewards/margins": 0.08096680045127869,
+      "rewards/rejected": -0.2793126702308655,
+      "step": 275
+    },
+    {
+      "epoch": 0.6106870229007634,
+      "grad_norm": 1.7934268985600719,
+      "learning_rate": 1.970280304707447e-07,
+      "logits/chosen": -24.92822265625,
+      "logits/rejected": -25.202295303344727,
+      "logps/chosen": -14.220446586608887,
+      "logps/rejected": -22.21088981628418,
+      "loss": 0.661,
+      "rewards/accuracies": 0.78125,
+      "rewards/chosen": -0.14220447838306427,
+      "rewards/margins": 0.07990442216396332,
+      "rewards/rejected": -0.22210891544818878,
+      "step": 280
+    },
+    {
+      "epoch": 0.6215921483097055,
+      "grad_norm": 2.748107098549448,
+      "learning_rate": 1.8775367445764204e-07,
+      "logits/chosen": -24.601261138916016,
+      "logits/rejected": -25.075407028198242,
+      "logps/chosen": -17.336923599243164,
+      "logps/rejected": -23.92327880859375,
+      "loss": 0.6637,
+      "rewards/accuracies": 0.7437499761581421,
+      "rewards/chosen": -0.17336924374103546,
+      "rewards/margins": 0.06586353480815887,
+      "rewards/rejected": -0.23923280835151672,
+      "step": 285
+    },
+    {
+      "epoch": 0.6324972737186477,
+      "grad_norm": 2.6772560842896804,
+      "learning_rate": 1.785697888285011e-07,
+      "logits/chosen": -25.160493850708008,
+      "logits/rejected": -25.41177749633789,
+      "logps/chosen": -18.49761390686035,
+      "logps/rejected": -24.65076446533203,
+      "loss": 0.6609,
+      "rewards/accuracies": 0.731249988079071,
+      "rewards/chosen": -0.18497613072395325,
+      "rewards/margins": 0.061531513929367065,
+      "rewards/rejected": -0.2465076446533203,
+      "step": 290
+    },
+    {
+      "epoch": 0.64340239912759,
+      "grad_norm": 2.024301326251699,
+      "learning_rate": 1.6948972167498649e-07,
+      "logits/chosen": -25.294282913208008,
+      "logits/rejected": -25.73337745666504,
+      "logps/chosen": -17.11026954650879,
+      "logps/rejected": -24.634340286254883,
+      "loss": 0.6587,
+      "rewards/accuracies": 0.6875,
+      "rewards/chosen": -0.17110270261764526,
+      "rewards/margins": 0.07524070143699646,
+      "rewards/rejected": -0.24634337425231934,
+      "step": 295
+    },
+    {
+      "epoch": 0.6543075245365322,
+      "grad_norm": 2.7307785987453963,
+      "learning_rate": 1.605266701963646e-07,
+      "logits/chosen": -25.117237091064453,
+      "logits/rejected": -25.448089599609375,
+      "logps/chosen": -17.745468139648438,
+      "logps/rejected": -26.143442153930664,
+      "loss": 0.6547,
+      "rewards/accuracies": 0.7562500238418579,
+      "rewards/chosen": -0.17745468020439148,
+      "rewards/margins": 0.08397971838712692,
+      "rewards/rejected": -0.261434406042099,
+      "step": 300
+    },
+    {
+      "epoch": 0.6652126499454744,
+      "grad_norm": 1.857948182227819,
+      "learning_rate": 1.5169366151835994e-07,
+      "logits/chosen": -25.616552352905273,
+      "logits/rejected": -25.87392807006836,
+      "logps/chosen": -23.979658126831055,
+      "logps/rejected": -29.496234893798828,
+      "loss": 0.6583,
+      "rewards/accuracies": 0.6625000238418579,
+      "rewards/chosen": -0.23979659378528595,
+      "rewards/margins": 0.05516573786735535,
+      "rewards/rejected": -0.2949623465538025,
+      "step": 305
+    },
+    {
+      "epoch": 0.6761177753544165,
+      "grad_norm": 2.5256107025497885,
+      "learning_rate": 1.430035337592018e-07,
+      "logits/chosen": -25.01370620727539,
+      "logits/rejected": -25.45524787902832,
+      "logps/chosen": -19.495426177978516,
+      "logps/rejected": -28.25351333618164,
+      "loss": 0.6583,
+      "rewards/accuracies": 0.7437499761581421,
+      "rewards/chosen": -0.19495424628257751,
+      "rewards/margins": 0.0875808596611023,
+      "rewards/rejected": -0.2825351357460022,
+      "step": 310
+    },
+    {
+      "epoch": 0.6870229007633588,
+      "grad_norm": 2.8615478848624023,
+      "learning_rate": 1.3446891737037762e-07,
+      "logits/chosen": -24.765771865844727,
+      "logits/rejected": -25.376537322998047,
+      "logps/chosen": -17.96234130859375,
+      "logps/rejected": -26.534252166748047,
+      "loss": 0.6546,
+      "rewards/accuracies": 0.7124999761581421,
+      "rewards/chosen": -0.17962339520454407,
+      "rewards/margins": 0.08571910858154297,
+      "rewards/rejected": -0.26534250378608704,
+      "step": 315
+    },
+    {
+      "epoch": 0.697928026172301,
+      "grad_norm": 2.9485949341956044,
+      "learning_rate": 1.261022167792161e-07,
+      "logits/chosen": -25.101655960083008,
+      "logits/rejected": -25.687358856201172,
+      "logps/chosen": -20.705102920532227,
+      "logps/rejected": -29.696491241455078,
+      "loss": 0.6557,
+      "rewards/accuracies": 0.7437499761581421,
+      "rewards/chosen": -0.20705100893974304,
+      "rewards/margins": 0.08991388976573944,
+      "rewards/rejected": -0.2969648838043213,
+      "step": 320
+    },
+    {
+      "epoch": 0.7088331515812432,
+      "grad_norm": 2.2993871611090584,
+      "learning_rate": 1.1791559235997833e-07,
+      "logits/chosen": -25.622081756591797,
+      "logits/rejected": -25.974563598632812,
+      "logps/chosen": -25.657718658447266,
+      "logps/rejected": -31.523218154907227,
+      "loss": 0.6603,
+      "rewards/accuracies": 0.65625,
+      "rewards/chosen": -0.25657719373703003,
+      "rewards/margins": 0.05865497514605522,
+      "rewards/rejected": -0.31523215770721436,
+      "step": 325
+    },
+    {
+      "epoch": 0.7197382769901853,
+      "grad_norm": 2.777928096129579,
+      "learning_rate": 1.0992094275966254e-07,
+      "logits/chosen": -25.372146606445312,
+      "logits/rejected": -25.923391342163086,
+      "logps/chosen": -21.982194900512695,
+      "logps/rejected": -31.152759552001953,
+      "loss": 0.6558,
+      "rewards/accuracies": 0.699999988079071,
+      "rewards/chosen": -0.21982192993164062,
+      "rewards/margins": 0.09170565754175186,
+      "rewards/rejected": -0.3115276098251343,
+      "step": 330
+    },
+    {
+      "epoch": 0.7306434023991276,
+      "grad_norm": 2.83155273465803,
+      "learning_rate": 1.0212988760420918e-07,
+      "logits/chosen": -24.919795989990234,
+      "logits/rejected": -25.5791015625,
+      "logps/chosen": -18.939666748046875,
+      "logps/rejected": -28.968042373657227,
+      "loss": 0.649,
+      "rewards/accuracies": 0.768750011920929,
+      "rewards/chosen": -0.1893966645002365,
+      "rewards/margins": 0.10028377920389175,
+      "rewards/rejected": -0.28968045115470886,
+      "step": 335
+    },
+    {
+      "epoch": 0.7415485278080698,
+      "grad_norm": 3.183452120789625,
+      "learning_rate": 9.455375061024317e-08,
+      "logits/chosen": -24.678586959838867,
+      "logits/rejected": -25.56129264831543,
+      "logps/chosen": -15.477849006652832,
+      "logps/rejected": -26.394145965576172,
+      "loss": 0.6509,
+      "rewards/accuracies": 0.793749988079071,
+      "rewards/chosen": -0.15477848052978516,
+      "rewards/margins": 0.10916298627853394,
+      "rewards/rejected": -0.2639414668083191,
+      "step": 340
+    },
+    {
+      "epoch": 0.752453653217012,
+      "grad_norm": 3.530921544143697,
+      "learning_rate": 8.720354312689751e-08,
+      "logits/chosen": -24.498729705810547,
+      "logits/rejected": -25.28777313232422,
+      "logps/chosen": -20.05240249633789,
+      "logps/rejected": -29.38003158569336,
+      "loss": 0.6518,
+      "rewards/accuracies": 0.7250000238418579,
+      "rewards/chosen": -0.20052401721477509,
+      "rewards/margins": 0.09327626973390579,
+      "rewards/rejected": -0.2938002943992615,
+      "step": 345
+    },
+    {
+      "epoch": 0.7633587786259542,
+      "grad_norm": 3.0505910146648727,
+      "learning_rate": 8.008994813163994e-08,
+      "logits/chosen": -24.892383575439453,
+      "logits/rejected": -25.568103790283203,
+      "logps/chosen": -21.45001220703125,
+      "logps/rejected": -31.58233070373535,
+      "loss": 0.6533,
+      "rewards/accuracies": 0.731249988079071,
+      "rewards/chosen": -0.21450011432170868,
+      "rewards/margins": 0.1013231873512268,
+      "rewards/rejected": -0.3158233165740967,
+      "step": 350
+    },
+    {
+      "epoch": 0.7742639040348964,
+      "grad_norm": 3.3936252259046684,
+      "learning_rate": 7.322330470336313e-08,
+      "logits/chosen": -25.23516845703125,
+      "logits/rejected": -25.73038673400879,
+      "logps/chosen": -23.45631217956543,
+      "logps/rejected": -33.43750762939453,
+      "loss": 0.652,
+      "rewards/accuracies": 0.675000011920929,
+      "rewards/chosen": -0.23456314206123352,
+      "rewards/margins": 0.09981193393468857,
+      "rewards/rejected": -0.3343750536441803,
+      "step": 355
+    },
+    {
+      "epoch": 0.7851690294438386,
+      "grad_norm": 3.1909990002329427,
+      "learning_rate": 6.661359299530626e-08,
+      "logits/chosen": -24.70314598083496,
+      "logits/rejected": -25.460840225219727,
+      "logps/chosen": -20.430919647216797,
+      "logps/rejected": -31.104305267333984,
+      "loss": 0.6483,
+      "rewards/accuracies": 0.793749988079071,
+      "rewards/chosen": -0.20430922508239746,
+      "rewards/margins": 0.10673384368419647,
+      "rewards/rejected": -0.3110430836677551,
+      "step": 360
+    },
+    {
+      "epoch": 0.7960741548527808,
+      "grad_norm": 3.291992655231796,
+      "learning_rate": 6.027041972964766e-08,
+      "logits/chosen": -24.697559356689453,
+      "logits/rejected": -25.534496307373047,
+      "logps/chosen": -20.09422492980957,
+      "logps/rejected": -30.154062271118164,
+      "loss": 0.6521,
+      "rewards/accuracies": 0.731249988079071,
+      "rewards/chosen": -0.2009422332048416,
+      "rewards/margins": 0.10059837996959686,
+      "rewards/rejected": -0.3015406131744385,
+      "step": 365
+    },
+    {
+      "epoch": 0.806979280261723,
+      "grad_norm": 3.2949559481980986,
+      "learning_rate": 5.4203004234851664e-08,
+      "logits/chosen": -24.35785675048828,
+      "logits/rejected": -25.193906784057617,
+      "logps/chosen": -21.446033477783203,
+      "logps/rejected": -31.33773422241211,
+      "loss": 0.647,
+      "rewards/accuracies": 0.731249988079071,
+      "rewards/chosen": -0.2144603431224823,
+      "rewards/margins": 0.09891701489686966,
+      "rewards/rejected": -0.313377320766449,
+      "step": 370
+    },
+    {
+      "epoch": 0.8178844056706652,
+      "grad_norm": 3.594122355037719,
+      "learning_rate": 4.842016504606375e-08,
+      "logits/chosen": -24.57286262512207,
+      "logits/rejected": -25.43461036682129,
+      "logps/chosen": -21.629987716674805,
+      "logps/rejected": -32.51239776611328,
+      "loss": 0.6504,
+      "rewards/accuracies": 0.768750011920929,
+      "rewards/chosen": -0.2162998616695404,
+      "rewards/margins": 0.10882411152124405,
+      "rewards/rejected": -0.32512399554252625,
+      "step": 375
+    },
+    {
+      "epoch": 0.8287895310796074,
+      "grad_norm": 3.1838157026232135,
+      "learning_rate": 4.293030708802833e-08,
+      "logits/chosen": -24.03682518005371,
+      "logits/rejected": -24.663799285888672,
+      "logps/chosen": -21.035297393798828,
+      "logps/rejected": -28.495147705078125,
+      "loss": 0.6511,
+      "rewards/accuracies": 0.7124999761581421,
+      "rewards/chosen": -0.21035297214984894,
+      "rewards/margins": 0.0745985358953476,
+      "rewards/rejected": -0.28495147824287415,
+      "step": 380
+    },
+    {
+      "epoch": 0.8396946564885496,
+      "grad_norm": 3.6512115795796376,
+      "learning_rate": 3.7741409459158576e-08,
+      "logits/chosen": -23.64044761657715,
+      "logits/rejected": -24.594707489013672,
+      "logps/chosen": -18.868743896484375,
+      "logps/rejected": -27.33072280883789,
+      "loss": 0.6558,
+      "rewards/accuracies": 0.699999988079071,
+      "rewards/chosen": -0.18868745863437653,
+      "rewards/margins": 0.08461979776620865,
+      "rewards/rejected": -0.2733072340488434,
+      "step": 385
+    },
+    {
+      "epoch": 0.8505997818974919,
+      "grad_norm": 3.336997336682689,
+      "learning_rate": 3.286101383451284e-08,
+      "logits/chosen": -23.809391021728516,
+      "logits/rejected": -24.789653778076172,
+      "logps/chosen": -19.703807830810547,
+      "logps/rejected": -31.056636810302734,
+      "loss": 0.6508,
+      "rewards/accuracies": 0.7437499761581421,
+      "rewards/chosen": -0.19703808426856995,
+      "rewards/margins": 0.1135282889008522,
+      "rewards/rejected": -0.31056636571884155,
+      "step": 390
+    },
+    {
+      "epoch": 0.861504907306434,
+      "grad_norm": 3.650332846610603,
+      "learning_rate": 2.8296213504533596e-08,
+      "logits/chosen": -23.612497329711914,
+      "logits/rejected": -24.643917083740234,
+      "logps/chosen": -18.772497177124023,
+      "logps/rejected": -29.0363712310791,
+      "loss": 0.6549,
+      "rewards/accuracies": 0.768750011920929,
+      "rewards/chosen": -0.1877249926328659,
+      "rewards/margins": 0.10263872146606445,
+      "rewards/rejected": -0.29036369919776917,
+      "step": 395
+    },
+    {
+      "epoch": 0.8724100327153762,
+      "grad_norm": 3.1463089972485228,
+      "learning_rate": 2.405364306547955e-08,
+      "logits/chosen": -23.916240692138672,
+      "logits/rejected": -24.74551773071289,
+      "logps/chosen": -18.31199073791504,
+      "logps/rejected": -27.967397689819336,
+      "loss": 0.6503,
+      "rewards/accuracies": 0.731249988079071,
+      "rewards/chosen": -0.18311990797519684,
+      "rewards/margins": 0.09655408561229706,
+      "rewards/rejected": -0.2796739935874939,
+      "step": 400
+    },
+    {
+      "epoch": 0.8724100327153762,
+      "eval_logits/chosen": -23.932262420654297,
+      "eval_logits/rejected": -24.931102752685547,
+      "eval_logps/chosen": -17.48914909362793,
+      "eval_logps/rejected": -26.70553207397461,
+      "eval_loss": 0.6508545279502869,
+      "eval_rewards/accuracies": 0.756147563457489,
+      "eval_rewards/chosen": -0.17489148676395416,
+      "eval_rewards/margins": 0.09216383844614029,
+      "eval_rewards/rejected": -0.26705533266067505,
+      "eval_runtime": 90.5043,
+      "eval_samples_per_second": 21.446,
+      "eval_steps_per_second": 1.348,
+      "step": 400
+    },
+    {
+      "epoch": 0.8833151581243184,
+      "grad_norm": 3.1425773193083977,
+      "learning_rate": 2.0139468776535718e-08,
+      "logits/chosen": -23.820995330810547,
+      "logits/rejected": -24.96358299255371,
+      "logps/chosen": -19.28446388244629,
+      "logps/rejected": -29.758419036865234,
+      "loss": 0.6522,
+      "rewards/accuracies": 0.762499988079071,
+      "rewards/chosen": -0.19284464418888092,
+      "rewards/margins": 0.10473956167697906,
+      "rewards/rejected": -0.2975841462612152,
+      "step": 405
+    },
+    {
+      "epoch": 0.8942202835332607,
+      "grad_norm": 2.745854518026326,
+      "learning_rate": 1.6559379597616136e-08,
+      "logits/chosen": -23.470592498779297,
+      "logits/rejected": -24.550033569335938,
+      "logps/chosen": -18.000120162963867,
+      "logps/rejected": -28.478153228759766,
+      "loss": 0.6498,
+      "rewards/accuracies": 0.762499988079071,
+      "rewards/chosen": -0.18000119924545288,
+      "rewards/margins": 0.10478033125400543,
+      "rewards/rejected": -0.2847815155982971,
+      "step": 410
+    },
+    {
+      "epoch": 0.9051254089422028,
+      "grad_norm": 3.963393459289569,
+      "learning_rate": 1.3318578920886003e-08,
+      "logits/chosen": -23.724620819091797,
+      "logits/rejected": -24.733633041381836,
+      "logps/chosen": -21.162649154663086,
+      "logps/rejected": -32.321632385253906,
+      "loss": 0.6462,
+      "rewards/accuracies": 0.768750011920929,
+      "rewards/chosen": -0.21162649989128113,
+      "rewards/margins": 0.11158982664346695,
+      "rewards/rejected": -0.3232163190841675,
+      "step": 415
+    },
+    {
+      "epoch": 0.916030534351145,
+      "grad_norm": 3.646649584092885,
+      "learning_rate": 1.042177700801966e-08,
+      "logits/chosen": -23.974315643310547,
+      "logits/rejected": -25.02378273010254,
+      "logps/chosen": -19.89191436767578,
+      "logps/rejected": -31.294485092163086,
+      "loss": 0.6466,
+      "rewards/accuracies": 0.699999988079071,
+      "rewards/chosen": -0.1989191472530365,
+      "rewards/margins": 0.11402571201324463,
+      "rewards/rejected": -0.31294482946395874,
+      "step": 420
+    },
+    {
+      "epoch": 0.9269356597600873,
+      "grad_norm": 2.910140700569881,
+      "learning_rate": 7.873184144186972e-09,
+      "logits/chosen": -23.87249183654785,
+      "logits/rejected": -24.726253509521484,
+      "logps/chosen": -19.595571517944336,
+      "logps/rejected": -29.641769409179688,
+      "loss": 0.646,
+      "rewards/accuracies": 0.75,
+      "rewards/chosen": -0.19595569372177124,
+      "rewards/margins": 0.10046200454235077,
+      "rewards/rejected": -0.2964176833629608,
+      "step": 425
+    },
+    {
+      "epoch": 0.9378407851690295,
+      "grad_norm": 2.4959827939301458,
+      "learning_rate": 5.676504518718761e-09,
+      "logits/chosen": -24.001766204833984,
+      "logits/rejected": -24.754119873046875,
+      "logps/chosen": -19.787260055541992,
+      "logps/rejected": -29.593700408935547,
+      "loss": 0.6471,
+      "rewards/accuracies": 0.706250011920929,
+      "rewards/chosen": -0.197872593998909,
+      "rewards/margins": 0.09806440025568008,
+      "rewards/rejected": -0.2959369719028473,
+      "step": 430
+    },
+    {
+      "epoch": 0.9487459105779716,
+      "grad_norm": 4.224961358571927,
+      "learning_rate": 3.834930841344119e-09,
+      "logits/chosen": -24.059728622436523,
+      "logits/rejected": -24.974977493286133,
+      "logps/chosen": -21.249866485595703,
+      "logps/rejected": -30.9521427154541,
+      "loss": 0.648,
+      "rewards/accuracies": 0.7562500238418579,
+      "rewards/chosen": -0.21249863505363464,
+      "rewards/margins": 0.09702278673648834,
+      "rewards/rejected": -0.30952146649360657,
+      "step": 435
+    },
+    {
+      "epoch": 0.9596510359869138,
+      "grad_norm": 3.2680876255639357,
+      "learning_rate": 2.3511397018252664e-09,
+      "logits/chosen": -23.782819747924805,
+      "logits/rejected": -24.70381736755371,
+      "logps/chosen": -21.060720443725586,
+      "logps/rejected": -28.60394859313965,
+      "loss": 0.649,
+      "rewards/accuracies": 0.6937500238418579,
+      "rewards/chosen": -0.21060721576213837,
+      "rewards/margins": 0.07543227076530457,
+      "rewards/rejected": -0.28603947162628174,
+      "step": 440
+    },
+    {
+      "epoch": 0.9705561613958561,
+      "grad_norm": 3.1683461698436934,
+      "learning_rate": 1.227287679734429e-09,
+      "logits/chosen": -23.462703704833984,
+      "logits/rejected": -24.59607696533203,
+      "logps/chosen": -18.034442901611328,
+      "logps/rejected": -29.30611801147461,
+      "loss": 0.6478,
+      "rewards/accuracies": 0.7875000238418579,
+      "rewards/chosen": -0.18034443259239197,
+      "rewards/margins": 0.11271677166223526,
+      "rewards/rejected": -0.29306119680404663,
+      "step": 445
+    },
+    {
+      "epoch": 0.9814612868047983,
+      "grad_norm": 3.2800584418942376,
+      "learning_rate": 4.650082100265407e-10,
+      "logits/chosen": -23.944908142089844,
+      "logits/rejected": -24.941503524780273,
+      "logps/chosen": -20.453250885009766,
+      "logps/rejected": -31.82330322265625,
+      "loss": 0.648,
+      "rewards/accuracies": 0.793749988079071,
+      "rewards/chosen": -0.2045324742794037,
+      "rewards/margins": 0.11370055377483368,
+      "rewards/rejected": -0.31823304295539856,
+      "step": 450
+    },
+    {
+      "epoch": 0.9923664122137404,
+      "grad_norm": 3.2265616237555537,
+      "learning_rate": 6.54092089634739e-11,
+      "logits/chosen": -23.501338958740234,
+      "logits/rejected": -24.58260726928711,
+      "logps/chosen": -19.16789436340332,
+      "logps/rejected": -28.90520668029785,
+      "loss": 0.6484,
+      "rewards/accuracies": 0.7437499761581421,
+      "rewards/chosen": -0.1916789561510086,
+      "rewards/margins": 0.0973731279373169,
+      "rewards/rejected": -0.2890520691871643,
+      "step": 455
+    },
+    {
+      "epoch": 0.9989094874591058,
+      "step": 458,
+      "total_flos": 0.0,
+      "train_loss": 0.6719389883191305,
+      "train_runtime": 8958.0836,
+      "train_samples_per_second": 6.551,
+      "train_steps_per_second": 0.051
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 458,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 1000000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}