Instructions to use mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("EleutherAI/pythia-2.8b")
model = PeftModel.from_pretrained(base_model, "mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan")

Transformers

How to use mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan

SGLang

How to use mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan with Docker Model Runner:
```
docker model run hf.co/mia-llm/pythia-2.8b-wikitext2raw-LoRA-ilhan
```

ilhanakbudak commited on Jan 8

Commit

48b042f

verified ·

1 Parent(s): 6e936be

Upload folder using huggingface_hub

Browse files

Files changed (29) hide show

README.md +62 -0
adapter_config.json +43 -0
adapter_model.safetensors +3 -0
checkpoint-1250/README.md +209 -0
checkpoint-1250/adapter_config.json +43 -0
checkpoint-1250/adapter_model.safetensors +3 -0
checkpoint-1250/optimizer.pt +3 -0
checkpoint-1250/rng_state.pth +3 -0
checkpoint-1250/scheduler.pt +3 -0
checkpoint-1250/special_tokens_map.json +24 -0
checkpoint-1250/tokenizer.json +0 -0
checkpoint-1250/tokenizer_config.json +215 -0
checkpoint-1250/trainer_state.json +1416 -0
checkpoint-1250/training_args.bin +3 -0
checkpoint-625/README.md +209 -0
checkpoint-625/adapter_config.json +43 -0
checkpoint-625/adapter_model.safetensors +3 -0
checkpoint-625/optimizer.pt +3 -0
checkpoint-625/rng_state.pth +3 -0
checkpoint-625/scheduler.pt +3 -0
checkpoint-625/special_tokens_map.json +24 -0
checkpoint-625/tokenizer.json +0 -0
checkpoint-625/tokenizer_config.json +215 -0
checkpoint-625/trainer_state.json +720 -0
checkpoint-625/training_args.bin +3 -0
special_tokens_map.json +24 -0
tokenizer.json +0 -0
tokenizer_config.json +215 -0
training_args.bin +3 -0

README.md ADDED Viewed

	@@ -0,0 +1,62 @@

+---
+base_model: EleutherAI/pythia-2.8b
+library_name: peft
+model_name: pythia-2.8b-LoRA-lora_v1.r64.a128.e2.1
+tags:
+- base_model:adapter:EleutherAI/pythia-2.8b
+- lora
+- sft
+- transformers
+- trl
+licence: license
+pipeline_tag: text-generation
+---
+# Model Card for pythia-2.8b-LoRA-lora_v1.r64.a128.e2.1
+This model is a fine-tuned version of [EleutherAI/pythia-2.8b](https://huggingface.co/EleutherAI/pythia-2.8b).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+## Quick start
+```python
+from transformers import pipeline
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+## Training procedure
+This model was trained with SFT.
+### Framework versions
+- PEFT 0.18.0
+- TRL: 0.26.0
+- Transformers: 4.57.3
+- Pytorch: 2.9.1
+- Datasets: 4.4.1
+- Tokenizers: 0.22.1
+## Citations
+Cite TRL as:
+```bibtex
+@misc{vonwerra2022trl,
+	title        = {{TRL: Transformer Reinforcement Learning}},
+	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
+	year         = 2020,
+	journal      = {GitHub repository},
+	publisher    = {GitHub},
+	howpublished = {\url{https://github.com/huggingface/trl}}
+}
+```

adapter_config.json ADDED Viewed

	@@ -0,0 +1,43 @@

+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "EleutherAI/pythia-2.8b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "dense",
+    "query_key_value",
+    "dense_h_to_4h",
+    "dense_4h_to_h"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9847426500490d020eef835d209086baeb9881301f121a4bd52fe768a2b843e2
+size 335580648

checkpoint-1250/README.md ADDED Viewed

	@@ -0,0 +1,209 @@

+---
+base_model: EleutherAI/pythia-2.8b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:EleutherAI/pythia-2.8b
+- lora
+- sft
+- transformers
+- trl
+---
+# Model Card for Model ID
+<!-- Provide a quick summary of what the model is/does. -->
+## Model Details
+### Model Description
+<!-- Provide a longer summary of what this model is. -->
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+### Model Sources [optional]
+<!-- Provide the basic links for the model. -->
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+## Uses
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+### Direct Use
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+[More Information Needed]
+### Downstream Use [optional]
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+[More Information Needed]
+### Out-of-Scope Use
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+[More Information Needed]
+## Bias, Risks, and Limitations
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+[More Information Needed]
+### Recommendations
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+## How to Get Started with the Model
+Use the code below to get started with the model.
+[More Information Needed]
+## Training Details
+### Training Data
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+[More Information Needed]
+### Training Procedure
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+#### Preprocessing [optional]
+[More Information Needed]
+#### Training Hyperparameters
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+#### Speeds, Sizes, Times [optional]
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+[More Information Needed]
+## Evaluation
+<!-- This section describes the evaluation protocols and provides the results. -->
+### Testing Data, Factors & Metrics
+#### Testing Data
+<!-- This should link to a Dataset Card if possible. -->
+[More Information Needed]
+#### Factors
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+[More Information Needed]
+#### Metrics
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+[More Information Needed]
+### Results
+[More Information Needed]
+#### Summary
+## Model Examination [optional]
+<!-- Relevant interpretability work for the model goes here -->
+[More Information Needed]
+## Environmental Impact
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+## Technical Specifications [optional]
+### Model Architecture and Objective
+[More Information Needed]
+### Compute Infrastructure
+[More Information Needed]
+#### Hardware
+[More Information Needed]
+#### Software
+[More Information Needed]
+## Citation [optional]
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+**BibTeX:**
+[More Information Needed]
+**APA:**
+[More Information Needed]
+## Glossary [optional]
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+[More Information Needed]
+## More Information [optional]
+[More Information Needed]
+## Model Card Authors [optional]
+[More Information Needed]
+## Model Card Contact
+[More Information Needed]
+### Framework versions
+- PEFT 0.18.0

checkpoint-1250/adapter_config.json ADDED Viewed

	@@ -0,0 +1,43 @@

+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "EleutherAI/pythia-2.8b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "dense",
+    "query_key_value",
+    "dense_h_to_4h",
+    "dense_4h_to_h"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-1250/adapter_model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9847426500490d020eef835d209086baeb9881301f121a4bd52fe768a2b843e2
+size 335580648

checkpoint-1250/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8395d1857d3da020257ce78eab907a0707adc8b7ae9952436c6d99a1b20965c0
+size 671308491

checkpoint-1250/rng_state.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fb28ac77641ae73a837afa0612828c2d5aa469601213ba2443ab548739301a7e
+size 14645

checkpoint-1250/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:149613c522fb20c719da4556156e70073e5e848f79b5b3e8448635d9c634cbfc
+size 1465

checkpoint-1250/special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,24 @@

+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-1250/tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

checkpoint-1250/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,215 @@

+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50254": {
+      "content": "                        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50255": {
+      "content": "                       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50256": {
+      "content": "                      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50257": {
+      "content": "                     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "                    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "                   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "                  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "                 ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "                ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "               ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "              ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "             ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "            ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "           ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "          ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "         ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-1250/trainer_state.json ADDED Viewed

	@@ -0,0 +1,1416 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 2.0,
+  "eval_steps": 100,
+  "global_step": 1250,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 2.927482557296753,
+      "epoch": 0.016,
+      "grad_norm": 1.3331365585327148,
+      "learning_rate": 4.736842105263158e-05,
+      "loss": 2.9205,
+      "mean_token_accuracy": 0.42432610020041467,
+      "num_tokens": 21466.0,
+      "step": 10
+    },
+    {
+      "entropy": 2.7787948846817017,
+      "epoch": 0.032,
+      "grad_norm": 1.0274778604507446,
+      "learning_rate": 0.0001,
+      "loss": 2.7193,
+      "mean_token_accuracy": 0.4424279421567917,
+      "num_tokens": 44204.0,
+      "step": 20
+    },
+    {
+      "entropy": 2.7391002118587493,
+      "epoch": 0.048,
+      "grad_norm": 0.9975764751434326,
+      "learning_rate": 0.00015263157894736845,
+      "loss": 2.6235,
+      "mean_token_accuracy": 0.46106590032577516,
+      "num_tokens": 63982.0,
+      "step": 30
+    },
+    {
+      "entropy": 2.69979213476181,
+      "epoch": 0.064,
+      "grad_norm": 0.9850704669952393,
+      "learning_rate": 0.00019999966405802826,
+      "loss": 2.5825,
+      "mean_token_accuracy": 0.4666088931262493,
+      "num_tokens": 84531.0,
+      "step": 40
+    },
+    {
+      "entropy": 2.6038752436637878,
+      "epoch": 0.08,
+      "grad_norm": 0.9963482022285461,
+      "learning_rate": 0.00019995935375248606,
+      "loss": 2.5338,
+      "mean_token_accuracy": 0.4776237912476063,
+      "num_tokens": 104496.0,
+      "step": 50
+    },
+    {
+      "entropy": 2.6396988570690154,
+      "epoch": 0.096,
+      "grad_norm": 0.8940117955207825,
+      "learning_rate": 0.000199851886084842,
+      "loss": 2.5631,
+      "mean_token_accuracy": 0.4724681057035923,
+      "num_tokens": 125055.0,
+      "step": 60
+    },
+    {
+      "entropy": 2.624293786287308,
+      "epoch": 0.112,
+      "grad_norm": 0.7860824465751648,
+      "learning_rate": 0.0001996773332568941,
+      "loss": 2.5196,
+      "mean_token_accuracy": 0.47606254518032076,
+      "num_tokens": 145437.0,
+      "step": 70
+    },
+    {
+      "entropy": 2.523421919345856,
+      "epoch": 0.128,
+      "grad_norm": 0.7847617268562317,
+      "learning_rate": 0.0001994358125413841,
+      "loss": 2.438,
+      "mean_token_accuracy": 0.4916156314313412,
+      "num_tokens": 164918.0,
+      "step": 80
+    },
+    {
+      "entropy": 2.5297053515911103,
+      "epoch": 0.144,
+      "grad_norm": 0.8938744068145752,
+      "learning_rate": 0.00019912748620320794,
+      "loss": 2.477,
+      "mean_token_accuracy": 0.4871461518108845,
+      "num_tokens": 186602.0,
+      "step": 90
+    },
+    {
+      "entropy": 2.549694550037384,
+      "epoch": 0.16,
+      "grad_norm": 0.8062200546264648,
+      "learning_rate": 0.00019875256139039902,
+      "loss": 2.4538,
+      "mean_token_accuracy": 0.48964215591549876,
+      "num_tokens": 208334.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.16,
+      "eval_entropy": 2.5797194150777965,
+      "eval_loss": 2.5428755283355713,
+      "eval_mean_token_accuracy": 0.48654096172406125,
+      "eval_num_tokens": 208334.0,
+      "eval_runtime": 0.7687,
+      "eval_samples_per_second": 130.09,
+      "eval_steps_per_second": 16.912,
+      "step": 100
+    },
+    {
+      "entropy": 2.5369359493255614,
+      "epoch": 0.176,
+      "grad_norm": 0.7858350276947021,
+      "learning_rate": 0.00019831128999495606,
+      "loss": 2.451,
+      "mean_token_accuracy": 0.49068020582199096,
+      "num_tokens": 229573.0,
+      "step": 110
+    },
+    {
+      "entropy": 2.622505843639374,
+      "epoch": 0.192,
+      "grad_norm": 0.853503942489624,
+      "learning_rate": 0.0001978039684836106,
+      "loss": 2.513,
+      "mean_token_accuracy": 0.4789179593324661,
+      "num_tokens": 250739.0,
+      "step": 120
+    },
+    {
+      "entropy": 2.538061708211899,
+      "epoch": 0.208,
+      "grad_norm": 0.701422393321991,
+      "learning_rate": 0.00019723093769864663,
+      "loss": 2.4881,
+      "mean_token_accuracy": 0.485482332110405,
+      "num_tokens": 274310.0,
+      "step": 130
+    },
+    {
+      "entropy": 2.633000612258911,
+      "epoch": 0.224,
+      "grad_norm": 0.8620439171791077,
+      "learning_rate": 0.00019659258262890683,
+      "loss": 2.5312,
+      "mean_token_accuracy": 0.4774567261338234,
+      "num_tokens": 295034.0,
+      "step": 140
+    },
+    {
+      "entropy": 2.501206773519516,
+      "epoch": 0.24,
+      "grad_norm": 0.8027399778366089,
+      "learning_rate": 0.00019588933215113926,
+      "loss": 2.4184,
+      "mean_token_accuracy": 0.4926304392516613,
+      "num_tokens": 315487.0,
+      "step": 150
+    },
+    {
+      "entropy": 2.51069019138813,
+      "epoch": 0.256,
+      "grad_norm": 0.7904124855995178,
+      "learning_rate": 0.00019512165874185767,
+      "loss": 2.4492,
+      "mean_token_accuracy": 0.49440020844340327,
+      "num_tokens": 336958.0,
+      "step": 160
+    },
+    {
+      "entropy": 2.5472387611865996,
+      "epoch": 0.272,
+      "grad_norm": 0.8189327120780945,
+      "learning_rate": 0.00019429007815990993,
+      "loss": 2.4559,
+      "mean_token_accuracy": 0.48381901159882545,
+      "num_tokens": 358616.0,
+      "step": 170
+    },
+    {
+      "entropy": 2.547665923833847,
+      "epoch": 0.288,
+      "grad_norm": 0.7772969603538513,
+      "learning_rate": 0.00019339514909996706,
+      "loss": 2.4706,
+      "mean_token_accuracy": 0.4824284970760345,
+      "num_tokens": 379872.0,
+      "step": 180
+    },
+    {
+      "entropy": 2.504627192020416,
+      "epoch": 0.304,
+      "grad_norm": 0.6993403434753418,
+      "learning_rate": 0.000192437472817166,
+      "loss": 2.4443,
+      "mean_token_accuracy": 0.4916023284196854,
+      "num_tokens": 400768.0,
+      "step": 190
+    },
+    {
+      "entropy": 2.5959001183509827,
+      "epoch": 0.32,
+      "grad_norm": 0.8900535106658936,
+      "learning_rate": 0.00019141769272315858,
+      "loss": 2.5256,
+      "mean_token_accuracy": 0.4826003320515156,
+      "num_tokens": 420318.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.32,
+      "eval_entropy": 2.615577771113469,
+      "eval_loss": 2.544278144836426,
+      "eval_mean_token_accuracy": 0.4823318009193127,
+      "eval_num_tokens": 420318.0,
+      "eval_runtime": 0.6723,
+      "eval_samples_per_second": 148.739,
+      "eval_steps_per_second": 19.336,
+      "step": 200
+    },
+    {
+      "entropy": 2.5907178699970244,
+      "epoch": 0.336,
+      "grad_norm": 0.7852405309677124,
+      "learning_rate": 0.00019033649395383702,
+      "loss": 2.5025,
+      "mean_token_accuracy": 0.48569325879216196,
+      "num_tokens": 440493.0,
+      "step": 210
+    },
+    {
+      "entropy": 2.5373390048742293,
+      "epoch": 0.352,
+      "grad_norm": 0.8513323664665222,
+      "learning_rate": 0.00018919460290902826,
+      "loss": 2.4644,
+      "mean_token_accuracy": 0.48737777322530745,
+      "num_tokens": 460347.0,
+      "step": 220
+    },
+    {
+      "entropy": 2.5982253909111024,
+      "epoch": 0.368,
+      "grad_norm": 0.7951892614364624,
+      "learning_rate": 0.00018799278676446423,
+      "loss": 2.5088,
+      "mean_token_accuracy": 0.4845408618450165,
+      "num_tokens": 479686.0,
+      "step": 230
+    },
+    {
+      "entropy": 2.5408327341079713,
+      "epoch": 0.384,
+      "grad_norm": 0.818739652633667,
+      "learning_rate": 0.0001867318529563574,
+      "loss": 2.4453,
+      "mean_token_accuracy": 0.49112383723258973,
+      "num_tokens": 499382.0,
+      "step": 240
+    },
+    {
+      "entropy": 2.566703599691391,
+      "epoch": 0.4,
+      "grad_norm": 0.8289343118667603,
+      "learning_rate": 0.00018541264863892754,
+      "loss": 2.5047,
+      "mean_token_accuracy": 0.48281258195638654,
+      "num_tokens": 520865.0,
+      "step": 250
+    },
+    {
+      "entropy": 2.522556847333908,
+      "epoch": 0.416,
+      "grad_norm": 0.9309546947479248,
+      "learning_rate": 0.000184036060115244,
+      "loss": 2.4434,
+      "mean_token_accuracy": 0.4828334227204323,
+      "num_tokens": 540616.0,
+      "step": 260
+    },
+    {
+      "entropy": 2.5191095739603044,
+      "epoch": 0.432,
+      "grad_norm": 0.7696341276168823,
+      "learning_rate": 0.00018260301224176558,
+      "loss": 2.4318,
+      "mean_token_accuracy": 0.4892562575638294,
+      "num_tokens": 561098.0,
+      "step": 270
+    },
+    {
+      "entropy": 2.5569592356681823,
+      "epoch": 0.448,
+      "grad_norm": 0.7310771346092224,
+      "learning_rate": 0.00018111446780697929,
+      "loss": 2.505,
+      "mean_token_accuracy": 0.476550355553627,
+      "num_tokens": 581527.0,
+      "step": 280
+    },
+    {
+      "entropy": 2.6550574243068694,
+      "epoch": 0.464,
+      "grad_norm": 0.8539873957633972,
+      "learning_rate": 0.00017957142688455362,
+      "loss": 2.5309,
+      "mean_token_accuracy": 0.479371577501297,
+      "num_tokens": 601002.0,
+      "step": 290
+    },
+    {
+      "entropy": 2.5720873057842253,
+      "epoch": 0.48,
+      "grad_norm": 0.7261855006217957,
+      "learning_rate": 0.00017797492616144256,
+      "loss": 2.5289,
+      "mean_token_accuracy": 0.4779372848570347,
+      "num_tokens": 621836.0,
+      "step": 300
+    },
+    {
+      "epoch": 0.48,
+      "eval_entropy": 2.58707464658297,
+      "eval_loss": 2.5298879146575928,
+      "eval_mean_token_accuracy": 0.4877227544784546,
+      "eval_num_tokens": 621836.0,
+      "eval_runtime": 0.6775,
+      "eval_samples_per_second": 147.608,
+      "eval_steps_per_second": 19.189,
+      "step": 300
+    },
+    {
+      "entropy": 2.56911381483078,
+      "epoch": 0.496,
+      "grad_norm": 0.8215358257293701,
+      "learning_rate": 0.00017632603824139085,
+      "loss": 2.4587,
+      "mean_token_accuracy": 0.4831582002341747,
+      "num_tokens": 642910.0,
+      "step": 310
+    },
+    {
+      "entropy": 2.518597739934921,
+      "epoch": 0.512,
+      "grad_norm": 0.7745339274406433,
+      "learning_rate": 0.00017462587092430875,
+      "loss": 2.4552,
+      "mean_token_accuracy": 0.48742511570453645,
+      "num_tokens": 665281.0,
+      "step": 320
+    },
+    {
+      "entropy": 2.5539629220962525,
+      "epoch": 0.528,
+      "grad_norm": 1.3013852834701538,
+      "learning_rate": 0.00017287556646200018,
+      "loss": 2.5049,
+      "mean_token_accuracy": 0.48348471373319624,
+      "num_tokens": 686496.0,
+      "step": 330
+    },
+    {
+      "entropy": 2.549721211194992,
+      "epoch": 0.544,
+      "grad_norm": 0.9040252566337585,
+      "learning_rate": 0.00017107630079074478,
+      "loss": 2.4749,
+      "mean_token_accuracy": 0.49018326848745347,
+      "num_tokens": 708041.0,
+      "step": 340
+    },
+    {
+      "entropy": 2.5657508850097654,
+      "epoch": 0.56,
+      "grad_norm": 0.7486541867256165,
+      "learning_rate": 0.00016922928274124886,
+      "loss": 2.496,
+      "mean_token_accuracy": 0.4882775545120239,
+      "num_tokens": 729373.0,
+      "step": 350
+    },
+    {
+      "entropy": 2.49460369348526,
+      "epoch": 0.576,
+      "grad_norm": 0.7519723176956177,
+      "learning_rate": 0.00016733575322649657,
+      "loss": 2.4358,
+      "mean_token_accuracy": 0.4897538281977177,
+      "num_tokens": 751428.0,
+      "step": 360
+    },
+    {
+      "entropy": 2.485343414545059,
+      "epoch": 0.592,
+      "grad_norm": 0.7465973496437073,
+      "learning_rate": 0.00016539698440804661,
+      "loss": 2.4446,
+      "mean_token_accuracy": 0.4956702694296837,
+      "num_tokens": 773898.0,
+      "step": 370
+    },
+    {
+      "entropy": 2.5664760768413544,
+      "epoch": 0.608,
+      "grad_norm": 0.792957603931427,
+      "learning_rate": 0.0001634142788413346,
+      "loss": 2.4411,
+      "mean_token_accuracy": 0.49218829050660134,
+      "num_tokens": 793921.0,
+      "step": 380
+    },
+    {
+      "entropy": 2.4981964468955993,
+      "epoch": 0.624,
+      "grad_norm": 0.9188303351402283,
+      "learning_rate": 0.00016138896860055555,
+      "loss": 2.4527,
+      "mean_token_accuracy": 0.4891628086566925,
+      "num_tokens": 814549.0,
+      "step": 390
+    },
+    {
+      "entropy": 2.5058595299720765,
+      "epoch": 0.64,
+      "grad_norm": 0.8261463642120361,
+      "learning_rate": 0.0001593224143837142,
+      "loss": 2.3825,
+      "mean_token_accuracy": 0.49732586443424226,
+      "num_tokens": 835738.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.64,
+      "eval_entropy": 2.5427963917072,
+      "eval_loss": 2.528325080871582,
+      "eval_mean_token_accuracy": 0.48722161238010114,
+      "eval_num_tokens": 835738.0,
+      "eval_runtime": 0.6794,
+      "eval_samples_per_second": 147.184,
+      "eval_steps_per_second": 19.134,
+      "step": 400
+    },
+    {
+      "entropy": 2.549647295475006,
+      "epoch": 0.656,
+      "grad_norm": 0.9669468402862549,
+      "learning_rate": 0.00015721600459844468,
+      "loss": 2.4992,
+      "mean_token_accuracy": 0.4811323322355747,
+      "num_tokens": 856308.0,
+      "step": 410
+    },
+    {
+      "entropy": 2.50358769595623,
+      "epoch": 0.672,
+      "grad_norm": 0.7736469507217407,
+      "learning_rate": 0.0001550711544292131,
+      "loss": 2.4233,
+      "mean_token_accuracy": 0.4960208125412464,
+      "num_tokens": 877984.0,
+      "step": 420
+    },
+    {
+      "entropy": 2.513987493515015,
+      "epoch": 0.688,
+      "grad_norm": 1.0335294008255005,
+      "learning_rate": 0.00015288930488653094,
+      "loss": 2.4349,
+      "mean_token_accuracy": 0.490431859344244,
+      "num_tokens": 899593.0,
+      "step": 430
+    },
+    {
+      "entropy": 2.5122825741767882,
+      "epoch": 0.704,
+      "grad_norm": 1.0326985120773315,
+      "learning_rate": 0.00015067192183881658,
+      "loss": 2.4435,
+      "mean_token_accuracy": 0.49002100676298144,
+      "num_tokens": 919549.0,
+      "step": 440
+    },
+    {
+      "entropy": 2.5058334708213805,
+      "epoch": 0.72,
+      "grad_norm": 0.8282021284103394,
+      "learning_rate": 0.0001484204950275565,
+      "loss": 2.3755,
+      "mean_token_accuracy": 0.49881314411759375,
+      "num_tokens": 938371.0,
+      "step": 450
+    },
+    {
+      "entropy": 2.5388923943042756,
+      "epoch": 0.736,
+      "grad_norm": 0.8473682403564453,
+      "learning_rate": 0.0001461365370664276,
+      "loss": 2.4695,
+      "mean_token_accuracy": 0.48638317435979844,
+      "num_tokens": 959081.0,
+      "step": 460
+    },
+    {
+      "entropy": 2.4723076283931733,
+      "epoch": 0.752,
+      "grad_norm": 0.964932918548584,
+      "learning_rate": 0.00014382158242505234,
+      "loss": 2.4151,
+      "mean_token_accuracy": 0.4942440494894981,
+      "num_tokens": 980709.0,
+      "step": 470
+    },
+    {
+      "entropy": 2.4699651658535005,
+      "epoch": 0.768,
+      "grad_norm": 0.8604968190193176,
+      "learning_rate": 0.0001414771863980707,
+      "loss": 2.4249,
+      "mean_token_accuracy": 0.49896738603711127,
+      "num_tokens": 998177.0,
+      "step": 480
+    },
+    {
+      "entropy": 2.5598580718040465,
+      "epoch": 0.784,
+      "grad_norm": 0.8044160604476929,
+      "learning_rate": 0.00013910492406022033,
+      "loss": 2.5119,
+      "mean_token_accuracy": 0.48558894991874696,
+      "num_tokens": 1019231.0,
+      "step": 490
+    },
+    {
+      "entropy": 2.5347262740135195,
+      "epoch": 0.8,
+      "grad_norm": 0.989012598991394,
+      "learning_rate": 0.000136706389208128,
+      "loss": 2.457,
+      "mean_token_accuracy": 0.48754729256033896,
+      "num_tokens": 1039963.0,
+      "step": 500
+    },
+    {
+      "epoch": 0.8,
+      "eval_entropy": 2.5207965007195106,
+      "eval_loss": 2.532066822052002,
+      "eval_mean_token_accuracy": 0.4870225145266606,
+      "eval_num_tokens": 1039963.0,
+      "eval_runtime": 0.6776,
+      "eval_samples_per_second": 147.573,
+      "eval_steps_per_second": 19.185,
+      "step": 500
+    },
+    {
+      "entropy": 2.4576663255691527,
+      "epoch": 0.816,
+      "grad_norm": 0.8774695992469788,
+      "learning_rate": 0.00013428319328952253,
+      "loss": 2.3836,
+      "mean_token_accuracy": 0.4995129629969597,
+      "num_tokens": 1061370.0,
+      "step": 510
+    },
+    {
+      "entropy": 2.5250146806240084,
+      "epoch": 0.832,
+      "grad_norm": 0.7958005666732788,
+      "learning_rate": 0.00013183696432058888,
+      "loss": 2.4315,
+      "mean_token_accuracy": 0.4982760787010193,
+      "num_tokens": 1081842.0,
+      "step": 520
+    },
+    {
+      "entropy": 2.4409106403589247,
+      "epoch": 0.848,
+      "grad_norm": 0.7063387036323547,
+      "learning_rate": 0.00012936934579219094,
+      "loss": 2.3816,
+      "mean_token_accuracy": 0.5006642855703831,
+      "num_tokens": 1103737.0,
+      "step": 530
+    },
+    {
+      "entropy": 2.503674066066742,
+      "epoch": 0.864,
+      "grad_norm": 0.712315559387207,
+      "learning_rate": 0.00012688199556569753,
+      "loss": 2.4427,
+      "mean_token_accuracy": 0.4878311850130558,
+      "num_tokens": 1124676.0,
+      "step": 540
+    },
+    {
+      "entropy": 2.4937715351581575,
+      "epoch": 0.88,
+      "grad_norm": 0.7796215415000916,
+      "learning_rate": 0.00012437658475915377,
+      "loss": 2.4148,
+      "mean_token_accuracy": 0.4976658411324024,
+      "num_tokens": 1147148.0,
+      "step": 550
+    },
+    {
+      "entropy": 2.458594244718552,
+      "epoch": 0.896,
+      "grad_norm": 0.8076977133750916,
+      "learning_rate": 0.00012185479662454595,
+      "loss": 2.414,
+      "mean_token_accuracy": 0.4989797443151474,
+      "num_tokens": 1167948.0,
+      "step": 560
+    },
+    {
+      "entropy": 2.5129783391952514,
+      "epoch": 0.912,
+      "grad_norm": 0.9025003910064697,
+      "learning_rate": 0.00011931832541691418,
+      "loss": 2.4198,
+      "mean_token_accuracy": 0.4928160272538662,
+      "num_tokens": 1187896.0,
+      "step": 570
+    },
+    {
+      "entropy": 2.429825708270073,
+      "epoch": 0.928,
+      "grad_norm": 0.9303080439567566,
+      "learning_rate": 0.00011676887525607271,
+      "loss": 2.336,
+      "mean_token_accuracy": 0.507265867292881,
+      "num_tokens": 1207963.0,
+      "step": 580
+    },
+    {
+      "entropy": 2.483963069319725,
+      "epoch": 0.944,
+      "grad_norm": 0.9662640690803528,
+      "learning_rate": 0.0001142081589817027,
+      "loss": 2.4441,
+      "mean_token_accuracy": 0.4951579734683037,
+      "num_tokens": 1228312.0,
+      "step": 590
+    },
+    {
+      "entropy": 2.5128854513168335,
+      "epoch": 0.96,
+      "grad_norm": 0.7802348732948303,
+      "learning_rate": 0.00011163789700258655,
+      "loss": 2.481,
+      "mean_token_accuracy": 0.4941797606647015,
+      "num_tokens": 1250029.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.96,
+      "eval_entropy": 2.5629419730259824,
+      "eval_loss": 2.526008367538452,
+      "eval_mean_token_accuracy": 0.4862596323856941,
+      "eval_num_tokens": 1250029.0,
+      "eval_runtime": 0.6764,
+      "eval_samples_per_second": 147.833,
+      "eval_steps_per_second": 19.218,
+      "step": 600
+    },
+    {
+      "entropy": 2.45339612364769,
+      "epoch": 0.976,
+      "grad_norm": 0.7918068766593933,
+      "learning_rate": 0.00010905981614075693,
+      "loss": 2.3798,
+      "mean_token_accuracy": 0.5059582263231277,
+      "num_tokens": 1273572.0,
+      "step": 610
+    },
+    {
+      "entropy": 2.5436563432216643,
+      "epoch": 0.992,
+      "grad_norm": 0.9120656847953796,
+      "learning_rate": 0.000106475648471337,
+      "loss": 2.4771,
+      "mean_token_accuracy": 0.49086978659033775,
+      "num_tokens": 1295353.0,
+      "step": 620
+    },
+    {
+      "entropy": 2.410617971420288,
+      "epoch": 1.008,
+      "grad_norm": 0.7449939250946045,
+      "learning_rate": 0.00010388713015885161,
+      "loss": 2.2071,
+      "mean_token_accuracy": 0.5270172961056232,
+      "num_tokens": 1317590.0,
+      "step": 630
+    },
+    {
+      "entropy": 2.223151522874832,
+      "epoch": 1.024,
+      "grad_norm": 0.8070065975189209,
+      "learning_rate": 0.00010129600029079072,
+      "loss": 2.0441,
+      "mean_token_accuracy": 0.5562543638050557,
+      "num_tokens": 1338182.0,
+      "step": 640
+    },
+    {
+      "entropy": 2.1513119250535966,
+      "epoch": 1.04,
+      "grad_norm": 0.7470272779464722,
+      "learning_rate": 9.870399970920932e-05,
+      "loss": 2.0792,
+      "mean_token_accuracy": 0.54966342151165,
+      "num_tokens": 1358822.0,
+      "step": 650
+    },
+    {
+      "entropy": 2.2236103057861327,
+      "epoch": 1.056,
+      "grad_norm": 0.8093722462654114,
+      "learning_rate": 9.611286984114841e-05,
+      "loss": 2.0749,
+      "mean_token_accuracy": 0.5545817121863366,
+      "num_tokens": 1379005.0,
+      "step": 660
+    },
+    {
+      "entropy": 2.2356995731592177,
+      "epoch": 1.072,
+      "grad_norm": 0.8296709656715393,
+      "learning_rate": 9.352435152866298e-05,
+      "loss": 2.0653,
+      "mean_token_accuracy": 0.5554739162325859,
+      "num_tokens": 1401238.0,
+      "step": 670
+    },
+    {
+      "entropy": 2.240691155195236,
+      "epoch": 1.088,
+      "grad_norm": 0.864894688129425,
+      "learning_rate": 9.09401838592431e-05,
+      "loss": 2.1055,
+      "mean_token_accuracy": 0.5520414739847184,
+      "num_tokens": 1422418.0,
+      "step": 680
+    },
+    {
+      "entropy": 2.2082038104534147,
+      "epoch": 1.104,
+      "grad_norm": 1.003994345664978,
+      "learning_rate": 8.836210299741346e-05,
+      "loss": 2.0112,
+      "mean_token_accuracy": 0.5623031251132489,
+      "num_tokens": 1442486.0,
+      "step": 690
+    },
+    {
+      "entropy": 2.1362645506858824,
+      "epoch": 1.12,
+      "grad_norm": 0.9215587973594666,
+      "learning_rate": 8.579184101829734e-05,
+      "loss": 2.0474,
+      "mean_token_accuracy": 0.5591939069330693,
+      "num_tokens": 1465340.0,
+      "step": 700
+    },
+    {
+      "epoch": 1.12,
+      "eval_entropy": 2.2982528393085184,
+      "eval_loss": 2.5718441009521484,
+      "eval_mean_token_accuracy": 0.4881514035738431,
+      "eval_num_tokens": 1465340.0,
+      "eval_runtime": 0.6785,
+      "eval_samples_per_second": 147.375,
+      "eval_steps_per_second": 19.159,
+      "step": 700
+    },
+    {
+      "entropy": 2.177516022324562,
+      "epoch": 1.1360000000000001,
+      "grad_norm": 0.8390331268310547,
+      "learning_rate": 8.323112474392731e-05,
+      "loss": 2.0602,
+      "mean_token_accuracy": 0.5524092435836792,
+      "num_tokens": 1484560.0,
+      "step": 710
+    },
+    {
+      "entropy": 2.2126996397972105,
+      "epoch": 1.152,
+      "grad_norm": 0.7812036275863647,
+      "learning_rate": 8.068167458308582e-05,
+      "loss": 2.08,
+      "mean_token_accuracy": 0.5531082898378372,
+      "num_tokens": 1506886.0,
+      "step": 720
+    },
+    {
+      "entropy": 2.172422558069229,
+      "epoch": 1.168,
+      "grad_norm": 0.9416456818580627,
+      "learning_rate": 7.814520337545406e-05,
+      "loss": 2.0252,
+      "mean_token_accuracy": 0.5575959712266922,
+      "num_tokens": 1529928.0,
+      "step": 730
+    },
+    {
+      "entropy": 2.161852565407753,
+      "epoch": 1.184,
+      "grad_norm": 1.1978275775909424,
+      "learning_rate": 7.562341524084623e-05,
+      "loss": 2.0087,
+      "mean_token_accuracy": 0.5669111452996731,
+      "num_tokens": 1547619.0,
+      "step": 740
+    },
+    {
+      "entropy": 2.0998224556446075,
+      "epoch": 1.2,
+      "grad_norm": 0.9615102410316467,
+      "learning_rate": 7.311800443430251e-05,
+      "loss": 1.9751,
+      "mean_token_accuracy": 0.5734243541955948,
+      "num_tokens": 1569169.0,
+      "step": 750
+    },
+    {
+      "entropy": 2.205637964606285,
+      "epoch": 1.216,
+      "grad_norm": 0.838168740272522,
+      "learning_rate": 7.06306542078091e-05,
+      "loss": 2.0775,
+      "mean_token_accuracy": 0.5475761353969574,
+      "num_tokens": 1591379.0,
+      "step": 760
+    },
+    {
+      "entropy": 2.220550259947777,
+      "epoch": 1.232,
+      "grad_norm": 1.063684344291687,
+      "learning_rate": 6.816303567941112e-05,
+      "loss": 2.0495,
+      "mean_token_accuracy": 0.558470205962658,
+      "num_tokens": 1611202.0,
+      "step": 770
+    },
+    {
+      "entropy": 2.1949700146913527,
+      "epoch": 1.248,
+      "grad_norm": 0.814487099647522,
+      "learning_rate": 6.571680671047749e-05,
+      "loss": 2.059,
+      "mean_token_accuracy": 0.5488512426614761,
+      "num_tokens": 1633430.0,
+      "step": 780
+    },
+    {
+      "entropy": 2.1767837584018705,
+      "epoch": 1.264,
+      "grad_norm": 0.8867488503456116,
+      "learning_rate": 6.329361079187199e-05,
+      "loss": 2.0492,
+      "mean_token_accuracy": 0.5580153577029705,
+      "num_tokens": 1654807.0,
+      "step": 790
+    },
+    {
+      "entropy": 2.15340778529644,
+      "epoch": 1.28,
+      "grad_norm": 0.8776085376739502,
+      "learning_rate": 6.08950759397797e-05,
+      "loss": 2.0017,
+      "mean_token_accuracy": 0.5655137121677398,
+      "num_tokens": 1676241.0,
+      "step": 800
+    },
+    {
+      "epoch": 1.28,
+      "eval_entropy": 2.318764512355511,
+      "eval_loss": 2.570997953414917,
+      "eval_mean_token_accuracy": 0.4826100170612335,
+      "eval_num_tokens": 1676241.0,
+      "eval_runtime": 0.6801,
+      "eval_samples_per_second": 147.041,
+      "eval_steps_per_second": 19.115,
+      "step": 800
+    },
+    {
+      "entropy": 2.224538028240204,
+      "epoch": 1.296,
+      "grad_norm": 0.9554702043533325,
+      "learning_rate": 5.8522813601929324e-05,
+      "loss": 2.0907,
+      "mean_token_accuracy": 0.5463516399264335,
+      "num_tokens": 1696836.0,
+      "step": 810
+    },
+    {
+      "entropy": 2.1161206275224687,
+      "epoch": 1.312,
+      "grad_norm": 0.9823859333992004,
+      "learning_rate": 5.617841757494762e-05,
+      "loss": 1.954,
+      "mean_token_accuracy": 0.5701966688036919,
+      "num_tokens": 1717750.0,
+      "step": 820
+    },
+    {
+      "entropy": 2.187773588299751,
+      "epoch": 1.328,
+      "grad_norm": 0.9549944996833801,
+      "learning_rate": 5.386346293357242e-05,
+      "loss": 2.0556,
+      "mean_token_accuracy": 0.5585292562842369,
+      "num_tokens": 1738383.0,
+      "step": 830
+    },
+    {
+      "entropy": 2.142790347337723,
+      "epoch": 1.3439999999999999,
+      "grad_norm": 1.1180299520492554,
+      "learning_rate": 5.15795049724435e-05,
+      "loss": 1.9954,
+      "mean_token_accuracy": 0.561890059709549,
+      "num_tokens": 1757450.0,
+      "step": 840
+    },
+    {
+      "entropy": 2.137694215774536,
+      "epoch": 1.3599999999999999,
+      "grad_norm": 0.8447200059890747,
+      "learning_rate": 4.9328078161183464e-05,
+      "loss": 1.9993,
+      "mean_token_accuracy": 0.5649342782795429,
+      "num_tokens": 1777797.0,
+      "step": 850
+    },
+    {
+      "entropy": 2.1595336198806763,
+      "epoch": 1.376,
+      "grad_norm": 1.0689432621002197,
+      "learning_rate": 4.7110695113469085e-05,
+      "loss": 2.0498,
+      "mean_token_accuracy": 0.5661192789673806,
+      "num_tokens": 1799140.0,
+      "step": 860
+    },
+    {
+      "entropy": 2.1561016231775283,
+      "epoch": 1.392,
+      "grad_norm": 0.8344489336013794,
+      "learning_rate": 4.492884557078688e-05,
+      "loss": 2.0489,
+      "mean_token_accuracy": 0.5576602071523666,
+      "num_tokens": 1820412.0,
+      "step": 870
+    },
+    {
+      "entropy": 2.1310487359762194,
+      "epoch": 1.408,
+      "grad_norm": 0.8696361184120178,
+      "learning_rate": 4.278399540155536e-05,
+      "loss": 1.9619,
+      "mean_token_accuracy": 0.5695676565170288,
+      "num_tokens": 1841046.0,
+      "step": 880
+    },
+    {
+      "entropy": 2.093841528892517,
+      "epoch": 1.424,
+      "grad_norm": 0.9314078688621521,
+      "learning_rate": 4.0677585616285774e-05,
+      "loss": 1.9841,
+      "mean_token_accuracy": 0.5746666699647903,
+      "num_tokens": 1861480.0,
+      "step": 890
+    },
+    {
+      "entropy": 2.175295126438141,
+      "epoch": 1.44,
+      "grad_norm": 0.7976773977279663,
+      "learning_rate": 3.861103139944449e-05,
+      "loss": 2.0547,
+      "mean_token_accuracy": 0.5568442553281784,
+      "num_tokens": 1886031.0,
+      "step": 900
+    },
+    {
+      "epoch": 1.44,
+      "eval_entropy": 2.285807416989253,
+      "eval_loss": 2.586611032485962,
+      "eval_mean_token_accuracy": 0.48370331755051243,
+      "eval_num_tokens": 1886031.0,
+      "eval_runtime": 0.6775,
+      "eval_samples_per_second": 147.605,
+      "eval_steps_per_second": 19.189,
+      "step": 900
+    },
+    {
+      "entropy": 2.1779676496982576,
+      "epoch": 1.456,
+      "grad_norm": 1.0191247463226318,
+      "learning_rate": 3.658572115866541e-05,
+      "loss": 2.0442,
+      "mean_token_accuracy": 0.5562367714941502,
+      "num_tokens": 1905543.0,
+      "step": 910
+    },
+    {
+      "entropy": 2.1235474199056625,
+      "epoch": 1.472,
+      "grad_norm": 0.811194896697998,
+      "learning_rate": 3.4603015591953395e-05,
+      "loss": 2.0,
+      "mean_token_accuracy": 0.5647696286439896,
+      "num_tokens": 1928338.0,
+      "step": 920
+    },
+    {
+      "entropy": 2.2050245642662047,
+      "epoch": 1.488,
+      "grad_norm": 1.0259207487106323,
+      "learning_rate": 3.266424677350346e-05,
+      "loss": 2.076,
+      "mean_token_accuracy": 0.5486814387142658,
+      "num_tokens": 1948809.0,
+      "step": 930
+    },
+    {
+      "entropy": 2.13138128221035,
+      "epoch": 1.504,
+      "grad_norm": 0.9353992342948914,
+      "learning_rate": 3.077071725875116e-05,
+      "loss": 1.9515,
+      "mean_token_accuracy": 0.571763351559639,
+      "num_tokens": 1969617.0,
+      "step": 940
+    },
+    {
+      "entropy": 2.1393859505653383,
+      "epoch": 1.52,
+      "grad_norm": 0.8784123659133911,
+      "learning_rate": 2.8923699209255284e-05,
+      "loss": 1.997,
+      "mean_token_accuracy": 0.5623179793357849,
+      "num_tokens": 1991588.0,
+      "step": 950
+    },
+    {
+      "entropy": 2.1316426753997804,
+      "epoch": 1.536,
+      "grad_norm": 0.9279586672782898,
+      "learning_rate": 2.712443353799984e-05,
+      "loss": 1.9962,
+      "mean_token_accuracy": 0.5668175779283047,
+      "num_tokens": 2013970.0,
+      "step": 960
+    },
+    {
+      "entropy": 2.1412221670150755,
+      "epoch": 1.552,
+      "grad_norm": 1.0614557266235352,
+      "learning_rate": 2.5374129075691265e-05,
+      "loss": 1.981,
+      "mean_token_accuracy": 0.5675079435110092,
+      "num_tokens": 2033870.0,
+      "step": 970
+    },
+    {
+      "entropy": 2.1519955545663834,
+      "epoch": 1.568,
+      "grad_norm": 1.0409959554672241,
+      "learning_rate": 2.3673961758609152e-05,
+      "loss": 2.0254,
+      "mean_token_accuracy": 0.5568312801420688,
+      "num_tokens": 2053742.0,
+      "step": 980
+    },
+    {
+      "entropy": 2.13378230035305,
+      "epoch": 1.584,
+      "grad_norm": 1.051721215248108,
+      "learning_rate": 2.2025073838557454e-05,
+      "loss": 1.9988,
+      "mean_token_accuracy": 0.5596944905817509,
+      "num_tokens": 2073771.0,
+      "step": 990
+    },
+    {
+      "entropy": 2.139891889691353,
+      "epoch": 1.6,
+      "grad_norm": 0.9190446138381958,
+      "learning_rate": 2.0428573115446392e-05,
+      "loss": 2.0098,
+      "mean_token_accuracy": 0.5630866177380085,
+      "num_tokens": 2093165.0,
+      "step": 1000
+    },
+    {
+      "epoch": 1.6,
+      "eval_entropy": 2.29392801798307,
+      "eval_loss": 2.5830881595611572,
+      "eval_mean_token_accuracy": 0.48317546798632693,
+      "eval_num_tokens": 2093165.0,
+      "eval_runtime": 0.6748,
+      "eval_samples_per_second": 148.201,
+      "eval_steps_per_second": 19.266,
+      "step": 1000
+    },
+    {
+      "entropy": 2.152450978755951,
+      "epoch": 1.616,
+      "grad_norm": 0.9870165586471558,
+      "learning_rate": 1.8885532193020704e-05,
+      "loss": 2.0398,
+      "mean_token_accuracy": 0.5565521039068699,
+      "num_tokens": 2113836.0,
+      "step": 1010
+    },
+    {
+      "entropy": 2.1866767942905425,
+      "epoch": 1.6320000000000001,
+      "grad_norm": 0.9748448133468628,
+      "learning_rate": 1.739698775823442e-05,
+      "loss": 2.0618,
+      "mean_token_accuracy": 0.55533082857728,
+      "num_tokens": 2135597.0,
+      "step": 1020
+    },
+    {
+      "entropy": 2.1291246324777604,
+      "epoch": 1.6480000000000001,
+      "grad_norm": 0.9417283535003662,
+      "learning_rate": 1.5963939884756042e-05,
+      "loss": 2.0084,
+      "mean_token_accuracy": 0.5552340872585774,
+      "num_tokens": 2155871.0,
+      "step": 1030
+    },
+    {
+      "entropy": 2.1946566879749296,
+      "epoch": 1.6640000000000001,
+      "grad_norm": 0.8462656736373901,
+      "learning_rate": 1.4587351361072454e-05,
+      "loss": 2.0577,
+      "mean_token_accuracy": 0.5526181682944298,
+      "num_tokens": 2177686.0,
+      "step": 1040
+    },
+    {
+      "entropy": 2.1540873527526854,
+      "epoch": 1.6800000000000002,
+      "grad_norm": 0.8386402726173401,
+      "learning_rate": 1.326814704364262e-05,
+      "loss": 1.9869,
+      "mean_token_accuracy": 0.5597876310348511,
+      "num_tokens": 2197554.0,
+      "step": 1050
+    },
+    {
+      "entropy": 2.158621978759766,
+      "epoch": 1.696,
+      "grad_norm": 0.9277564287185669,
+      "learning_rate": 1.2007213235535786e-05,
+      "loss": 2.0627,
+      "mean_token_accuracy": 0.5556139029562474,
+      "num_tokens": 2216537.0,
+      "step": 1060
+    },
+    {
+      "entropy": 2.174393981695175,
+      "epoch": 1.712,
+      "grad_norm": 1.1009163856506348,
+      "learning_rate": 1.0805397090971737e-05,
+      "loss": 2.0276,
+      "mean_token_accuracy": 0.562708406150341,
+      "num_tokens": 2236200.0,
+      "step": 1070
+    },
+    {
+      "entropy": 2.1389470905065537,
+      "epoch": 1.728,
+      "grad_norm": 0.989471435546875,
+      "learning_rate": 9.663506046162985e-06,
+      "loss": 2.0102,
+      "mean_token_accuracy": 0.5642252482473851,
+      "num_tokens": 2258460.0,
+      "step": 1080
+    },
+    {
+      "entropy": 2.1603562027215957,
+      "epoch": 1.744,
+      "grad_norm": 1.0297486782073975,
+      "learning_rate": 8.582307276841462e-06,
+      "loss": 2.0182,
+      "mean_token_accuracy": 0.5630522392690182,
+      "num_tokens": 2278818.0,
+      "step": 1090
+    },
+    {
+      "entropy": 2.1662287533283235,
+      "epoch": 1.76,
+      "grad_norm": 0.8877721428871155,
+      "learning_rate": 7.562527182833978e-06,
+      "loss": 2.0168,
+      "mean_token_accuracy": 0.563526850938797,
+      "num_tokens": 2299692.0,
+      "step": 1100
+    },
+    {
+      "epoch": 1.76,
+      "eval_entropy": 2.3022796649199266,
+      "eval_loss": 2.5798184871673584,
+      "eval_mean_token_accuracy": 0.4844009096805866,
+      "eval_num_tokens": 2299692.0,
+      "eval_runtime": 0.6747,
+      "eval_samples_per_second": 148.214,
+      "eval_steps_per_second": 19.268,
+      "step": 1100
+    },
+    {
+      "entropy": 2.186119019985199,
+      "epoch": 1.776,
+      "grad_norm": 1.1785519123077393,
+      "learning_rate": 6.604850900032955e-06,
+      "loss": 2.0263,
+      "mean_token_accuracy": 0.5561132155358791,
+      "num_tokens": 2320606.0,
+      "step": 1110
+    },
+    {
+      "entropy": 2.1382477432489395,
+      "epoch": 1.792,
+      "grad_norm": 0.9997260570526123,
+      "learning_rate": 5.7099218400900716e-06,
+      "loss": 2.0155,
+      "mean_token_accuracy": 0.5585696510970592,
+      "num_tokens": 2341589.0,
+      "step": 1120
+    },
+    {
+      "entropy": 2.1318988502025604,
+      "epoch": 1.808,
+      "grad_norm": 1.0082330703735352,
+      "learning_rate": 4.87834125814235e-06,
+      "loss": 2.013,
+      "mean_token_accuracy": 0.562056128680706,
+      "num_tokens": 2364065.0,
+      "step": 1130
+    },
+    {
+      "entropy": 2.1655119955539703,
+      "epoch": 1.8239999999999998,
+      "grad_norm": 0.8680001497268677,
+      "learning_rate": 4.1106678488607495e-06,
+      "loss": 2.0232,
+      "mean_token_accuracy": 0.5640067130327224,
+      "num_tokens": 2385547.0,
+      "step": 1140
+    },
+    {
+      "entropy": 2.133523789048195,
+      "epoch": 1.8399999999999999,
+      "grad_norm": 0.8609693050384521,
+      "learning_rate": 3.40741737109318e-06,
+      "loss": 1.971,
+      "mean_token_accuracy": 0.5636379756033421,
+      "num_tokens": 2406399.0,
+      "step": 1150
+    },
+    {
+      "entropy": 2.140376663208008,
+      "epoch": 1.8559999999999999,
+      "grad_norm": 0.917898952960968,
+      "learning_rate": 2.7690623013533976e-06,
+      "loss": 2.0089,
+      "mean_token_accuracy": 0.5577343553304672,
+      "num_tokens": 2425661.0,
+      "step": 1160
+    },
+    {
+      "entropy": 2.1588161110877992,
+      "epoch": 1.8719999999999999,
+      "grad_norm": 0.9883831739425659,
+      "learning_rate": 2.1960315163894075e-06,
+      "loss": 2.0176,
+      "mean_token_accuracy": 0.5622549809515476,
+      "num_tokens": 2445659.0,
+      "step": 1170
+    },
+    {
+      "entropy": 2.2101395189762116,
+      "epoch": 1.888,
+      "grad_norm": 0.7731896042823792,
+      "learning_rate": 1.6887100050439587e-06,
+      "loss": 2.0781,
+      "mean_token_accuracy": 0.5513335958123207,
+      "num_tokens": 2467209.0,
+      "step": 1180
+    },
+    {
+      "entropy": 2.124671292304993,
+      "epoch": 1.904,
+      "grad_norm": 0.8258634805679321,
+      "learning_rate": 1.2474386096010039e-06,
+      "loss": 1.986,
+      "mean_token_accuracy": 0.5669601626694203,
+      "num_tokens": 2489020.0,
+      "step": 1190
+    },
+    {
+      "entropy": 2.1453059643507,
+      "epoch": 1.92,
+      "grad_norm": 0.9933258295059204,
+      "learning_rate": 8.725137967920738e-07,
+      "loss": 2.0125,
+      "mean_token_accuracy": 0.5601302161812782,
+      "num_tokens": 2508813.0,
+      "step": 1200
+    },
+    {
+      "epoch": 1.92,
+      "eval_entropy": 2.2944095684931827,
+      "eval_loss": 2.5806496143341064,
+      "eval_mean_token_accuracy": 0.48303504632069516,
+      "eval_num_tokens": 2508813.0,
+      "eval_runtime": 0.6714,
+      "eval_samples_per_second": 148.939,
+      "eval_steps_per_second": 19.362,
+      "step": 1200
+    },
+    {
+      "entropy": 2.151768389344215,
+      "epoch": 1.936,
+      "grad_norm": 0.8982884287834167,
+      "learning_rate": 5.64187458615939e-07,
+      "loss": 1.9924,
+      "mean_token_accuracy": 0.5615834034979343,
+      "num_tokens": 2528858.0,
+      "step": 1210
+    },
+    {
+      "entropy": 2.140746533870697,
+      "epoch": 1.952,
+      "grad_norm": 1.1360710859298706,
+      "learning_rate": 3.2266674310589273e-07,
+      "loss": 1.9953,
+      "mean_token_accuracy": 0.5639697797596455,
+      "num_tokens": 2548500.0,
+      "step": 1220
+    },
+    {
+      "entropy": 2.2176344990730286,
+      "epoch": 1.968,
+      "grad_norm": 1.0805954933166504,
+      "learning_rate": 1.481139151579991e-07,
+      "loss": 2.0792,
+      "mean_token_accuracy": 0.5535091876983642,
+      "num_tokens": 2568981.0,
+      "step": 1230
+    },
+    {
+      "entropy": 2.1372744172811506,
+      "epoch": 1.984,
+      "grad_norm": 0.936008870601654,
+      "learning_rate": 4.064624751394242e-08,
+      "loss": 2.0289,
+      "mean_token_accuracy": 0.5637350581586361,
+      "num_tokens": 2591580.0,
+      "step": 1240
+    },
+    {
+      "entropy": 2.156083407998085,
+      "epoch": 2.0,
+      "grad_norm": 0.864385724067688,
+      "learning_rate": 3.3594197175190745e-10,
+      "loss": 2.0083,
+      "mean_token_accuracy": 0.5578774020075798,
+      "num_tokens": 2612778.0,
+      "step": 1250
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1250,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 7.51189440079872e+16,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1250/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:879f340dca9a81b05ff3f767198bc8a294c35f0efe288b9f690178d342307834
+size 6353

checkpoint-625/README.md ADDED Viewed

	@@ -0,0 +1,209 @@

+---
+base_model: EleutherAI/pythia-2.8b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:EleutherAI/pythia-2.8b
+- lora
+- sft
+- transformers
+- trl
+---
+# Model Card for Model ID
+<!-- Provide a quick summary of what the model is/does. -->
+## Model Details
+### Model Description
+<!-- Provide a longer summary of what this model is. -->
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+### Model Sources [optional]
+<!-- Provide the basic links for the model. -->
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+## Uses
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+### Direct Use
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+[More Information Needed]
+### Downstream Use [optional]
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+[More Information Needed]
+### Out-of-Scope Use
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+[More Information Needed]
+## Bias, Risks, and Limitations
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+[More Information Needed]
+### Recommendations
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+## How to Get Started with the Model
+Use the code below to get started with the model.
+[More Information Needed]
+## Training Details
+### Training Data
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+[More Information Needed]
+### Training Procedure
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+#### Preprocessing [optional]
+[More Information Needed]
+#### Training Hyperparameters
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+#### Speeds, Sizes, Times [optional]
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+[More Information Needed]
+## Evaluation
+<!-- This section describes the evaluation protocols and provides the results. -->
+### Testing Data, Factors & Metrics
+#### Testing Data
+<!-- This should link to a Dataset Card if possible. -->
+[More Information Needed]
+#### Factors
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+[More Information Needed]
+#### Metrics
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+[More Information Needed]
+### Results
+[More Information Needed]
+#### Summary
+## Model Examination [optional]
+<!-- Relevant interpretability work for the model goes here -->
+[More Information Needed]
+## Environmental Impact
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+## Technical Specifications [optional]
+### Model Architecture and Objective
+[More Information Needed]
+### Compute Infrastructure
+[More Information Needed]
+#### Hardware
+[More Information Needed]
+#### Software
+[More Information Needed]
+## Citation [optional]
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+**BibTeX:**
+[More Information Needed]
+**APA:**
+[More Information Needed]
+## Glossary [optional]
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+[More Information Needed]
+## More Information [optional]
+[More Information Needed]
+## Model Card Authors [optional]
+[More Information Needed]
+## Model Card Contact
+[More Information Needed]
+### Framework versions
+- PEFT 0.18.0

checkpoint-625/adapter_config.json ADDED Viewed

	@@ -0,0 +1,43 @@

+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "EleutherAI/pythia-2.8b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "dense",
+    "query_key_value",
+    "dense_h_to_4h",
+    "dense_4h_to_h"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-625/adapter_model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1ccfefe7a2064e3395163c868acd08121de22856abb4e814fb07586fc66238ab
+size 335580648

checkpoint-625/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b0714aa86711ce6e4b13e70c183a3c1fb92bd2bc36ab911c3377f1228f56aec0
+size 671308491

checkpoint-625/rng_state.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4182c44906c914758f2e5dea2e01a62e3ccd3f25d5ec02d800c4cd4da65dfef4
+size 14645

checkpoint-625/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:552da27b2294ad9b7370549845686ae6db1e70f2682382baac9dc5c4fe33d3c5
+size 1465

checkpoint-625/special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,24 @@

+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-625/tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

checkpoint-625/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,215 @@

+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50254": {
+      "content": "                        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50255": {
+      "content": "                       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50256": {
+      "content": "                      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50257": {
+      "content": "                     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "                    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "                   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "                  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "                 ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "                ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "               ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "              ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "             ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "            ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "           ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "          ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "         ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-625/trainer_state.json ADDED Viewed

	@@ -0,0 +1,720 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 100,
+  "global_step": 625,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 2.927482557296753,
+      "epoch": 0.016,
+      "grad_norm": 1.3331365585327148,
+      "learning_rate": 4.736842105263158e-05,
+      "loss": 2.9205,
+      "mean_token_accuracy": 0.42432610020041467,
+      "num_tokens": 21466.0,
+      "step": 10
+    },
+    {
+      "entropy": 2.7787948846817017,
+      "epoch": 0.032,
+      "grad_norm": 1.0274778604507446,
+      "learning_rate": 0.0001,
+      "loss": 2.7193,
+      "mean_token_accuracy": 0.4424279421567917,
+      "num_tokens": 44204.0,
+      "step": 20
+    },
+    {
+      "entropy": 2.7391002118587493,
+      "epoch": 0.048,
+      "grad_norm": 0.9975764751434326,
+      "learning_rate": 0.00015263157894736845,
+      "loss": 2.6235,
+      "mean_token_accuracy": 0.46106590032577516,
+      "num_tokens": 63982.0,
+      "step": 30
+    },
+    {
+      "entropy": 2.69979213476181,
+      "epoch": 0.064,
+      "grad_norm": 0.9850704669952393,
+      "learning_rate": 0.00019999966405802826,
+      "loss": 2.5825,
+      "mean_token_accuracy": 0.4666088931262493,
+      "num_tokens": 84531.0,
+      "step": 40
+    },
+    {
+      "entropy": 2.6038752436637878,
+      "epoch": 0.08,
+      "grad_norm": 0.9963482022285461,
+      "learning_rate": 0.00019995935375248606,
+      "loss": 2.5338,
+      "mean_token_accuracy": 0.4776237912476063,
+      "num_tokens": 104496.0,
+      "step": 50
+    },
+    {
+      "entropy": 2.6396988570690154,
+      "epoch": 0.096,
+      "grad_norm": 0.8940117955207825,
+      "learning_rate": 0.000199851886084842,
+      "loss": 2.5631,
+      "mean_token_accuracy": 0.4724681057035923,
+      "num_tokens": 125055.0,
+      "step": 60
+    },
+    {
+      "entropy": 2.624293786287308,
+      "epoch": 0.112,
+      "grad_norm": 0.7860824465751648,
+      "learning_rate": 0.0001996773332568941,
+      "loss": 2.5196,
+      "mean_token_accuracy": 0.47606254518032076,
+      "num_tokens": 145437.0,
+      "step": 70
+    },
+    {
+      "entropy": 2.523421919345856,
+      "epoch": 0.128,
+      "grad_norm": 0.7847617268562317,
+      "learning_rate": 0.0001994358125413841,
+      "loss": 2.438,
+      "mean_token_accuracy": 0.4916156314313412,
+      "num_tokens": 164918.0,
+      "step": 80
+    },
+    {
+      "entropy": 2.5297053515911103,
+      "epoch": 0.144,
+      "grad_norm": 0.8938744068145752,
+      "learning_rate": 0.00019912748620320794,
+      "loss": 2.477,
+      "mean_token_accuracy": 0.4871461518108845,
+      "num_tokens": 186602.0,
+      "step": 90
+    },
+    {
+      "entropy": 2.549694550037384,
+      "epoch": 0.16,
+      "grad_norm": 0.8062200546264648,
+      "learning_rate": 0.00019875256139039902,
+      "loss": 2.4538,
+      "mean_token_accuracy": 0.48964215591549876,
+      "num_tokens": 208334.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.16,
+      "eval_entropy": 2.5797194150777965,
+      "eval_loss": 2.5428755283355713,
+      "eval_mean_token_accuracy": 0.48654096172406125,
+      "eval_num_tokens": 208334.0,
+      "eval_runtime": 0.7687,
+      "eval_samples_per_second": 130.09,
+      "eval_steps_per_second": 16.912,
+      "step": 100
+    },
+    {
+      "entropy": 2.5369359493255614,
+      "epoch": 0.176,
+      "grad_norm": 0.7858350276947021,
+      "learning_rate": 0.00019831128999495606,
+      "loss": 2.451,
+      "mean_token_accuracy": 0.49068020582199096,
+      "num_tokens": 229573.0,
+      "step": 110
+    },
+    {
+      "entropy": 2.622505843639374,
+      "epoch": 0.192,
+      "grad_norm": 0.853503942489624,
+      "learning_rate": 0.0001978039684836106,
+      "loss": 2.513,
+      "mean_token_accuracy": 0.4789179593324661,
+      "num_tokens": 250739.0,
+      "step": 120
+    },
+    {
+      "entropy": 2.538061708211899,
+      "epoch": 0.208,
+      "grad_norm": 0.701422393321991,
+      "learning_rate": 0.00019723093769864663,
+      "loss": 2.4881,
+      "mean_token_accuracy": 0.485482332110405,
+      "num_tokens": 274310.0,
+      "step": 130
+    },
+    {
+      "entropy": 2.633000612258911,
+      "epoch": 0.224,
+      "grad_norm": 0.8620439171791077,
+      "learning_rate": 0.00019659258262890683,
+      "loss": 2.5312,
+      "mean_token_accuracy": 0.4774567261338234,
+      "num_tokens": 295034.0,
+      "step": 140
+    },
+    {
+      "entropy": 2.501206773519516,
+      "epoch": 0.24,
+      "grad_norm": 0.8027399778366089,
+      "learning_rate": 0.00019588933215113926,
+      "loss": 2.4184,
+      "mean_token_accuracy": 0.4926304392516613,
+      "num_tokens": 315487.0,
+      "step": 150
+    },
+    {
+      "entropy": 2.51069019138813,
+      "epoch": 0.256,
+      "grad_norm": 0.7904124855995178,
+      "learning_rate": 0.00019512165874185767,
+      "loss": 2.4492,
+      "mean_token_accuracy": 0.49440020844340327,
+      "num_tokens": 336958.0,
+      "step": 160
+    },
+    {
+      "entropy": 2.5472387611865996,
+      "epoch": 0.272,
+      "grad_norm": 0.8189327120780945,
+      "learning_rate": 0.00019429007815990993,
+      "loss": 2.4559,
+      "mean_token_accuracy": 0.48381901159882545,
+      "num_tokens": 358616.0,
+      "step": 170
+    },
+    {
+      "entropy": 2.547665923833847,
+      "epoch": 0.288,
+      "grad_norm": 0.7772969603538513,
+      "learning_rate": 0.00019339514909996706,
+      "loss": 2.4706,
+      "mean_token_accuracy": 0.4824284970760345,
+      "num_tokens": 379872.0,
+      "step": 180
+    },
+    {
+      "entropy": 2.504627192020416,
+      "epoch": 0.304,
+      "grad_norm": 0.6993403434753418,
+      "learning_rate": 0.000192437472817166,
+      "loss": 2.4443,
+      "mean_token_accuracy": 0.4916023284196854,
+      "num_tokens": 400768.0,
+      "step": 190
+    },
+    {
+      "entropy": 2.5959001183509827,
+      "epoch": 0.32,
+      "grad_norm": 0.8900535106658936,
+      "learning_rate": 0.00019141769272315858,
+      "loss": 2.5256,
+      "mean_token_accuracy": 0.4826003320515156,
+      "num_tokens": 420318.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.32,
+      "eval_entropy": 2.615577771113469,
+      "eval_loss": 2.544278144836426,
+      "eval_mean_token_accuracy": 0.4823318009193127,
+      "eval_num_tokens": 420318.0,
+      "eval_runtime": 0.6723,
+      "eval_samples_per_second": 148.739,
+      "eval_steps_per_second": 19.336,
+      "step": 200
+    },
+    {
+      "entropy": 2.5907178699970244,
+      "epoch": 0.336,
+      "grad_norm": 0.7852405309677124,
+      "learning_rate": 0.00019033649395383702,
+      "loss": 2.5025,
+      "mean_token_accuracy": 0.48569325879216196,
+      "num_tokens": 440493.0,
+      "step": 210
+    },
+    {
+      "entropy": 2.5373390048742293,
+      "epoch": 0.352,
+      "grad_norm": 0.8513323664665222,
+      "learning_rate": 0.00018919460290902826,
+      "loss": 2.4644,
+      "mean_token_accuracy": 0.48737777322530745,
+      "num_tokens": 460347.0,
+      "step": 220
+    },
+    {
+      "entropy": 2.5982253909111024,
+      "epoch": 0.368,
+      "grad_norm": 0.7951892614364624,
+      "learning_rate": 0.00018799278676446423,
+      "loss": 2.5088,
+      "mean_token_accuracy": 0.4845408618450165,
+      "num_tokens": 479686.0,
+      "step": 230
+    },
+    {
+      "entropy": 2.5408327341079713,
+      "epoch": 0.384,
+      "grad_norm": 0.818739652633667,
+      "learning_rate": 0.0001867318529563574,
+      "loss": 2.4453,
+      "mean_token_accuracy": 0.49112383723258973,
+      "num_tokens": 499382.0,
+      "step": 240
+    },
+    {
+      "entropy": 2.566703599691391,
+      "epoch": 0.4,
+      "grad_norm": 0.8289343118667603,
+      "learning_rate": 0.00018541264863892754,
+      "loss": 2.5047,
+      "mean_token_accuracy": 0.48281258195638654,
+      "num_tokens": 520865.0,
+      "step": 250
+    },
+    {
+      "entropy": 2.522556847333908,
+      "epoch": 0.416,
+      "grad_norm": 0.9309546947479248,
+      "learning_rate": 0.000184036060115244,
+      "loss": 2.4434,
+      "mean_token_accuracy": 0.4828334227204323,
+      "num_tokens": 540616.0,
+      "step": 260
+    },
+    {
+      "entropy": 2.5191095739603044,
+      "epoch": 0.432,
+      "grad_norm": 0.7696341276168823,
+      "learning_rate": 0.00018260301224176558,
+      "loss": 2.4318,
+      "mean_token_accuracy": 0.4892562575638294,
+      "num_tokens": 561098.0,
+      "step": 270
+    },
+    {
+      "entropy": 2.5569592356681823,
+      "epoch": 0.448,
+      "grad_norm": 0.7310771346092224,
+      "learning_rate": 0.00018111446780697929,
+      "loss": 2.505,
+      "mean_token_accuracy": 0.476550355553627,
+      "num_tokens": 581527.0,
+      "step": 280
+    },
+    {
+      "entropy": 2.6550574243068694,
+      "epoch": 0.464,
+      "grad_norm": 0.8539873957633972,
+      "learning_rate": 0.00017957142688455362,
+      "loss": 2.5309,
+      "mean_token_accuracy": 0.479371577501297,
+      "num_tokens": 601002.0,
+      "step": 290
+    },
+    {
+      "entropy": 2.5720873057842253,
+      "epoch": 0.48,
+      "grad_norm": 0.7261855006217957,
+      "learning_rate": 0.00017797492616144256,
+      "loss": 2.5289,
+      "mean_token_accuracy": 0.4779372848570347,
+      "num_tokens": 621836.0,
+      "step": 300
+    },
+    {
+      "epoch": 0.48,
+      "eval_entropy": 2.58707464658297,
+      "eval_loss": 2.5298879146575928,
+      "eval_mean_token_accuracy": 0.4877227544784546,
+      "eval_num_tokens": 621836.0,
+      "eval_runtime": 0.6775,
+      "eval_samples_per_second": 147.608,
+      "eval_steps_per_second": 19.189,
+      "step": 300
+    },
+    {
+      "entropy": 2.56911381483078,
+      "epoch": 0.496,
+      "grad_norm": 0.8215358257293701,
+      "learning_rate": 0.00017632603824139085,
+      "loss": 2.4587,
+      "mean_token_accuracy": 0.4831582002341747,
+      "num_tokens": 642910.0,
+      "step": 310
+    },
+    {
+      "entropy": 2.518597739934921,
+      "epoch": 0.512,
+      "grad_norm": 0.7745339274406433,
+      "learning_rate": 0.00017462587092430875,
+      "loss": 2.4552,
+      "mean_token_accuracy": 0.48742511570453645,
+      "num_tokens": 665281.0,
+      "step": 320
+    },
+    {
+      "entropy": 2.5539629220962525,
+      "epoch": 0.528,
+      "grad_norm": 1.3013852834701538,
+      "learning_rate": 0.00017287556646200018,
+      "loss": 2.5049,
+      "mean_token_accuracy": 0.48348471373319624,
+      "num_tokens": 686496.0,
+      "step": 330
+    },
+    {
+      "entropy": 2.549721211194992,
+      "epoch": 0.544,
+      "grad_norm": 0.9040252566337585,
+      "learning_rate": 0.00017107630079074478,
+      "loss": 2.4749,
+      "mean_token_accuracy": 0.49018326848745347,
+      "num_tokens": 708041.0,
+      "step": 340
+    },
+    {
+      "entropy": 2.5657508850097654,
+      "epoch": 0.56,
+      "grad_norm": 0.7486541867256165,
+      "learning_rate": 0.00016922928274124886,
+      "loss": 2.496,
+      "mean_token_accuracy": 0.4882775545120239,
+      "num_tokens": 729373.0,
+      "step": 350
+    },
+    {
+      "entropy": 2.49460369348526,
+      "epoch": 0.576,
+      "grad_norm": 0.7519723176956177,
+      "learning_rate": 0.00016733575322649657,
+      "loss": 2.4358,
+      "mean_token_accuracy": 0.4897538281977177,
+      "num_tokens": 751428.0,
+      "step": 360
+    },
+    {
+      "entropy": 2.485343414545059,
+      "epoch": 0.592,
+      "grad_norm": 0.7465973496437073,
+      "learning_rate": 0.00016539698440804661,
+      "loss": 2.4446,
+      "mean_token_accuracy": 0.4956702694296837,
+      "num_tokens": 773898.0,
+      "step": 370
+    },
+    {
+      "entropy": 2.5664760768413544,
+      "epoch": 0.608,
+      "grad_norm": 0.792957603931427,
+      "learning_rate": 0.0001634142788413346,
+      "loss": 2.4411,
+      "mean_token_accuracy": 0.49218829050660134,
+      "num_tokens": 793921.0,
+      "step": 380
+    },
+    {
+      "entropy": 2.4981964468955993,
+      "epoch": 0.624,
+      "grad_norm": 0.9188303351402283,
+      "learning_rate": 0.00016138896860055555,
+      "loss": 2.4527,
+      "mean_token_accuracy": 0.4891628086566925,
+      "num_tokens": 814549.0,
+      "step": 390
+    },
+    {
+      "entropy": 2.5058595299720765,
+      "epoch": 0.64,
+      "grad_norm": 0.8261463642120361,
+      "learning_rate": 0.0001593224143837142,
+      "loss": 2.3825,
+      "mean_token_accuracy": 0.49732586443424226,
+      "num_tokens": 835738.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.64,
+      "eval_entropy": 2.5427963917072,
+      "eval_loss": 2.528325080871582,
+      "eval_mean_token_accuracy": 0.48722161238010114,
+      "eval_num_tokens": 835738.0,
+      "eval_runtime": 0.6794,
+      "eval_samples_per_second": 147.184,
+      "eval_steps_per_second": 19.134,
+      "step": 400
+    },
+    {
+      "entropy": 2.549647295475006,
+      "epoch": 0.656,
+      "grad_norm": 0.9669468402862549,
+      "learning_rate": 0.00015721600459844468,
+      "loss": 2.4992,
+      "mean_token_accuracy": 0.4811323322355747,
+      "num_tokens": 856308.0,
+      "step": 410
+    },
+    {
+      "entropy": 2.50358769595623,
+      "epoch": 0.672,
+      "grad_norm": 0.7736469507217407,
+      "learning_rate": 0.0001550711544292131,
+      "loss": 2.4233,
+      "mean_token_accuracy": 0.4960208125412464,
+      "num_tokens": 877984.0,
+      "step": 420
+    },
+    {
+      "entropy": 2.513987493515015,
+      "epoch": 0.688,
+      "grad_norm": 1.0335294008255005,
+      "learning_rate": 0.00015288930488653094,
+      "loss": 2.4349,
+      "mean_token_accuracy": 0.490431859344244,
+      "num_tokens": 899593.0,
+      "step": 430
+    },
+    {
+      "entropy": 2.5122825741767882,
+      "epoch": 0.704,
+      "grad_norm": 1.0326985120773315,
+      "learning_rate": 0.00015067192183881658,
+      "loss": 2.4435,
+      "mean_token_accuracy": 0.49002100676298144,
+      "num_tokens": 919549.0,
+      "step": 440
+    },
+    {
+      "entropy": 2.5058334708213805,
+      "epoch": 0.72,
+      "grad_norm": 0.8282021284103394,
+      "learning_rate": 0.0001484204950275565,
+      "loss": 2.3755,
+      "mean_token_accuracy": 0.49881314411759375,
+      "num_tokens": 938371.0,
+      "step": 450
+    },
+    {
+      "entropy": 2.5388923943042756,
+      "epoch": 0.736,
+      "grad_norm": 0.8473682403564453,
+      "learning_rate": 0.0001461365370664276,
+      "loss": 2.4695,
+      "mean_token_accuracy": 0.48638317435979844,
+      "num_tokens": 959081.0,
+      "step": 460
+    },
+    {
+      "entropy": 2.4723076283931733,
+      "epoch": 0.752,
+      "grad_norm": 0.964932918548584,
+      "learning_rate": 0.00014382158242505234,
+      "loss": 2.4151,
+      "mean_token_accuracy": 0.4942440494894981,
+      "num_tokens": 980709.0,
+      "step": 470
+    },
+    {
+      "entropy": 2.4699651658535005,
+      "epoch": 0.768,
+      "grad_norm": 0.8604968190193176,
+      "learning_rate": 0.0001414771863980707,
+      "loss": 2.4249,
+      "mean_token_accuracy": 0.49896738603711127,
+      "num_tokens": 998177.0,
+      "step": 480
+    },
+    {
+      "entropy": 2.5598580718040465,
+      "epoch": 0.784,
+      "grad_norm": 0.8044160604476929,
+      "learning_rate": 0.00013910492406022033,
+      "loss": 2.5119,
+      "mean_token_accuracy": 0.48558894991874696,
+      "num_tokens": 1019231.0,
+      "step": 490
+    },
+    {
+      "entropy": 2.5347262740135195,
+      "epoch": 0.8,
+      "grad_norm": 0.989012598991394,
+      "learning_rate": 0.000136706389208128,
+      "loss": 2.457,
+      "mean_token_accuracy": 0.48754729256033896,
+      "num_tokens": 1039963.0,
+      "step": 500
+    },
+    {
+      "epoch": 0.8,
+      "eval_entropy": 2.5207965007195106,
+      "eval_loss": 2.532066822052002,
+      "eval_mean_token_accuracy": 0.4870225145266606,
+      "eval_num_tokens": 1039963.0,
+      "eval_runtime": 0.6776,
+      "eval_samples_per_second": 147.573,
+      "eval_steps_per_second": 19.185,
+      "step": 500
+    },
+    {
+      "entropy": 2.4576663255691527,
+      "epoch": 0.816,
+      "grad_norm": 0.8774695992469788,
+      "learning_rate": 0.00013428319328952253,
+      "loss": 2.3836,
+      "mean_token_accuracy": 0.4995129629969597,
+      "num_tokens": 1061370.0,
+      "step": 510
+    },
+    {
+      "entropy": 2.5250146806240084,
+      "epoch": 0.832,
+      "grad_norm": 0.7958005666732788,
+      "learning_rate": 0.00013183696432058888,
+      "loss": 2.4315,
+      "mean_token_accuracy": 0.4982760787010193,
+      "num_tokens": 1081842.0,
+      "step": 520
+    },
+    {
+      "entropy": 2.4409106403589247,
+      "epoch": 0.848,
+      "grad_norm": 0.7063387036323547,
+      "learning_rate": 0.00012936934579219094,
+      "loss": 2.3816,
+      "mean_token_accuracy": 0.5006642855703831,
+      "num_tokens": 1103737.0,
+      "step": 530
+    },
+    {
+      "entropy": 2.503674066066742,
+      "epoch": 0.864,
+      "grad_norm": 0.712315559387207,
+      "learning_rate": 0.00012688199556569753,
+      "loss": 2.4427,
+      "mean_token_accuracy": 0.4878311850130558,
+      "num_tokens": 1124676.0,
+      "step": 540
+    },
+    {
+      "entropy": 2.4937715351581575,
+      "epoch": 0.88,
+      "grad_norm": 0.7796215415000916,
+      "learning_rate": 0.00012437658475915377,
+      "loss": 2.4148,
+      "mean_token_accuracy": 0.4976658411324024,
+      "num_tokens": 1147148.0,
+      "step": 550
+    },
+    {
+      "entropy": 2.458594244718552,
+      "epoch": 0.896,
+      "grad_norm": 0.8076977133750916,
+      "learning_rate": 0.00012185479662454595,
+      "loss": 2.414,
+      "mean_token_accuracy": 0.4989797443151474,
+      "num_tokens": 1167948.0,
+      "step": 560
+    },
+    {
+      "entropy": 2.5129783391952514,
+      "epoch": 0.912,
+      "grad_norm": 0.9025003910064697,
+      "learning_rate": 0.00011931832541691418,
+      "loss": 2.4198,
+      "mean_token_accuracy": 0.4928160272538662,
+      "num_tokens": 1187896.0,
+      "step": 570
+    },
+    {
+      "entropy": 2.429825708270073,
+      "epoch": 0.928,
+      "grad_norm": 0.9303080439567566,
+      "learning_rate": 0.00011676887525607271,
+      "loss": 2.336,
+      "mean_token_accuracy": 0.507265867292881,
+      "num_tokens": 1207963.0,
+      "step": 580
+    },
+    {
+      "entropy": 2.483963069319725,
+      "epoch": 0.944,
+      "grad_norm": 0.9662640690803528,
+      "learning_rate": 0.0001142081589817027,
+      "loss": 2.4441,
+      "mean_token_accuracy": 0.4951579734683037,
+      "num_tokens": 1228312.0,
+      "step": 590
+    },
+    {
+      "entropy": 2.5128854513168335,
+      "epoch": 0.96,
+      "grad_norm": 0.7802348732948303,
+      "learning_rate": 0.00011163789700258655,
+      "loss": 2.481,
+      "mean_token_accuracy": 0.4941797606647015,
+      "num_tokens": 1250029.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.96,
+      "eval_entropy": 2.5629419730259824,
+      "eval_loss": 2.526008367538452,
+      "eval_mean_token_accuracy": 0.4862596323856941,
+      "eval_num_tokens": 1250029.0,
+      "eval_runtime": 0.6764,
+      "eval_samples_per_second": 147.833,
+      "eval_steps_per_second": 19.218,
+      "step": 600
+    },
+    {
+      "entropy": 2.45339612364769,
+      "epoch": 0.976,
+      "grad_norm": 0.7918068766593933,
+      "learning_rate": 0.00010905981614075693,
+      "loss": 2.3798,
+      "mean_token_accuracy": 0.5059582263231277,
+      "num_tokens": 1273572.0,
+      "step": 610
+    },
+    {
+      "entropy": 2.5436563432216643,
+      "epoch": 0.992,
+      "grad_norm": 0.9120656847953796,
+      "learning_rate": 0.000106475648471337,
+      "loss": 2.4771,
+      "mean_token_accuracy": 0.49086978659033775,
+      "num_tokens": 1295353.0,
+      "step": 620
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1250,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.76891183669248e+16,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-625/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:879f340dca9a81b05ff3f767198bc8a294c35f0efe288b9f690178d342307834
+size 6353

special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,24 @@

+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,215 @@

+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50254": {
+      "content": "                        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50255": {
+      "content": "                       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50256": {
+      "content": "                      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50257": {
+      "content": "                     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "                    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "                   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "                  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "                 ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "                ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "               ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "              ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "             ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "            ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "           ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "          ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "         ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:879f340dca9a81b05ff3f767198bc8a294c35f0efe288b9f690178d342307834
+size 6353