Upload folder using huggingface_hub

README.md

@@ -1,62 +1,209 @@
 ---
 base_model: Qwen/Qwen2.5-7B-Instruct
 library_name: peft
-model_name: foamqwen
+pipeline_tag: text-generation
 tags:
 - base_model:adapter:Qwen/Qwen2.5-7B-Instruct
 - lora
 - sft
 - transformers
 - trl
-licence: license
-pipeline_tag: text-generation
 ---
 
-# Model Card for foamqwen
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
 
-This model is a fine-tuned version of [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct).
-It has been trained using [TRL](https://github.com/huggingface/trl).
+## Glossary [optional]
 
-## Quick start
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
 
-```python
-from transformers import pipeline
+[More Information Needed]
 
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="None", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
-```
+## More Information [optional]
 
-## Training procedure
+[More Information Needed]
 
- 
+## Model Card Authors [optional]
 
+[More Information Needed]
 
-This model was trained with SFT.
+## Model Card Contact
 
+[More Information Needed]
 ### Framework versions
 
-- PEFT 0.17.0
-- TRL: 0.19.0
-- Transformers: 4.52.3
-- Pytorch: 2.7.0
-- Datasets: 4.4.1
-- Tokenizers: 0.21.4
-
-## Citations
-
-
-
-Cite TRL as:
-    
-```bibtex
-@misc{vonwerra2022trl,
-	title        = {{TRL: Transformer Reinforcement Learning}},
-	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
-	year         = 2020,
-	journal      = {GitHub repository},
-	publisher    = {GitHub},
-	howpublished = {\url{https://github.com/huggingface/trl}}
-}
-```
+- PEFT 0.17.0

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5b94987d8426aeb00a412e4085918f2279de9f2c02430c44b8b06c4f032df1a6
+oid sha256:af95b1c5a220d6a55dfa035d107d9e91bfabf1b2db706471abe4481fe57a2c18
 size 645975704

optimizer.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5c99e8926d4c596b7e29f88b3e7af9802b8452379b4a272adec05c9e868858bd
+oid sha256:a94642b6cbfd74bb89c873dfb0f163ca25cacf3de48ebe5602d93d089f703814
 size 1292087499

rng_state_0.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ba38bdecc343a6514c6f5ba7ea1cc651e5dadf81e9b371b1ca715c7bad2e3dc6
+oid sha256:1076c60313859db66ff0de37f0791c974b8c08d8b63ddb5f7cbe475f61adaae8
 size 14917

rng_state_1.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4e1050514bdfa0ed27320f41e9c255c0e138bd459c14db299a56274532d69dd2
+oid sha256:4b18641396ff6afcf33050e5d126e243d87afe83252f83da2f3ac56c52a96e88
 size 14917

scaler.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:25c7f0c11a632f64b9e7e8aaa231e0f7ca0acbc577fe4fef5c0b323f9b2e5a60
+oid sha256:5aa1cb0b33cdae18d9d01e7f84345dc8aa89b0d8db4af3c1d3869e0251d9ef27
 size 1383

scheduler.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:54093d653e19af28f1975f14611ada654488878917f02e00680cafe281c9818a
+oid sha256:b679dc767877b4670dc9f5034576dafc9e3e774d98ec6626b0317647bc6c0dbb
 size 1465

trainer_state.json

@@ -2,9 +2,9 @@
   "best_global_step": null,
   "best_metric": null,
   "best_model_checkpoint": null,
-  "epoch": 3.0,
+  "epoch": 5.0,
   "eval_steps": 500,
-  "global_step": 624,
+  "global_step": 1040,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
@@ -254,6 +254,179 @@
       "eval_samples_per_second": 25.623,
       "eval_steps_per_second": 6.458,
       "step": 624
+    },
+    {
+      "epoch": 3.004825090470446,
+      "grad_norm": 0.07378800958395004,
+      "learning_rate": 0.0003261924543602117,
+      "loss": 0.0346,
+      "mean_token_accuracy": 0.990131803394593,
+      "num_tokens": 7081259.0,
+      "step": 625
+    },
+    {
+      "epoch": 3.1254523522316044,
+      "grad_norm": 0.06171029433608055,
+      "learning_rate": 0.00031243326938037763,
+      "loss": 0.0295,
+      "mean_token_accuracy": 0.9908573353290557,
+      "num_tokens": 7365656.0,
+      "step": 650
+    },
+    {
+      "epoch": 3.2460796139927623,
+      "grad_norm": 0.058736372739076614,
+      "learning_rate": 0.00029849798230773775,
+      "loss": 0.0282,
+      "mean_token_accuracy": 0.9913074177503586,
+      "num_tokens": 7649008.0,
+      "step": 675
+    },
+    {
+      "epoch": 3.3667068757539202,
+      "grad_norm": 0.05701276659965515,
+      "learning_rate": 0.0002844296968186495,
+      "loss": 0.0281,
+      "mean_token_accuracy": 0.9912878751754761,
+      "num_tokens": 7933268.0,
+      "step": 700
+    },
+    {
+      "epoch": 3.4873341375150786,
+      "grad_norm": 0.05498967319726944,
+      "learning_rate": 0.00027027192797107074,
+      "loss": 0.0294,
+      "mean_token_accuracy": 0.9906794387102127,
+      "num_tokens": 8215897.0,
+      "step": 725
+    },
+    {
+      "epoch": 3.6079613992762365,
+      "grad_norm": 0.06823839247226715,
+      "learning_rate": 0.00025606846760676025,
+      "loss": 0.0277,
+      "mean_token_accuracy": 0.9914608508348465,
+      "num_tokens": 8501329.0,
+      "step": 750
+    },
+    {
+      "epoch": 3.7285886610373944,
+      "grad_norm": 0.0680052787065506,
+      "learning_rate": 0.00024186324889734987,
+      "loss": 0.0274,
+      "mean_token_accuracy": 0.9915587675571441,
+      "num_tokens": 8785631.0,
+      "step": 775
+    },
+    {
+      "epoch": 3.8492159227985523,
+      "grad_norm": 0.05490550398826599,
+      "learning_rate": 0.00022770021045326374,
+      "loss": 0.0258,
+      "mean_token_accuracy": 0.9919438254833222,
+      "num_tokens": 9069077.0,
+      "step": 800
+    },
+    {
+      "epoch": 3.9698431845597106,
+      "grad_norm": 0.03772555664181709,
+      "learning_rate": 0.00021362316041581623,
+      "loss": 0.0265,
+      "mean_token_accuracy": 0.9916725689172745,
+      "num_tokens": 9353723.0,
+      "step": 825
+    },
+    {
+      "epoch": 4.0,
+      "eval_loss": 0.04437188059091568,
+      "eval_mean_token_accuracy": 0.9882329240922005,
+      "eval_num_tokens": 9420685.0,
+      "eval_runtime": 14.4213,
+      "eval_samples_per_second": 25.587,
+      "eval_steps_per_second": 6.449,
+      "step": 832
+    },
+    {
+      "epoch": 4.086851628468033,
+      "grad_norm": 0.02592896670103073,
+      "learning_rate": 0.00019967564095286626,
+      "loss": 0.0234,
+      "mean_token_accuracy": 0.9925551224000675,
+      "num_tokens": 9636965.0,
+      "step": 850
+    },
+    {
+      "epoch": 4.207478890229192,
+      "grad_norm": 0.023598628118634224,
+      "learning_rate": 0.00018590079357716402,
+      "loss": 0.0238,
+      "mean_token_accuracy": 0.9923965859413147,
+      "num_tokens": 9921224.0,
+      "step": 875
+    },
+    {
+      "epoch": 4.32810615199035,
+      "grad_norm": 0.021419132128357887,
+      "learning_rate": 0.00017234122570397543,
+      "loss": 0.0224,
+      "mean_token_accuracy": 0.99283282995224,
+      "num_tokens": 10206105.0,
+      "step": 900
+    },
+    {
+      "epoch": 4.4487334137515075,
+      "grad_norm": 0.029326746240258217,
+      "learning_rate": 0.00015903887886074125,
+      "loss": 0.0227,
+      "mean_token_accuracy": 0.9927335000038147,
+      "num_tokens": 10489494.0,
+      "step": 925
+    },
+    {
+      "epoch": 4.569360675512666,
+      "grad_norm": 0.029370704665780067,
+      "learning_rate": 0.00014603489895641456,
+      "loss": 0.0225,
+      "mean_token_accuracy": 0.9927709919214248,
+      "num_tokens": 10774122.0,
+      "step": 950
+    },
+    {
+      "epoch": 4.689987937273824,
+      "grad_norm": 0.024111004546284676,
+      "learning_rate": 0.00013336950901175073,
+      "loss": 0.0205,
+      "mean_token_accuracy": 0.9931790089607239,
+      "num_tokens": 11058651.0,
+      "step": 975
+    },
+    {
+      "epoch": 4.810615199034982,
+      "grad_norm": 0.022787606343626976,
+      "learning_rate": 0.00012108188474421214,
+      "loss": 0.0206,
+      "mean_token_accuracy": 0.9933073389530181,
+      "num_tokens": 11342586.0,
+      "step": 1000
+    },
+    {
+      "epoch": 4.93124246079614,
+      "grad_norm": 0.04723374545574188,
+      "learning_rate": 0.00010921003339231984,
+      "loss": 0.0213,
+      "mean_token_accuracy": 0.9930336177349091,
+      "num_tokens": 11627305.0,
+      "step": 1025
+    },
+    {
+      "epoch": 5.0,
+      "eval_loss": 0.03776097670197487,
+      "eval_mean_token_accuracy": 0.9898016606607745,
+      "eval_num_tokens": 11775709.0,
+      "eval_runtime": 14.4053,
+      "eval_samples_per_second": 25.616,
+      "eval_steps_per_second": 6.456,
+      "step": 1040
     }
   ],
   "logging_steps": 25,
@@ -273,7 +446,7 @@
       "attributes": {}
     }
   },
-  "total_flos": 3.073091636570358e+17,
+  "total_flos": 5.12159461381505e+17,
   "train_batch_size": 2,
   "trial_name": null,
   "trial_params": null