Update README.md
Browse files
README.md
CHANGED
|
@@ -18,5 +18,14 @@ datasets:
|
|
| 18 |
- interview-eval/MATH
|
| 19 |
- RLAIF/mbpp
|
| 20 |
---
|
| 21 |
-
Checkpoint info: Loss around 1.8. Epoch 2. More data unavaiable due to my trainer bugging out.<br>
|
| 22 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
- interview-eval/MATH
|
| 19 |
- RLAIF/mbpp
|
| 20 |
---
|
| 21 |
+
Checkpoint info: Loss around 1.8 when training. Epoch 2. More data unavaiable due to my trainer bugging out.<br>
|
| 22 |
+
Loss on whole dataset: 2.0506176433255594 <br>
|
| 23 |
+
Perplexity on whole dataset: 7.77 <br>
|
| 24 |
+
Base model: https://huggingface.co/simonko912/simllama-1.1-base <br>
|
| 25 |
+
|
| 26 |
+
| Dataset | Loss | Perplexity |
|
| 27 |
+
|----------------|-------|------------|
|
| 28 |
+
| WikiText-2 | 3.188 | 24.24 |
|
| 29 |
+
| Penn Treebank | 4.059 | 57.94 |
|
| 30 |
+
| LAMBADA | 3.656 | 38.70 |
|
| 31 |
+
| C4 | 3.302 | 27.16 |
|