Publish math LoRA adapter with benchmark results
Browse files
README.md
CHANGED
|
@@ -10,26 +10,13 @@ tags:
|
|
| 10 |
- math
|
| 11 |
---
|
| 12 |
|
| 13 |
-
# math-lora
|
| 14 |
|
| 15 |
-
QLoRA adapter for
|
| 16 |
|
| 17 |
-
|
| 18 |
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
- eval profile: `math`
|
| 22 |
-
- gate: **PASSED**
|
| 23 |
-
|
| 24 |
-
| check | value | result |
|
| 25 |
-
| --- | ---: | --- |
|
| 26 |
-
| gsm8k >= 0.05 | 0.4000 | pass |
|
| 27 |
-
| gsm8k improve >= 0.02 | 0.0700 | pass |
|
| 28 |
-
| arc_challenge regress <= 0.03 | -0.0500 | pass |
|
| 29 |
-
| hellaswag regress <= 0.03 | 0.0000 | pass |
|
| 30 |
-
| piqa regress <= 0.03 | 0.0200 | pass |
|
| 31 |
-
|
| 32 |
-
## lm-eval results
|
| 33 |
|
| 34 |
| task | metric | baseline | candidate | delta |
|
| 35 |
| --- | --- | ---: | ---: | ---: |
|
|
@@ -40,11 +27,9 @@ Trained, evaluated, and gated on [Modal](https://modal.com/docs/guide) via `rese
|
|
| 40 |
|
| 41 |
## Training
|
| 42 |
|
| 43 |
-
-
|
| 44 |
-
- mode: `qlora`
|
| 45 |
-
- samples: {'train': 3528, 'eval': 72}
|
| 46 |
-
- final train loss: 0.340698
|
| 47 |
- eval loss: 0.494981
|
|
|
|
| 48 |
|
| 49 |
## Load with PEFT
|
| 50 |
|
|
@@ -54,11 +39,7 @@ from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
| 54 |
|
| 55 |
base = "openbmb/MiniCPM5-1B"
|
| 56 |
adapter = "MSGEncrypted/minicpm5-1b-math-lora"
|
| 57 |
-
|
| 58 |
tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
|
| 59 |
-
model = AutoModelForCausalLM.from_pretrained(
|
| 60 |
-
base, torch_dtype="auto", device_map="auto", trust_remote_code=True
|
| 61 |
-
)
|
| 62 |
model = PeftModel.from_pretrained(model, adapter)
|
| 63 |
```
|
| 64 |
-
|
|
|
|
| 10 |
- math
|
| 11 |
---
|
| 12 |
|
| 13 |
+
# minicpm5-1b-math-lora
|
| 14 |
|
| 15 |
+
QLoRA math adapter for openbmb/MiniCPM5-1B, trained on meta-math/MetaMathQA with tatsu-lab/alpaca replay.
|
| 16 |
|
| 17 |
+
## Benchmark comparison
|
| 18 |
|
| 19 |
+
Evaluated with research/evals/configs/lm_eval_math.yaml on Modal using slm-lm-eval.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
| task | metric | baseline | candidate | delta |
|
| 22 |
| --- | --- | ---: | ---: | ---: |
|
|
|
|
| 27 |
|
| 28 |
## Training
|
| 29 |
|
| 30 |
+
- train loss: -
|
|
|
|
|
|
|
|
|
|
| 31 |
- eval loss: 0.494981
|
| 32 |
+
- result score: -
|
| 33 |
|
| 34 |
## Load with PEFT
|
| 35 |
|
|
|
|
| 39 |
|
| 40 |
base = "openbmb/MiniCPM5-1B"
|
| 41 |
adapter = "MSGEncrypted/minicpm5-1b-math-lora"
|
|
|
|
| 42 |
tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
|
| 43 |
+
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto", trust_remote_code=True)
|
|
|
|
|
|
|
| 44 |
model = PeftModel.from_pretrained(model, adapter)
|
| 45 |
```
|
|
|