Update README.md
Browse files
README.md
CHANGED
|
@@ -65,22 +65,24 @@ The model maintains the original Supra-1.5-Base parameter structure and tokenize
|
|
| 65 |
|
| 66 |
## Model Benchmarks
|
| 67 |
|
| 68 |
-
We evaluated the model against the official test splits of
|
| 69 |
|
| 70 |
-
|
| 71 |
-
| :--- | :---: | :---: | :---: |
|
| 72 |
-
| **ARC-Easy Accuracy** | `40.66%` | **`42.26%`** | **`+1.60%`** |
|
| 73 |
-
| **SciQ Accuracy** | `40.80%` | **`44.90%`** | **`+4.10%`** |
|
| 74 |
-
| **HellaSwag Accuracy** | `31.50%` | **`31.56%`** | **`+0.06%`** |
|
| 75 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
|
| 77 |
-
|
| 78 |
-
> Please treat these evaluation scores as **highly preliminary and rough approximations**. Language model benchmarks are notoriously unstable and hyper-sensitive to minor variations in prompt formatting, tokenizer alignment, and specific subset slicing.
|
| 79 |
-
>
|
| 80 |
-
> While we have used a mathematically rigorous harness, these results are strictly internal observations. Actual performance on the official Open LLM Leaderboard or in localized user environments can, and likely will, differ significantly. **We are currently performing an extensive audit of our evaluation code to maximize accuracy and parity with industry standards; we fully expect to revise and update these metrics as more precise data becomes available.**
|
| 81 |
-
>
|
| 82 |
-
---
|
| 83 |
|
|
|
|
|
|
|
|
|
|
| 84 |
## Intended Use & Chat Template
|
| 85 |
|
| 86 |
This model is intended for experimental research, lightweight conversational prototyping, and low-latency edge deployment. It is formatted natively to understand standard **ChatML** syntax:
|
|
|
|
| 65 |
|
| 66 |
## Model Benchmarks
|
| 67 |
|
| 68 |
+
We evaluated the model against the official test splits of six standard datasets using a mathematically rigorous log-likelihood evaluation harness (`lm-evaluation-harness`).
|
| 69 |
|
| 70 |
+
Below is the verified comparison across both raw accuracy (`acc`) and length-normalized accuracy (`acc_norm`) against the v1.5 baseline.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
|
| 72 |
+
| Benchmark Task | Metric | Supra-1.5-Instruct (Baseline)* | Supra-1.6-Instruct-Ultra-exp (v1.6) | Performance Delta |
|
| 73 |
+
| :--- | :---: | :---: | :---: | :---: |
|
| 74 |
+
| **SciQ** | `acc` <br> `acc_norm` | `60.90%` <br> `57.40%` | **`72.70%`** <br> **`66.00%`** | **`+11.80%`** <br> **`+8.60%`** |
|
| 75 |
+
| **PIQA** | `acc` <br> `acc_norm` | `59.60%` <br> `59.30%` | **`60.61%`** <br> **`59.41%`** | **`+1.01%`** <br> **`+0.11%`** |
|
| 76 |
+
| **HellaSwag** | `acc` <br> `acc_norm` | `27.90%` <br> `29.30%` | **`28.11%`** <br> **`29.66%`** | **`+0.21%`** <br> **`+0.36%`** |
|
| 77 |
+
| **OpenBookQA** | `acc` <br> `acc_norm` | **`17.80%`** <br> `26.60%` | `17.40%` <br> **`27.20%`** | `-0.40%` <br> **`+0.60%`** |
|
| 78 |
+
| **ARC-Easy** | `acc` <br> `acc_norm` | `45.90%` <br> **`44.10%`** | **`46.76%`** <br> `43.18%` | **`+0.86%`** <br> `-0.92%` |
|
| 79 |
+
| **ARC-Challenge** | `acc` <br> `acc_norm` | **`22.90%`** <br> **`25.90%`** | `22.35%` <br> `25.60%` | `-0.55%` <br> `-0.30%` |
|
| 80 |
|
| 81 |
+
*\*Supra-1.5-Instruct (Baseline) metrics are sourced directly from the official [SupraLabs/Supra-1.5-50M-Instruct-exp](https://huggingface.co/SupraLabs/Supra-1.5-50M-Instruct-exp) repository.*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
|
| 83 |
+
> **💡 Benchmarking Notes & Parity Disclaimer:**
|
| 84 |
+
> These evaluation scores are measured using standard benchmarking configurations. While the evaluation harness is mathematically standardized, minor variations in output can occur based on localized system setups, exact prompt formatting, and tokenizer defaults. These results are shared to provide an objective relative comparison between the model iterations under identical benchmarking conditions.
|
| 85 |
+
>
|
| 86 |
## Intended Use & Chat Template
|
| 87 |
|
| 88 |
This model is intended for experimental research, lightweight conversational prototyping, and low-latency edge deployment. It is formatted natively to understand standard **ChatML** syntax:
|