MultivexAI commited on
Commit
f3a7d63
·
verified ·
1 Parent(s): b4387d5

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -12
README.md CHANGED
@@ -65,22 +65,24 @@ The model maintains the original Supra-1.5-Base parameter structure and tokenize
65
 
66
  ## Model Benchmarks
67
 
68
- We evaluated the model against the official test splits of three standard datasets using a mathematically rigorous, token-space-aligned log-likelihood evaluation harness. The results demonstrate a clean sweep over the official v1.5 Instruct baseline.
69
 
70
- | Benchmark Task | Supra-1.5-Instruct (Baseline) | Supra-1.6-Instruct-Ultra-exp (v1.6) | Performance Delta |
71
- | :--- | :---: | :---: | :---: |
72
- | **ARC-Easy Accuracy** | `40.66%` | **`42.26%`** | **`+1.60%`** |
73
- | **SciQ Accuracy** | `40.80%` | **`44.90%`** | **`+4.10%`** |
74
- | **HellaSwag Accuracy** | `31.50%` | **`31.56%`** | **`+0.06%`** |
75
 
 
 
 
 
 
 
 
 
76
 
77
- > **🛑 CRITICAL BENCHMARK LIMITATION WARNING:**
78
- > Please treat these evaluation scores as **highly preliminary and rough approximations**. Language model benchmarks are notoriously unstable and hyper-sensitive to minor variations in prompt formatting, tokenizer alignment, and specific subset slicing.
79
- >
80
- > While we have used a mathematically rigorous harness, these results are strictly internal observations. Actual performance on the official Open LLM Leaderboard or in localized user environments can, and likely will, differ significantly. **We are currently performing an extensive audit of our evaluation code to maximize accuracy and parity with industry standards; we fully expect to revise and update these metrics as more precise data becomes available.**
81
- >
82
- ---
83
 
 
 
 
84
  ## Intended Use & Chat Template
85
 
86
  This model is intended for experimental research, lightweight conversational prototyping, and low-latency edge deployment. It is formatted natively to understand standard **ChatML** syntax:
 
65
 
66
  ## Model Benchmarks
67
 
68
+ We evaluated the model against the official test splits of six standard datasets using a mathematically rigorous log-likelihood evaluation harness (`lm-evaluation-harness`).
69
 
70
+ Below is the verified comparison across both raw accuracy (`acc`) and length-normalized accuracy (`acc_norm`) against the v1.5 baseline.
 
 
 
 
71
 
72
+ | Benchmark Task | Metric | Supra-1.5-Instruct (Baseline)* | Supra-1.6-Instruct-Ultra-exp (v1.6) | Performance Delta |
73
+ | :--- | :---: | :---: | :---: | :---: |
74
+ | **SciQ** | `acc` <br> `acc_norm` | `60.90%` <br> `57.40%` | **`72.70%`** <br> **`66.00%`** | **`+11.80%`** <br> **`+8.60%`** |
75
+ | **PIQA** | `acc` <br> `acc_norm` | `59.60%` <br> `59.30%` | **`60.61%`** <br> **`59.41%`** | **`+1.01%`** <br> **`+0.11%`** |
76
+ | **HellaSwag** | `acc` <br> `acc_norm` | `27.90%` <br> `29.30%` | **`28.11%`** <br> **`29.66%`** | **`+0.21%`** <br> **`+0.36%`** |
77
+ | **OpenBookQA** | `acc` <br> `acc_norm` | **`17.80%`** <br> `26.60%` | `17.40%` <br> **`27.20%`** | `-0.40%` <br> **`+0.60%`** |
78
+ | **ARC-Easy** | `acc` <br> `acc_norm` | `45.90%` <br> **`44.10%`** | **`46.76%`** <br> `43.18%` | **`+0.86%`** <br> `-0.92%` |
79
+ | **ARC-Challenge** | `acc` <br> `acc_norm` | **`22.90%`** <br> **`25.90%`** | `22.35%` <br> `25.60%` | `-0.55%` <br> `-0.30%` |
80
 
81
+ *\*Supra-1.5-Instruct (Baseline) metrics are sourced directly from the official [SupraLabs/Supra-1.5-50M-Instruct-exp](https://huggingface.co/SupraLabs/Supra-1.5-50M-Instruct-exp) repository.*
 
 
 
 
 
82
 
83
+ > **💡 Benchmarking Notes & Parity Disclaimer:**
84
+ > These evaluation scores are measured using standard benchmarking configurations. While the evaluation harness is mathematically standardized, minor variations in output can occur based on localized system setups, exact prompt formatting, and tokenizer defaults. These results are shared to provide an objective relative comparison between the model iterations under identical benchmarking conditions.
85
+ >
86
  ## Intended Use & Chat Template
87
 
88
  This model is intended for experimental research, lightweight conversational prototyping, and low-latency edge deployment. It is formatted natively to understand standard **ChatML** syntax: