docs: pin revision in usage, v2 benchmark table + reproduction section
Browse files
README.md
CHANGED
|
@@ -40,8 +40,13 @@ This is the **tutor (instruct) model**: it answers student questions directly wi
|
|
| 40 |
```python
|
| 41 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 42 |
|
| 43 |
-
|
| 44 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
|
| 46 |
prompt = "### Student:\nWhat is a stack in data structures?\n\n### Tutor:\n"
|
| 47 |
ids = tok(prompt, return_tensors="pt").input_ids
|
|
@@ -50,6 +55,8 @@ out = model.generate(ids, max_new_tokens=200, do_sample=True,
|
|
| 50 |
print(tok.decode(out[0][ids.shape[1]:]))
|
| 51 |
```
|
| 52 |
|
|
|
|
|
|
|
| 53 |
## Architecture (proprietary, from scratch)
|
| 54 |
|
| 55 |
| | |
|
|
@@ -66,15 +73,32 @@ print(tok.decode(out[0][ids.shape[1]:]))
|
|
| 66 |
|
| 67 |
## Benchmarks (0-shot, lm-evaluation-harness)
|
| 68 |
|
| 69 |
-
| Benchmark | AQ
|
| 70 |
|---|---|---|
|
| 71 |
-
| SciQ |
|
| 72 |
-
| PIQA |
|
| 73 |
-
| ARC-easy | 45.
|
| 74 |
-
| Winogrande |
|
| 75 |
| HellaSwag | 29.6 | |
|
| 76 |
| MMLU | 25.2 | at-chance, like all ~1B-class models |
|
| 77 |
-
| ARC-challenge |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
For context: models of this size trained on **15×–150× more tokens** (e.g. 300B–3T) reach SciQ ~84–89. AQ reaches ~70 on just **20B tokens** — the concept-first, knowledge-dense corpus is the point.
|
| 80 |
|
|
|
|
| 40 |
```python
|
| 41 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 42 |
|
| 43 |
+
# Pin a released revision so the loaded code + weights are immutable
|
| 44 |
+
# (trust_remote_code executes this repo's modeling files).
|
| 45 |
+
REV = "v2.0"
|
| 46 |
+
|
| 47 |
+
tok = AutoTokenizer.from_pretrained("zyoralabs/AQ-academic-ai", revision=REV)
|
| 48 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 49 |
+
"zyoralabs/AQ-academic-ai", revision=REV, trust_remote_code=True)
|
| 50 |
|
| 51 |
prompt = "### Student:\nWhat is a stack in data structures?\n\n### Tutor:\n"
|
| 52 |
ids = tok(prompt, return_tensors="pt").input_ids
|
|
|
|
| 55 |
print(tok.decode(out[0][ids.shape[1]:]))
|
| 56 |
```
|
| 57 |
|
| 58 |
+
Tested environment: `transformers==4.53.0`, `torch==2.7.1`, `tokenizers==0.21`, dtype `bfloat16`, single GPU or CPU.
|
| 59 |
+
|
| 60 |
## Architecture (proprietary, from scratch)
|
| 61 |
|
| 62 |
| | |
|
|
|
|
| 73 |
|
| 74 |
## Benchmarks (0-shot, lm-evaluation-harness)
|
| 75 |
|
| 76 |
+
| Benchmark | AQ v2 Tutor | Notes |
|
| 77 |
|---|---|---|
|
| 78 |
+
| SciQ | 70.0 | strong science knowledge for the size/data budget |
|
| 79 |
+
| PIQA | 61.9 | |
|
| 80 |
+
| ARC-easy | 45.2 | |
|
| 81 |
+
| Winogrande | 50.1 | |
|
| 82 |
| HellaSwag | 29.6 | |
|
| 83 |
| MMLU | 25.2 | at-chance, like all ~1B-class models |
|
| 84 |
+
| ARC-challenge | 20.8 | |
|
| 85 |
+
|
| 86 |
+
### Reproducing these numbers
|
| 87 |
+
|
| 88 |
+
Scores were produced with **lm-evaluation-harness 0.4.8** (`pip install lm-eval==0.4.8`),
|
| 89 |
+
0-shot, default task configs, on a single H100 (bf16):
|
| 90 |
+
|
| 91 |
+
```bash
|
| 92 |
+
lm_eval --model hf \
|
| 93 |
+
--model_args pretrained=zyoralabs/AQ-academic-ai,revision=v2.0,trust_remote_code=True,dtype=bfloat16 \
|
| 94 |
+
--tasks mmlu,arc_easy,arc_challenge,hellaswag,piqa,winogrande,sciq \
|
| 95 |
+
--batch_size auto
|
| 96 |
+
```
|
| 97 |
+
|
| 98 |
+
Environment: `transformers==4.53.0`, `torch==2.7.1` (cu128), `datasets==3.2.0`, `accelerate`.
|
| 99 |
+
The machine-readable harness output is published in this repo at
|
| 100 |
+
[`eval/aq_v2_tutor_results.json`](./eval/aq_v2_tutor_results.json).
|
| 101 |
+
Reported metric is `acc` (see the artifact for `acc_norm` and per-subtask MMLU results).
|
| 102 |
|
| 103 |
For context: models of this size trained on **15×–150× more tokens** (e.g. 300B–3T) reach SciQ ~84–89. AQ reaches ~70 on just **20B tokens** — the concept-first, knowledge-dense corpus is the point.
|
| 104 |
|