zyoralabs commited on
Commit
b2fcc53
·
verified ·
1 Parent(s): fb3996e

docs: pin revision in usage, v2 benchmark table + reproduction section

Browse files
Files changed (1) hide show
  1. README.md +32 -8
README.md CHANGED
@@ -40,8 +40,13 @@ This is the **tutor (instruct) model**: it answers student questions directly wi
40
  ```python
41
  from transformers import AutoModelForCausalLM, AutoTokenizer
42
 
43
- tok = AutoTokenizer.from_pretrained("zyoralabs/AQ-academic-ai")
44
- model = AutoModelForCausalLM.from_pretrained("zyoralabs/AQ-academic-ai", trust_remote_code=True)
 
 
 
 
 
45
 
46
  prompt = "### Student:\nWhat is a stack in data structures?\n\n### Tutor:\n"
47
  ids = tok(prompt, return_tensors="pt").input_ids
@@ -50,6 +55,8 @@ out = model.generate(ids, max_new_tokens=200, do_sample=True,
50
  print(tok.decode(out[0][ids.shape[1]:]))
51
  ```
52
 
 
 
53
  ## Architecture (proprietary, from scratch)
54
 
55
  | | |
@@ -66,15 +73,32 @@ print(tok.decode(out[0][ids.shape[1]:]))
66
 
67
  ## Benchmarks (0-shot, lm-evaluation-harness)
68
 
69
- | Benchmark | AQ v1 Tutor | Notes |
70
  |---|---|---|
71
- | SciQ | 69.2 | strong science knowledge for the size/data budget |
72
- | PIQA | 62.3 | |
73
- | ARC-easy | 45.1 | |
74
- | Winogrande | 49.5 | |
75
  | HellaSwag | 29.6 | |
76
  | MMLU | 25.2 | at-chance, like all ~1B-class models |
77
- | ARC-challenge | 21.2 | |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
78
 
79
  For context: models of this size trained on **15×–150× more tokens** (e.g. 300B–3T) reach SciQ ~84–89. AQ reaches ~70 on just **20B tokens** — the concept-first, knowledge-dense corpus is the point.
80
 
 
40
  ```python
41
  from transformers import AutoModelForCausalLM, AutoTokenizer
42
 
43
+ # Pin a released revision so the loaded code + weights are immutable
44
+ # (trust_remote_code executes this repo's modeling files).
45
+ REV = "v2.0"
46
+
47
+ tok = AutoTokenizer.from_pretrained("zyoralabs/AQ-academic-ai", revision=REV)
48
+ model = AutoModelForCausalLM.from_pretrained(
49
+ "zyoralabs/AQ-academic-ai", revision=REV, trust_remote_code=True)
50
 
51
  prompt = "### Student:\nWhat is a stack in data structures?\n\n### Tutor:\n"
52
  ids = tok(prompt, return_tensors="pt").input_ids
 
55
  print(tok.decode(out[0][ids.shape[1]:]))
56
  ```
57
 
58
+ Tested environment: `transformers==4.53.0`, `torch==2.7.1`, `tokenizers==0.21`, dtype `bfloat16`, single GPU or CPU.
59
+
60
  ## Architecture (proprietary, from scratch)
61
 
62
  | | |
 
73
 
74
  ## Benchmarks (0-shot, lm-evaluation-harness)
75
 
76
+ | Benchmark | AQ v2 Tutor | Notes |
77
  |---|---|---|
78
+ | SciQ | 70.0 | strong science knowledge for the size/data budget |
79
+ | PIQA | 61.9 | |
80
+ | ARC-easy | 45.2 | |
81
+ | Winogrande | 50.1 | |
82
  | HellaSwag | 29.6 | |
83
  | MMLU | 25.2 | at-chance, like all ~1B-class models |
84
+ | ARC-challenge | 20.8 | |
85
+
86
+ ### Reproducing these numbers
87
+
88
+ Scores were produced with **lm-evaluation-harness 0.4.8** (`pip install lm-eval==0.4.8`),
89
+ 0-shot, default task configs, on a single H100 (bf16):
90
+
91
+ ```bash
92
+ lm_eval --model hf \
93
+ --model_args pretrained=zyoralabs/AQ-academic-ai,revision=v2.0,trust_remote_code=True,dtype=bfloat16 \
94
+ --tasks mmlu,arc_easy,arc_challenge,hellaswag,piqa,winogrande,sciq \
95
+ --batch_size auto
96
+ ```
97
+
98
+ Environment: `transformers==4.53.0`, `torch==2.7.1` (cu128), `datasets==3.2.0`, `accelerate`.
99
+ The machine-readable harness output is published in this repo at
100
+ [`eval/aq_v2_tutor_results.json`](./eval/aq_v2_tutor_results.json).
101
+ Reported metric is `acc` (see the artifact for `acc_norm` and per-subtask MMLU results).
102
 
103
  For context: models of this size trained on **15×–150× more tokens** (e.g. 300B–3T) reach SciQ ~84–89. AQ reaches ~70 on just **20B tokens** — the concept-first, knowledge-dense corpus is the point.
104