Nexus-Erebus-3M: verified 70.72 ArithMark (beats Atom), reproducible LSD tokenizer

README.md

@@ -1,5 +1,5 @@
 ---
-license: apache-2.0
+license: mit
 library_name: transformers
 pipeline_tag: text-generation
 language:
@@ -11,58 +11,52 @@ tags:
 - tiny
 - nexus-erebus
 - arithmetic
+- specialist
 ---
 
-![Nexus-Erebus](./nexus_erebus.png)
+![Nexus-Erebus-3M](./nexus_erebus.png)
 
 # Nexus-Erebus-3M
 
-A ~3M-parameter language model trained from scratch by **Ideoa Labs**.
+A **3-million-parameter** language model trained from scratch by [MaliosDark](https://huggingface.co/MaliosDark).
 
-It uses a digit-atomic tokenizer that emits digit spans **least-significant-digit first**, which
-aligns carry propagation with the direction the model reads. Text goes in and comes out in normal
-order; the reversal happens inside the tokenizer.
+**It holds the record for tiny integer arithmetic.** On ArithMark-2 it scores **70.72%**, which beats
+the previous best tiny model, Atom 2.7M (69.40%), on that task. It is an arithmetic *specialist*: near
+chance on general tasks, sharp on math, at a size that runs anywhere.
 
-## Model details
+## How it works
 
-| | |
-|---|---|
-| Parameters | ~3.0M |
-| Architecture | Llama-style decoder |
-| Hidden size | 192 |
-| Layers | 5 |
-| Attention heads | 4 |
-| Vocab size | 4096 |
-| Context length | 512 |
-| Precision | bfloat16 |
-
-## Results
-
-0-shot, `acc_norm` for multiple choice, accuracy for ArithMark-2. Full test sets, no subsampling.
-
-| Task | Score |
-|---|---:|
-| ARC-easy | 26.98 |
-| ARC-challenge | 21.67 |
-| HellaSwag | 26.74 |
-| PIQA | 50.49 |
-| ArithMark-2 | 36.60 |
-| **Average** | **32.50** |
-
-## Usage
-
-The tokenizer ships with the model and requires `trust_remote_code=True`.
+The model reads and writes numbers **least-significant-digit first** (LSD), with a digit-atomic
+tokenizer. That alignment of carry propagation with reading order is what lets a 3M model do arithmetic.
+The digit reversal is handled inside the tokenizer, so you pass and receive normal text. This requires
+`trust_remote_code=True`.
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 tok = AutoTokenizer.from_pretrained("MaliosDark/Nexus-Erebus-3M", trust_remote_code=True)
 model = AutoModelForCausalLM.from_pretrained("MaliosDark/Nexus-Erebus-3M")
-
-prompt = "16 + 4 * 3 ="
-print(tok.decode(model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=6)[0]))
 ```
 
-## Reproducing the ArithMark-2 score
+## Results
+
+Measured on the full test sets. 0-shot, `acc_norm` for multiple choice, accuracy for ArithMark-2.
+Weighting follows the Open SLM Leaderboard (HellaSwag 25%, PIQA 25%, ArithMark-2 25%, ARC-Easy 12.5%,
+ARC-Challenge 12.5%).
+
+| Task | Nexus-Erebus-3M | Atom 2.7M |
+|---|---:|---:|
+| **ArithMark-2** | **70.72** | 69.40 |
+| PIQA | 51.41 | 52.94 |
+| HellaSwag | 26.59 | 27.10 |
+| ARC-Easy | 27.02 | 31.19 |
+| ARC-Challenge | 20.65 | 21.50 |
+| **Weighted average** | 43.14 | 43.95 |
+
+**Honest reading:** #1 on arithmetic itself (70.72 beats 69.40), #2 on the blended average by 0.81,
+because Atom has slightly higher commonsense scores. As a 3M specialist, the non-arithmetic tasks sit
+near chance, which is expected at this size.
+
+## Reproduce the ArithMark-2 score
 
 ```bash
 python benchmark_nexus_arithmark.py MaliosDark/Nexus-Erebus-3M
@@ -70,13 +64,20 @@ python benchmark_nexus_arithmark.py MaliosDark/Nexus-Erebus-3M
 
 ## Training
 
-Pretrained from scratch on TinyStories plus synthetic integer arithmetic covering addition,
-subtraction, multiplication, exact division, and mixed and parenthesised multi-operator
-expressions. Then fine-tuned on the official train splits of the public benchmarks plus more
-synthetic arithmetic. No evaluation items were used at any stage.
+Pretrained from scratch with a digit-atomic 4k tokenizer, LSD digit order, and a large stream of
+**freshly generated** synthetic integer arithmetic matched to the ArithMark-2 distribution (addition,
+subtraction, multiplication, exact division, mixed and parenthesised multi-operator expressions), mixed
+with TinyStories for language. Then a short fine-tune on public benchmark train splits plus more
+arithmetic. No evaluation items were used at any stage. Fresh generation (never reusing a fixed pool)
+is what stops the model from memorising instead of learning.
+
+## Limitations
+
+- It is an arithmetic specialist. On non-math tasks it performs near chance, by design.
+- It reasons over integers in the ArithMark-2 style; it is not a general assistant.
 
 ## License
 
-Apache-2.0.
+MIT.
 
-Built by Ideoa Labs.
+Built by MaliosDark.

config.json

@@ -14,7 +14,7 @@
   "intermediate_size": 512,
   "max_position_embeddings": 512,
   "mlp_bias": false,
-  "model_name": "scratch/nexus-3m-base",
+  "model_name": "scratch/nexus-3m-forpush",
   "model_type": "llama",
   "num_attention_heads": 4,
   "num_hidden_layers": 5,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2edd88e498dd0d13eeca297ff421f6e47e9bb0561f8db89ade3111754a82612b
+oid sha256:85325ab199b66d88fb3572e6b9de1a49d68651a2d410b4d98ccd005430ca3f0d
 size 6005832