---
license: mit
library_name: transformers
pipeline_tag: text-generation
language:
- en
tags:
- small-language-model
- slm
- from-scratch
- tiny
- nexus-erebus
- arithmetic
---

![Nexus-Erebus-3M](./nexus_erebus.png)

# Nexus-Erebus-3M

A **3-million-parameter** language model trained from scratch. It is an arithmetic specialist: it
handles integer arithmetic well for its size, and stays near chance on general tasks, at a footprint
that runs almost anywhere.

## How it works

The model reads and writes numbers **least-significant-digit first**, with a digit-atomic tokenizer.
Aligning carry propagation with reading order is what lets a model this small do arithmetic. The digit
reversal happens inside the tokenizer, so you pass and receive normal text. This requires
`trust_remote_code=True`.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("MaliosDark/Nexus-Erebus-3M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("MaliosDark/Nexus-Erebus-3M")
```

## Results

Full test sets, 0-shot. `acc_norm` for multiple choice, accuracy for ArithMark-2.

| Task | Score |
|---|---:|
| ArithMark-2 | 70.72 |
| PIQA | 51.41 |
| HellaSwag | 26.59 |
| ARC-Easy | 27.02 |
| ARC-Challenge | 20.65 |

The model is built for integer arithmetic. Non-arithmetic tasks sit near chance, which is expected at
this size.

## Reproduce the ArithMark-2 score

```bash
python benchmark_nexus_arithmark.py MaliosDark/Nexus-Erebus-3M
```

## Training

Pretrained from scratch with a digit-atomic 4k tokenizer, least-significant-digit order, and a large
stream of **freshly generated** synthetic integer arithmetic (addition, subtraction, multiplication,
exact division, mixed and parenthesised multi-operator expressions), mixed with TinyStories for
language. Then a short fine-tune on public benchmark train splits plus more arithmetic. No evaluation
items were used at any stage. Fresh generation, never reusing a fixed pool, is what stops the model
from memorising instead of learning.

## Limitations

- It is an arithmetic specialist. On non-math tasks it performs near chance, by design.
- It reasons over integers in the ArithMark-2 style; it is not a general assistant.

## License

MIT. Trained from scratch: 100% original weights, own tokenizer, no base model.