ARM Tokenizer
A byte-level BPE tokenizer built by NeoneAI for Russian, English, and emoji.
Part of the ARM research project.
Overview
- Vocab size: 65 536
- Algorithm: Byte-level BPE
- Languages: Russian + English
- Emoji: supported
- Special tokens:
<|endoftext|>,<|pad|>,<|user|>,<|assistant|>,<|system|>,<|bos|>,<|eos|> - Tool: Hugging Face Tokenizers
Why
Most open tokenizers (GPT-2, LLaMA, Mistral) are English-first. ARM Tokenizer is built for Russian + English + emoji.
It compresses Russian text 3.3× better than GPT-2 and beats Gemma-2 (256k vocab) with only a 65k vocabulary.
Benchmarks
All benchmarks use 1000 randomly sampled real-world texts
(500 Russian + 500 English) with fixed seed (42)
Margin of error: ±1%.
vs GPT-2 (50k)
| Metric | ARM | GPT-2 |
|---|---|---|
| Total tokens | 153 473 | 414 971 |
| Avg tok/char | 0.2965 | 0.8018 |
| Russian (500) | 111 803 | 371 960 |
| English (500) | 41 670 | 43 011 |
| Wins | 928 | 16 |
| Ties | 56 |
Result: ARM is 63.02% ±1% more efficient than GPT-2. On Russian, ARM is 3.3× more efficient than GPT-2.
vs Gemma-2 (256k)
| Metric | ARM | Gemma-2 |
|---|---|---|
| Total tokens | 153 473 | 167 105 |
| Avg tok/char | 0.2965 | 0.3198 |
| Russian (500) | 111 803 | 126 941 |
| English (500) | 41 670 | 40 164 |
| Wins | 791 | 108 |
| Ties | 101 |
Result: ARM is 8.16% ±1% more efficient than Gemma-2 overall, and 12% ±1% more efficient on Russian, with only a 65k vocabulary.
All benchmarks are fully reproducible. Two test scripts are included:
test_gpt-2.py compares ARM Tokenizer with GPT-2,
and test_gemma-2.py compares ARM Tokenizer with Gemma-2.
Both scripts read from the sample/ folder:
sample/gutenberg_sample.txt contains 5000 English texts from Project Gutenberg,
and sample/ru_books_sample.txt contains 5000 Russian texts from RuHeritage-Corpus.
To generate the sample files, run build_test_data.py.
To run the tests, run test_gpt-2.py and test_gemma-2.py.
The seed is fixed (42) for reproducibility, so results should match
the tables above within the ±1% margin of error.
Usage
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
tokenizer_path = hf_hub_download(
repo_id="andrey-neoneai/arm-tokenizer",
filename="arm_tokenizer/tokenizer.json"
)
tokenizer = Tokenizer.from_file(tokenizer_path)
encoded = tokenizer.encode("Привет, мир! 😊")
print(encoded.tokens)
print(encoded.ids)
decoded = tokenizer.decode(encoded.ids)
print(decoded)