File size: 2,795 Bytes
7061935 29d0271 7061935 29d0271 7061935 29d0271 7242ce1 29d0271 7061935 29d0271 7061935 29d0271 7061935 7c2a22e 7061935 e6ca410 7061935 03f0df2 7061935 03f0df2 5037a2f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 | ---
license: apache-2.0
language:
- ru
- en
tags:
- tokenizer
- bpe
- byte-level
- russian
- english
- emoji
---
# ARM Tokenizer
A byte-level BPE tokenizer built by **NeoneAI** for Russian, English, and emoji.
Part of the ARM research project.
## Overview
- **Vocab size:** 65 536
- **Algorithm:** Byte-level BPE
- **Languages:** Russian + English
- **Emoji:** supported
- **Special tokens:** `<|endoftext|>`, `<|pad|>`, `<|user|>`, `<|assistant|>`, `<|system|>`, `<|bos|>`, `<|eos|>`
- **Tool:** Hugging Face Tokenizers
## Why
Most open tokenizers (GPT-2, LLaMA, Mistral) are English-first.
ARM Tokenizer is built for **Russian + English + emoji**.
It compresses Russian text **3.3× better** than GPT-2
and beats Gemma-2 (256k vocab) with only a 65k vocabulary.
## Benchmarks
All benchmarks use **1000 randomly sampled real-world texts**
(500 Russian + 500 English) with fixed seed (`42`)
Margin of error: **±1%**.
### vs GPT-2 (50k)
| Metric | ARM | GPT-2 |
|---|---|---|
| Total tokens | **153 473** | 414 971 |
| Avg tok/char | **0.2965** | 0.8018 |
| Russian (500) | **111 803** | 371 960 |
| English (500) | **41 670** | 43 011 |
| Wins | **928** | 16 |
| Ties | 56 | |
**Result:** ARM is **63.02% ±1% more efficient** than GPT-2.
On Russian, ARM is **3.3× more efficient** than GPT-2.
### vs Gemma-2 (256k)
| Metric | ARM | Gemma-2 |
|---|---|---|
| Total tokens | **153 473** | 167 105 |
| Avg tok/char | **0.2965** | 0.3198 |
| Russian (500) | **111 803** | 126 941 |
| English (500) | 41 670 | **40 164** |
| Wins | **791** | 108 |
| Ties | 101 | |
**Result:** ARM is **8.16% ±1% more efficient** than Gemma-2 overall,
and **12% ±1% more efficient on Russian**, with only a 65k vocabulary.
All benchmarks are fully reproducible. Two test scripts are included:
`test_gpt-2.py` compares ARM Tokenizer with GPT-2,
and `test_gemma-2.py` compares ARM Tokenizer with Gemma-2.
Both scripts read from the `sample/` folder:
`sample/gutenberg_sample.txt` contains 5000 English texts from Project Gutenberg,
and `sample/ru_books_sample.txt` contains 5000 Russian texts from RuHeritage-Corpus.
To generate the sample files, run `build_test_data.py`.
To run the tests, run `test_gpt-2.py` and `test_gemma-2.py`.
The seed is fixed (`42`) for reproducibility, so results should match
the tables above within the ±1% margin of error.
## Usage
```python
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
tokenizer_path = hf_hub_download(
repo_id="andrey-neoneai/arm-tokenizer",
filename="arm_tokenizer/tokenizer.json"
)
tokenizer = Tokenizer.from_file(tokenizer_path)
encoded = tokenizer.encode("Привет, мир! 😊")
print(encoded.tokens)
print(encoded.ids)
decoded = tokenizer.decode(encoded.ids)
print(decoded) |