| ---
|
| license: apache-2.0
|
| language:
|
| - ru
|
| - en
|
| tags:
|
| - tokenizer
|
| - bpe
|
| - byte-level
|
| - russian
|
| - english
|
| - emoji
|
| ---
|
|
|
| # ARM Tokenizer
|
|
|
| A byte-level BPE tokenizer built by **NeoneAI** for Russian, English, and emoji.
|
|
|
| Part of the ARM research project.
|
|
|
| ## Overview
|
|
|
| - **Vocab size:** 65 536
|
| - **Algorithm:** Byte-level BPE
|
| - **Languages:** Russian + English
|
| - **Emoji:** supported
|
| - **Special tokens:** `<|endoftext|>`, `<|pad|>`, `<|user|>`, `<|assistant|>`, `<|system|>`, `<|bos|>`, `<|eos|>`
|
| - **Tool:** Hugging Face Tokenizers
|
|
|
| ## Why
|
|
|
| Most open tokenizers (GPT-2, LLaMA, Mistral) are English-first.
|
| ARM Tokenizer is built for **Russian + English + emoji**.
|
|
|
| It compresses Russian text **3.3× better** than GPT-2
|
| and beats Gemma-2 (256k vocab) with only a 65k vocabulary.
|
|
|
| ## Benchmarks
|
|
|
| All benchmarks use **1000 randomly sampled real-world texts**
|
| (500 Russian + 500 English) with fixed seed (`42`).
|
| Margin of error: **±1%**.
|
|
|
| ### vs GPT-2 (50k)
|
|
|
| | Metric | ARM | GPT-2 |
|
| |---|---|---|
|
| | Total tokens | **153 473** | 414 971 |
|
| | Avg tok/char | **0.2965** | 0.8018 |
|
| | Russian (500) | **111 803** | 371 960 |
|
| | English (500) | **41 670** | 43 011 |
|
| | Wins | **928** | 16 |
|
| | Ties | 56 | |
|
|
|
| **Result:** ARM is **63.02% ±1% more efficient** than GPT-2.
|
| On Russian, ARM is **3.3× more efficient** than GPT-2.
|
|
|
| ### vs Gemma-2 (256k)
|
|
|
| | Metric | ARM | Gemma-2 |
|
| |---|---|---|
|
| | Total tokens | **153 473** | 167 105 |
|
| | Avg tok/char | **0.2965** | 0.3198 |
|
| | Russian (500) | **111 803** | 126 941 |
|
| | English (500) | 41 670 | **40 164** |
|
| | Wins | **791** | 108 |
|
| | Ties | 101 | |
|
|
|
| **Result:** ARM is **8.16% ±1% more efficient** than Gemma-2 overall,
|
| and **12% ±1% more efficient on Russian**, with only a 65k vocabulary.
|
|
|
| ## Reproducing the benchmarks
|
|
|
| Two test scripts are included:
|
|
|
| - **`test_gpt-2.py`** — compares ARM Tokenizer with GPT-2 Tokenizer
|
| - **`test_gemma-2.py`** — compares ARM Tokenizer with Gemma-2 Tokenizer
|
|
|
| Both scripts read from `sample/` folder:
|
|
|
| - `sample/gutenberg_sample.txt` — 5000 English texts from Project Gutenberg
|
| - `sample/ru_books_sample.txt` — 5000 Russian texts from RuHeritage-Corpus
|
|
|
| ### Generate test data
|
|
|
| ```bash
|
| python build_test_data.py |
