---
license: apache-2.0
language:
  - ru
  - en
tags:
  - tokenizer
  - bpe
  - byte-level
  - russian
  - english
  - emoji
---

# ARM Tokenizer

A byte-level BPE tokenizer built by **NeoneAI** for Russian, English, and emoji.

Part of the ARM research project.

## Overview

- **Vocab size:** 65 536
- **Algorithm:** Byte-level BPE
- **Languages:** Russian + English
- **Emoji:** supported
- **Special tokens:** `<|endoftext|>`, `<|pad|>`, `<|user|>`, `<|assistant|>`, `<|system|>`, `<|bos|>`, `<|eos|>`
- **Tool:** Hugging Face Tokenizers

## Why

Most open tokenizers (GPT-2, LLaMA, Mistral) are English-first.
ARM Tokenizer is built for **Russian + English + emoji**.

It compresses Russian text **3.3× better** than GPT-2
and beats Gemma-2 (256k vocab) with only a 65k vocabulary.

## Benchmarks

All benchmarks use **1000 randomly sampled real-world texts**
(500 Russian + 500 English) with fixed seed (`42`)
Margin of error: **±1%**.

### vs GPT-2 (50k)

| Metric | ARM | GPT-2 |
|---|---|---|
| Total tokens | **153 473** | 414 971 |
| Avg tok/char | **0.2965** | 0.8018 |
| Russian (500) | **111 803** | 371 960 |
| English (500) | **41 670** | 43 011 |
| Wins | **928** | 16 |
| Ties | 56 | |

**Result:** ARM is **63.02% ±1% more efficient** than GPT-2.
On Russian, ARM is **3.3× more efficient** than GPT-2.

### vs Gemma-2 (256k)

| Metric | ARM | Gemma-2 |
|---|---|---|
| Total tokens | **153 473** | 167 105 |
| Avg tok/char | **0.2965** | 0.3198 |
| Russian (500) | **111 803** | 126 941 |
| English (500) | 41 670 | **40 164** |
| Wins | **791** | 108 |
| Ties | 101 | |

**Result:** ARM is **8.16% ±1% more efficient** than Gemma-2 overall,
and **12% ±1% more efficient on Russian**, with only a 65k vocabulary.

All benchmarks are fully reproducible. Two test scripts are included:
`test_gpt-2.py` compares ARM Tokenizer with GPT-2,
and `test_gemma-2.py` compares ARM Tokenizer with Gemma-2.

Both scripts read from the `sample/` folder:
`sample/gutenberg_sample.txt` contains 5000 English texts from Project Gutenberg,
and `sample/ru_books_sample.txt` contains 5000 Russian texts from RuHeritage-Corpus.

To generate the sample files, run `build_test_data.py`.
To run the tests, run `test_gpt-2.py` and `test_gemma-2.py`.
The seed is fixed (`42`) for reproducibility, so results should match
the tables above within the ±1% margin of error.

## Usage

```python
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

tokenizer_path = hf_hub_download(
    repo_id="andrey-neoneai/arm-tokenizer",
    filename="arm_tokenizer/tokenizer.json"
)
tokenizer = Tokenizer.from_file(tokenizer_path)

encoded = tokenizer.encode("Привет, мир! 😊")
print(encoded.tokens)
print(encoded.ids)

decoded = tokenizer.decode(encoded.ids)
print(decoded)