---
license: apache-2.0
language:
  - ru
  - en
tags:
  - tokenizer
  - bpe
  - byte-level
  - russian
  - english
  - emoji
---

# ARM Tokenizer

A byte-level BPE tokenizer built by **NeoneAI** for Russian, English, and emoji.

Part of the ARM research project.

## Overview

- **Vocab size:** 65 536
- **Algorithm:** Byte-level BPE
- **Languages:** Russian + English
- **Emoji:** supported
- **Special tokens:** `<|endoftext|>`, `<|pad|>`, `<|user|>`, `<|assistant|>`, `<|system|>`, `<|bos|>`, `<|eos|>`
- **Tool:** Hugging Face Tokenizers

## Why

Most open tokenizers (GPT-2, LLaMA, Mistral) are English-first.
ARM Tokenizer is built for **Russian + English + emoji**.

It compresses Russian text **3.3× better** than GPT-2
and beats Gemma-2 (256k vocab) with only a 65k vocabulary.

## Benchmarks

All benchmarks use **1000 randomly sampled real-world texts**
(500 Russian + 500 English) with fixed seed (`42`).
Margin of error: **±1%**.

### vs GPT-2 (50k)

| Metric | ARM | GPT-2 |
|---|---|---|
| Total tokens | **153 473** | 414 971 |
| Avg tok/char | **0.2965** | 0.8018 |
| Russian (500) | **111 803** | 371 960 |
| English (500) | **41 670** | 43 011 |
| Wins | **928** | 16 |
| Ties | 56 | |

**Result:** ARM is **63.02% ±1% more efficient** than GPT-2.
On Russian, ARM is **3.3× more efficient** than GPT-2.

### vs Gemma-2 (256k)

| Metric | ARM | Gemma-2 |
|---|---|---|
| Total tokens | **153 473** | 167 105 |
| Avg tok/char | **0.2965** | 0.3198 |
| Russian (500) | **111 803** | 126 941 |
| English (500) | 41 670 | **40 164** |
| Wins | **791** | 108 |
| Ties | 101 | |

**Result:** ARM is **8.16% ±1% more efficient** than Gemma-2 overall,
and **12% ±1% more efficient on Russian**, with only a 65k vocabulary.

## Reproducing the benchmarks

Two test scripts are included:

- **`test_gpt-2.py`** — compares ARM Tokenizer with GPT-2 Tokenizer
- **`test_gemma-2.py`** — compares ARM Tokenizer with Gemma-2 Tokenizer

Both scripts read from `sample/` folder:

- `sample/gutenberg_sample.txt` — 5000 English texts from Project Gutenberg
- `sample/ru_books_sample.txt` — 5000 Russian texts from RuHeritage-Corpus

### Generate test data

```bash
python build_test_data.py