arm_tokenizer/README.md

metadata

license: apache-2.0
language:
  - ru
  - en
tags:
  - tokenizer
  - bpe
  - byte-level
  - russian
  - english
  - emoji

ARM Tokenizer

A byte-level BPE tokenizer built by NeoneAI for Russian, English, and emoji.

Part of the ARM research project.

Overview

• Vocab size: 65 536
• Algorithm: Byte-level BPE
• Languages: Russian + English
• Emoji: supported
• Special tokens: <|endoftext|>, <|pad|>, <|user|>, <|assistant|>, <|system|>, <|bos|>, <|eos|>
• Tool: Hugging Face Tokenizers

Why

Most open tokenizers (GPT-2, LLaMA, Mistral) are English-first. ARM Tokenizer is built for Russian + English + emoji.

It compresses Russian text 3.3× better than GPT-2 and beats Gemma-2 (256k vocab) with only a 65k vocabulary.

Benchmarks

All benchmarks use 1000 randomly sampled real-world texts (500 Russian + 500 English) with fixed seed (42). Margin of error: ±1%.

vs GPT-2 (50k)

Metric	ARM	GPT-2
Total tokens	153 473	414 971
Avg tok/char	0.2965	0.8018
Russian (500)	111 803	371 960
English (500)	41 670	43 011
Wins	928	16
Ties	56

Result: ARM is 63.02% ±1% more efficient than GPT-2. On Russian, ARM is 3.3× more efficient than GPT-2.

vs Gemma-2 (256k)

Metric	ARM	Gemma-2
Total tokens	153 473	167 105
Avg tok/char	0.2965	0.3198
Russian (500)	111 803	126 941
English (500)	41 670	40 164
Wins	791	108
Ties	101

Result: ARM is 8.16% ±1% more efficient than Gemma-2 overall, and 12% ±1% more efficient on Russian, with only a 65k vocabulary.

Reproducing the benchmarks

Two test scripts are included:

• test_gpt-2.py — compares ARM Tokenizer with GPT-2 Tokenizer
• test_gemma-2.py — compares ARM Tokenizer with Gemma-2 Tokenizer

Both scripts read from sample/ folder:

• sample/gutenberg_sample.txt — 5000 English texts from Project Gutenberg
• sample/ru_books_sample.txt — 5000 Russian texts from RuHeritage-Corpus

Generate test data

python build_test_data.py