README.md

metadata

license: other
library_name: transformers
tags:
  - tokenizer
  - byte-level-bpe
  - traditional-chinese
  - taiwan
  - multilingual

PangolinTokenizer

Byte-level BPE tokenizer trained for Traditional Chinese, Taiwan text, multilingual text, rich transcription, OCR-style text, and generic multimodal control formats.

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "voidful/PangolinTokenizer",
    trust_remote_code=False,
)

text = "<|system|>台灣健保與注音ㄅㄆㄇ，Tailo: Tâi-uân"
ids = tokenizer.encode(text)
decoded = tokenizer.decode(ids)

Files

• config.json
• tokenizer_config.json
• tokenizer.json
• vocab.json
• merges.txt
• special_tokens_map.json
• evaluation_report.json

Tokenizer Details

• Type: Byte-level BPE
• Vocabulary size: 114,688
• Learned merges: 114,397
• Model max length metadata: 131,072
• Minimum merge frequency: 5
• Transformers class: GPT2TokenizerFast
• trust_remote_code: not required

Safety Notes

This tokenizer intentionally does not include discrete audio codec token ranges. It also intentionally does not include dense timestamp token ranges. Audio should be represented through external references or embeddings outside this tokenizer.

Evaluation checks confirmed:

• Transformers AutoTokenizer loading works with trust_remote_code=False
• Traditional Chinese and Bopomofo smoke roundtrip works
• No tokens matching <|audio_[0-9]+|>
• No dense timestamp token ranges matching <|ts_[0-9]+|>, <|timestamp_[0-9]+|>, or <|time_[0-9]+|>