README.md

---
license: other
library_name: transformers
tags:
  - tokenizer
  - byte-level-bpe
  - traditional-chinese
  - taiwan
  - multilingual
---

# PangolinTokenizer

Byte-level BPE tokenizer trained for Traditional Chinese, Taiwan text, multilingual
text, rich transcription, OCR-style text, and generic multimodal control formats.

## Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "voidful/PangolinTokenizer",
    trust_remote_code=False,
)

text = "<|system|>台灣健保與注音ㄅㄆㄇ，Tailo: Tâi-uân"
ids = tokenizer.encode(text)
decoded = tokenizer.decode(ids)
```

## Files

- `config.json`
- `tokenizer_config.json`
- `tokenizer.json`
- `vocab.json`
- `merges.txt`
- `special_tokens_map.json`
- `evaluation_report.json`

## Tokenizer Details

- Type: Byte-level BPE
- Vocabulary size: 114,688
- Learned merges: 114,397
- Model max length metadata: 131,072
- Minimum merge frequency: 5
- Transformers class: `GPT2TokenizerFast`
- `trust_remote_code`: not required

## Safety Notes

This tokenizer intentionally does not include discrete audio codec token ranges.
It also intentionally does not include dense timestamp token ranges. Audio should
be represented through external references or embeddings outside this tokenizer.

Evaluation checks confirmed:

- Transformers `AutoTokenizer` loading works with `trust_remote_code=False`
- Traditional Chinese and Bopomofo smoke roundtrip works
- No tokens matching `<|audio_[0-9]+|>`
- No dense timestamp token ranges matching `<|ts_[0-9]+|>`,
  `<|timestamp_[0-9]+|>`, or `<|time_[0-9]+|>`