Commit ·
93df154
1
Parent(s): 4384c77
Add pipeline tag, update library name and refine metadata (#1)
Browse files- Add pipeline tag, update library name and refine metadata (34a230b6e4e351586d1b02a20562b0546d60dbf9)
Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>
README.md
CHANGED
|
@@ -1,44 +1,44 @@
|
|
| 1 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
-
|
| 4 |
-
arxiv: 2605.29379
|
| 5 |
tags:
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
language:
|
| 23 |
-
- en
|
| 24 |
-
- hi
|
| 25 |
-
- bn
|
| 26 |
-
- ta
|
| 27 |
-
- te
|
| 28 |
-
- kn
|
| 29 |
-
- ml
|
| 30 |
-
- mr
|
| 31 |
-
- gu
|
| 32 |
-
- pa
|
| 33 |
-
- or
|
| 34 |
-
- as
|
| 35 |
---
|
| 36 |
|
| 37 |
# BrahmicTokenizer-131K
|
| 38 |
|
| 39 |
A 131,072-vocabulary byte-level BPE tokenizer that closes the Brahmic compression gap at the 131K-vocabulary class while preserving the English, EU-language, and code compression of OpenAI's o200k_base. Drop-in replacement for any o200k_base training pipeline: same byte-level BPE algorithm, same GPT-2 ByteLevel pre-tokenizer, same decoder, same vocabulary file format.
|
| 40 |
|
| 41 |
-
|
| 42 |
|
| 43 |
## Citation
|
| 44 |
|
|
@@ -123,4 +123,4 @@ The reproduction scripts (verification, fertility evaluation, 27M-corpus tokeniz
|
|
| 123 |
|
| 124 |
## License
|
| 125 |
|
| 126 |
-
Apache License 2.0. This work is a derivative of OpenAI's o200k_base tokenizer, released through the MIT-licensed [tiktoken](https://github.com/openai/tiktoken) repository; Apache 2.0 is compatible with incorporating MIT-licensed material. The bundled Brahmic-script fonts referenced in the paper (`NotoSansDevanagari`, `NotoSansBengali`, `NotoSansOriya`, `NotoSansTamil`) are redistributed under the SIL Open Font License 1.1.
|
|
|
|
| 1 |
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
- hi
|
| 5 |
+
- bn
|
| 6 |
+
- ta
|
| 7 |
+
- te
|
| 8 |
+
- kn
|
| 9 |
+
- ml
|
| 10 |
+
- mr
|
| 11 |
+
- gu
|
| 12 |
+
- pa
|
| 13 |
+
- or
|
| 14 |
+
- as
|
| 15 |
+
library_name: transformers
|
| 16 |
license: apache-2.0
|
| 17 |
+
pipeline_tag: feature-extraction
|
|
|
|
| 18 |
tags:
|
| 19 |
+
- tokenizer
|
| 20 |
+
- byte-level-bpe
|
| 21 |
+
- brahmic
|
| 22 |
+
- indic
|
| 23 |
+
- multilingual
|
| 24 |
+
- hindi
|
| 25 |
+
- bengali
|
| 26 |
+
- tamil
|
| 27 |
+
- telugu
|
| 28 |
+
- kannada
|
| 29 |
+
- malayalam
|
| 30 |
+
- marathi
|
| 31 |
+
- gujarati
|
| 32 |
+
- punjabi
|
| 33 |
+
- odia
|
| 34 |
+
- assamese
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
---
|
| 36 |
|
| 37 |
# BrahmicTokenizer-131K
|
| 38 |
|
| 39 |
A 131,072-vocabulary byte-level BPE tokenizer that closes the Brahmic compression gap at the 131K-vocabulary class while preserving the English, EU-language, and code compression of OpenAI's o200k_base. Drop-in replacement for any o200k_base training pipeline: same byte-level BPE algorithm, same GPT-2 ByteLevel pre-tokenizer, same decoder, same vocabulary file format.
|
| 40 |
|
| 41 |
+
The model was presented in the paper [BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base](https://arxiv.org/abs/2605.29379).
|
| 42 |
|
| 43 |
## Citation
|
| 44 |
|
|
|
|
| 123 |
|
| 124 |
## License
|
| 125 |
|
| 126 |
+
Apache License 2.0. This work is a derivative of OpenAI's o200k_base tokenizer, released through the MIT-licensed [tiktoken](https://github.com/openai/tiktoken) repository; Apache 2.0 is compatible with incorporating MIT-licensed material. The bundled Brahmic-script fonts referenced in the paper (`NotoSansDevanagari`, `NotoSansBengali`, `NotoSansOriya`, `NotoSansTamil`) are redistributed under the SIL Open Font License 1.1.
|