Hugging Face's logo

OiQ
/
daa-tokenizers

Arabic
Latin
tokenizer
moroccan-darija
arabic
bpe
unigram
wordpiece
bbpe
benchmark
Model card Files
xet
 Community
daa-tokenizers

Commit History

Upload transformers_tokenizers/concat_wordpiece_32000_tokenizer_az/tokenizer.json with huggingface_hub
8259f23
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_32000_tokenizer_az/tokenizer_config.json with huggingface_hub
6063aea
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_32000_tokenizer_az/special_tokens_map.json with huggingface_hub
867fda4
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_32000_tokenizer_ar/tokenizer.json with huggingface_hub
4b251d9
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_32000_tokenizer_ar/tokenizer_config.json with huggingface_hub
3f8f874
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_32000_tokenizer_ar/special_tokens_map.json with huggingface_hub
10d9159
verified

Ouaill commited on

Upload transformers_tokenizers/concat_unigram_32000_tokenizer_az/tokenizer.json with huggingface_hub
9dc8dbd
verified

Ouaill commited on

Upload transformers_tokenizers/concat_unigram_32000_tokenizer_az/tokenizer_config.json with huggingface_hub
ab6ab24
verified

Ouaill commited on

Upload transformers_tokenizers/concat_unigram_32000_tokenizer_az/special_tokens_map.json with huggingface_hub
84454db
verified

Ouaill commited on

Upload transformers_tokenizers/concat_unigram_32000_tokenizer_ar/tokenizer.json with huggingface_hub
d3984cd
verified

Ouaill commited on

Upload transformers_tokenizers/concat_unigram_32000_tokenizer_ar/tokenizer_config.json with huggingface_hub
c92ee3b
verified

Ouaill commited on

Upload transformers_tokenizers/concat_unigram_32000_tokenizer_ar/special_tokens_map.json with huggingface_hub
766372e
verified

Ouaill commited on

Upload transformers_tokenizers/concat_bpe_32000_tokenizer_az/tokenizer.json with huggingface_hub
7878194
verified

Ouaill commited on

Upload transformers_tokenizers/concat_bpe_32000_tokenizer_az/tokenizer_config.json with huggingface_hub
6adcd1b
verified

Ouaill commited on

Upload transformers_tokenizers/concat_bpe_32000_tokenizer_az/special_tokens_map.json with huggingface_hub
a660b7b
verified

Ouaill commited on

Upload transformers_tokenizers/concat_bpe_32000_tokenizer_ar/tokenizer.json with huggingface_hub
91f23a6
verified

Ouaill commited on

Upload transformers_tokenizers/concat_bpe_32000_tokenizer_ar/tokenizer_config.json with huggingface_hub
529fe3d
verified

Ouaill commited on

Upload transformers_tokenizers/concat_bpe_32000_tokenizer_ar/special_tokens_map.json with huggingface_hub
324d231
verified

Ouaill commited on

Upload compare_with_external.py with huggingface_hub
c754e45
verified

Ouaill commited on

Upload eval_and_compare.py with huggingface_hub
4927edf
verified

Ouaill commited on

Upload external_comparison.json with huggingface_hub
343bbb9
verified

Ouaill commited on

Upload README.md with huggingface_hub
b05b030
verified

Ouaill commited on

Upload external_comparison.csv with huggingface_hub
98c39de
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bpe_32000/tokenizer_config.json with huggingface_hub
9148049
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bpe_32000/special_tokens_map.json with huggingface_hub
114210b
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bpe_16000/tokenizer.json with huggingface_hub
33f680d
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bpe_16000/tokenizer_config.json with huggingface_hub
2a5ea3f
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bpe_16000/special_tokens_map.json with huggingface_hub
2abfb26
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bbpe_8000/tokenizer.json with huggingface_hub
18e6260
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bbpe_8000/tokenizer_config.json with huggingface_hub
19831e4
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bbpe_8000/special_tokens_map.json with huggingface_hub
875c37d
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bbpe_32000/tokenizer.json with huggingface_hub
47ae49b
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bbpe_32000/tokenizer_config.json with huggingface_hub
6d7b4d1
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bbpe_32000/special_tokens_map.json with huggingface_hub
cc1093f
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bbpe_16000/tokenizer.json with huggingface_hub
7d26a4d
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bbpe_16000/tokenizer_config.json with huggingface_hub
344ef3d
verified

Ouaill commited on

Upload transformers_tokenizers/shared_bbpe_16000/special_tokens_map.json with huggingface_hub
4fb4adc
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_8000_tokenizer_az/tokenizer.json with huggingface_hub
820948b
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_8000_tokenizer_az/tokenizer_config.json with huggingface_hub
5bd2485
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_8000_tokenizer_az/special_tokens_map.json with huggingface_hub
508f0bf
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_8000_tokenizer_ar/tokenizer.json with huggingface_hub
304c524
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_8000_tokenizer_ar/tokenizer_config.json with huggingface_hub
e66b3d9
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_8000_tokenizer_ar/special_tokens_map.json with huggingface_hub
2252e9c
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_16000_tokenizer_az/tokenizer.json with huggingface_hub
f307f1f
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_16000_tokenizer_az/tokenizer_config.json with huggingface_hub
6a373be
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_16000_tokenizer_az/special_tokens_map.json with huggingface_hub
38da11b
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_16000_tokenizer_ar/tokenizer.json with huggingface_hub
5372309
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_16000_tokenizer_ar/tokenizer_config.json with huggingface_hub
ecbfeb6
verified

Ouaill commited on

Upload transformers_tokenizers/concat_wordpiece_16000_tokenizer_ar/special_tokens_map.json with huggingface_hub
e6136de
verified

Ouaill commited on

Upload transformers_tokenizers/concat_unigram_8000_tokenizer_az/tokenizer.json with huggingface_hub
dcbd9fa
verified

Ouaill commited on