How to use AWeirdDev/zh-tw-llama3-tokenizer-3k with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("AWeirdDev/zh-tw-llama3-tokenizer-3k", dtype="auto")
