clean up tokenizer and add [unused0] and [unused1] as special tokens 85b42e5
Jo Kristian Bergum commited on
How to use vespa-engine/col-minilm with Transformers:
# Load model directly
from transformers import AutoTokenizer, ColBERT
tokenizer = AutoTokenizer.from_pretrained("vespa-engine/col-minilm")
model = ColBERT.from_pretrained("vespa-engine/col-minilm", device_map="auto")