omarmomen/babylm_10M
Viewer • Updated • 1.21M • 38
How to use omarmomen/ptb_filtered_lowcase_bpe_tokenizer_8 with Transformers:
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("omarmomen/ptb_filtered_lowcase_bpe_tokenizer_8", dtype="auto")This model is part of the experiments in my master's thesis titled "Linguistic Structure Induction from Language Models" (https://arxiv.org/abs/2403.09714).
"omarmomen/ptb_filtered_lowcase_bpe_tokenizer_8" is a RobertaTokenizer pretrained on the Penn Tree Bank Training dataset (uncased) with 8K tokens.