Adjust max length in tokenizer config
#4
by aehrm - opened
Update tokenizer configuration to align max length with config.json. This change does not affect the tokenizer's output. It only prevents (incorrect) warnings for sequences exceeding 1024 tokens.
JanPf changed pull request status to merged