Remove cased tokens from vocabulary
#1
by stephantulkens - opened
Hello!
Since the model is lower-cased, you can remove all cased tokens from the vocabulary. This would save about 720k params (15000 tokens are cased tokens). What you can also try, and which should improve performance, is uncasing the vocabulary.
If you want I can do a PR with the cased tokens removed or the vocabulary uncased.
Stéphan
Hello Stéphan
That would be awesome. We would appreciate a PR for that!!
Both options sound good. Feel free to go with whichever you think works better! Thank you.
Rikiya
I asked some questions on the mixedbread discord about it, I can also ask them here. What do you prefer?