formatting
Browse files
README.md
CHANGED
|
@@ -114,8 +114,8 @@ tokenizer = PreTrainedTokenizerFast.from_pretrained(
|
|
| 114 |
|
| 115 |
text = "ನಿಮ್ಮ ಹೆಸರು ಏನು?"
|
| 116 |
print(tokenizer.encode(text))
|
| 117 |
-
|
| 118 |
-
Related work :
|
| 119 |
|
| 120 |
- M. Velayuthan and K. Sarveswaran, “Egalitarian Language Representation in Language Models: It All Begins with Tokenizers,” COLING 2025.
|
| 121 |
arXiv:2409.11501 [cs.CL]. DOI: https://doi.org/10.48550/arXiv.2409.11501
|
|
|
|
| 114 |
|
| 115 |
text = "ನಿಮ್ಮ ಹೆಸರು ಏನು?"
|
| 116 |
print(tokenizer.encode(text))
|
| 117 |
+
```
|
| 118 |
+
# Related work :
|
| 119 |
|
| 120 |
- M. Velayuthan and K. Sarveswaran, “Egalitarian Language Representation in Language Models: It All Begins with Tokenizers,” COLING 2025.
|
| 121 |
arXiv:2409.11501 [cs.CL]. DOI: https://doi.org/10.48550/arXiv.2409.11501
|