Add DODa independent validation results, code-switching section
Browse files
README.md
CHANGED
|
@@ -244,6 +244,22 @@ daa-tokenizers/
|
|
| 244 |
| DarijaBERT-ar | 80K | 1.298 | 2.413 | 1.572 | 21.1% | 7.9% | 5.7% |
|
| 245 |
| Qwen2.5-Darija | 152K | 2.422 | 2.247 | 2.248 | 100.0% | 100.0% | 100.0% |
|
| 246 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 247 |
## Citation
|
| 248 |
|
| 249 |
```bibtex
|
|
|
|
| 244 |
| DarijaBERT-ar | 80K | 1.298 | 2.413 | 1.572 | 21.1% | 7.9% | 5.7% |
|
| 245 |
| Qwen2.5-Darija | 152K | 2.422 | 2.247 | 2.248 | 100.0% | 100.0% | 100.0% |
|
| 246 |
|
| 247 |
+
## Independent Validation (atlasia/DODa)
|
| 248 |
+
|
| 249 |
+
To address potential contamination, we evaluated on [atlasia/DODa](https://huggingface.co/datasets/atlasia/DODa) (87K Arabizi entries, not used in training):
|
| 250 |
+
|
| 251 |
+
| Tokenizer | V | Fert ↓ | CPT ↑ | EM | Notes |
|
| 252 |
+
|-----------|---|--------|-------|-----|-------|
|
| 253 |
+
| concat BPE 8K | 8K | 2.249 | 2.69 | 99.97% | |
|
| 254 |
+
| concat WP 16K | 16K | 2.029 | 2.98 | 1.65% | Metaspace artifact |
|
| 255 |
+
| concat BPE 32K | 32K | 2.020 | 3.00 | 0.15% | Metaspace artifact |
|
| 256 |
+
| **DarijaBERT-az** | **110K** | **1.314** | **4.55** | 94.4% | Best on DODa (Arabizi-specific) |
|
| 257 |
+
| Translit-Darija | 30K | 1.869 | 3.21 | 3.0% | |
|
| 258 |
+
| CaMeLBERT-MSA | 30K | 3.667 | 1.59 | 98.4% | Worst |
|
| 259 |
+
| Asafaya-BERT | 32K | 3.187 | 1.83 | 94.6% | |
|
| 260 |
+
|
| 261 |
+
Key: DODa is a dictionary (diverse vocabulary), not sentences — fertility is higher for all tokenizers vs. in-domain evaluation. DarijaBERT-az (trained for Arabizi) wins here, confirming domain specificity matters. MSA tokenizers remain 45–64% worse than our best.
|
| 262 |
+
|
| 263 |
## Citation
|
| 264 |
|
| 265 |
```bibtex
|