Upload latest_main.tex with huggingface_hub

latest_main.tex

@@ -55,7 +55,7 @@ No tokenizer benchmark exists for Darija. Practitioners building Darija language
 We fill this gap with the first comprehensive tokenizer benchmark for Moroccan Darija. Our benchmark covers:
 \begin{enumerate}[noitemsep,topsep=2pt]
 \item \textbf{Scale}: 40 tokenizer configurations (4 algorithms $\times$ 2 architectures $\times$ 5 vocabulary sizes from 8K to 110K)
-\item \textbf{Comparison}: Fair head-to-head evaluation against nine existing Arabic and Darija tokenizers from HuggingFace, including DarijaBERT \citep{gaanoun2023darijabert} at matching vocabulary sizes (80K and 110K)
+\item \textbf{Comparison}: Fair head-to-head evaluation against ten existing Arabic and Darija tokenizers from HuggingFace, including DarijaBERT \citep{gaanoun2023darijabert} at matching vocabulary sizes (80K and 110K)
 \item \textbf{Metrics}: Compression (fertility, characters per token), cross-script fairness (relative disparity, Gini coefficient), morphological fidelity (alignment with Farasa morpheme boundaries), and exact reconstruction
 \item \textbf{Proposed improvement}: A concatenated architecture that trains separate per-script sub-tokenizers, reducing cross-script disparity by up to $4\times$
 \end{enumerate}
@@ -305,7 +305,7 @@ Bootstrap 95\% confidence intervals (500 resamples) confirm that all reported me
 
 \section{Comparison with Existing Tokenizers}
 
-To contextualize our benchmark, we compare our best tokenizers against nine existing Arabic and Darija tokenizers from HuggingFace. The MSA group includes CaMeLBERT-MSA \citep{obeid2021camelbert} (30K WordPiece, 167 GB MSA), Asafaya-BERT \citep{safaya2020kuisail} (32K WordPiece, 95 GB Arabic), Aranizer-SP-86k \citep{koubaa2024arabian} (86K SentencePiece), and B2BERT \citep{b2bert2025} (30K, reuses CaMeLBERT-MSA tokenizer). The Darija group includes DarijaBERT \citep{gaanoun2023darijabert} (80K and 110K WordPiece variants), Darija-Tokenizer (30K WordPiece), Translit-Darija (30K BPE), and Qwen2.5-Darija \citep{skiredj2025gemmaroc} (152K SentencePiece). Table~\ref{tab:comparison} reports the results.
+To contextualize our benchmark, we compare our best tokenizers against ten existing Arabic and Darija tokenizers from HuggingFace. The MSA group includes CaMeLBERT-MSA \citep{obeid2021camelbert} (30K WordPiece, 167 GB MSA), Asafaya-BERT \citep{safaya2020kuisail} (32K WordPiece, 95 GB Arabic), Aranizer-SP-86k \citep{koubaa2024arabian} (86K SentencePiece), and B2BERT \citep{b2bert2025} (30K, reuses CaMeLBERT-MSA tokenizer). The Darija group includes DarijaBERT \citep{gaanoun2023darijabert} (80K and 110K WordPiece variants), DarijaBERT-mix (160K WordPiece), Moroccan-Darija-Tokenizer (30K BPE), Translit-Darija (30K BPE), and Qwen2.5-Darija \citep{skiredj2025gemmaroc} (152K SentencePiece). Table~\ref{tab:comparison} reports the results.
 
 \begin{table*}[t]
 \centering
@@ -334,27 +334,28 @@ B2BERT & \texttt{AHAAM/B2BERT} & 30K & 1.817 & 3.173 & 0.427 & 3.11 & 1.64 & 29.
 \midrule
 DarijaBERT-ar & \texttt{SI2M-Lab/DarijaBERT} & 80K & 1.417 & 2.404 & 0.410 & 3.98 & 2.17 & 13.7 & 8.0 \\
 DarijaBERT-az & \texttt{SI2M-Lab/DarijaBERT-arabizi} & 110K & 1.605 & 1.517 & \textbf{0.055} & 3.48 & 3.45 & 14.8 & 8.0 \\
-Darija-Tokenizer & \texttt{SI2M-Lab/DarijaBERT-mix} & 30K & 1.570 & 2.902 & 0.459 & 3.63 & 1.79 & 0.0 & 0.0 \\
+DarijaBERT-mix & \texttt{SI2M-Lab/DarijaBERT-mix} & 160K & 1.333 & 1.567 & 0.149 & 4.19 & 3.34 & 14.8 & 8.0 \\
+Moroccan-Darija-Tok & \texttt{BounharAbdelaziz/Moroccan-Darija-Tokenizer} & 30K & 1.570 & 2.902 & 0.459 & 3.63 & 1.79 & 0.0 & 0.0 \\
 Translit-Darija & \texttt{atlasia/Transliteration-Moroccan-Darija} & 30K & 1.796 & 1.658 & 0.077 & 3.11 & 3.16 & 0.0 & 0.0 \\
 Qwen2.5-Darija & \texttt{GemMaroc/Qwen2.5-7B-Instruct-darija} & 152K & 2.340 & 2.246 & 0.040 & 2.38 & 2.33 & 100.0 & 100.0 \\
 \bottomrule
 \end{tabular}
-\caption{Comparison of our best Darija tokenizers against nine existing Arabic and Darija tokenizers from HuggingFace. All evaluated on the same Darija test set (33,846 texts). Our 80K and 110K tokenizers match DarijaBERT's vocabulary sizes for direct comparison. $\Delta F$ = relative cross-script disparity (\S\ref{sec:fairnessmetrics}). Bold = lowest disparity.}
+\caption{Comparison of our best Darija tokenizers against ten existing Arabic and Darija tokenizers from HuggingFace. All evaluated on the same Darija test set (33,846 texts). Our 80K and 110K tokenizers match DarijaBERT's vocabulary sizes for direct comparison. $\Delta F$ = relative cross-script disparity (\S\ref{sec:fairnessmetrics}). Bold = lowest disparity.}
 \label{tab:comparison}
 \end{table*}
 
-Three findings stand out. First, Darija-specific tokenization dramatically improves compression at matching vocabulary sizes. Our 80K tokenizer (F = 1.183) achieves 33\% lower fertility than DarijaBERT-ar (F = 1.761) at the same vocabulary size. Our 110K tokenizer (F = 1.155) achieves 27\% lower fertility than DarijaBERT-az (F = 1.575). Even our 32K tokenizer (1.307) outperforms DarijaBERT-az (1.575) despite using 3.4$\times$ fewer vocabulary slots.
+Three findings stand out. First, Darija-specific tokenization dramatically improves compression at matching vocabulary sizes. Our 80K tokenizer (F = 1.183) achieves 33\% lower fertility than DarijaBERT-ar (F = 1.761) at the same vocabulary size. Our 110K tokenizer (F = 1.155) achieves 27\% lower fertility than DarijaBERT-az (F = 1.575). Even our 32K tokenizer (1.307) outperforms DarijaBERT-az (1.575) despite using 3.4$\times$ fewer vocabulary slots. DarijaBERT-mix, despite its massive 160K vocabulary (F = 1.414), still underperforms our 32K tokenizer---vocabulary size alone cannot compensate for suboptimal training architecture.
 
 Second, MSA-trained tokenizers transfer poorly to Darija. They allocate vocabulary to MSA patterns absent in Darija, inflating fertility by 32--43\%. Our smallest tokenizer (F = 1.307) outperforms all four MSA tokenizers, including Aranizer-SP-86k with 86K vocabulary (F = 1.918). Qwen2.5-Darija (F = 2.307) confirms that general-purpose multilingual tokenization is unsuitable for Darija regardless of vocabulary size.
 
-Third, exact reconstruction fails for most external tokenizers. BERT-based models achieve under 40\% exact match; DarijaBERT under 15\%; Darija-Tokenizer 0\%. Only Qwen2.5-Darija achieves 100\%---but at 77\% higher fertility than our 32K tokenizer and double the fertility of our 110K.
+Third, exact reconstruction fails for most external tokenizers. BERT-based models achieve under 40\% exact match; DarijaBERT variants under 15\%; Moroccan-Darija-Tokenizer 0\%. Only Qwen2.5-Darija achieves 100\%---but at 77\% higher fertility than our 32K tokenizer and double the fertility of our 110K.
 
 Figure~\ref{fig:comparison} visualizes these comparisons. The fertility panel shows a clear gap between our tokenizers and all external baselines. The disparity panel confirms that our architectures achieve balanced cross-script treatment, while most external tokenizers exhibit severe imbalance.
 
 \begin{figure*}[!t]
 \centering
 \includegraphics[width=0.95\textwidth]{external_comparison.png}
-\caption{Comparison of our best tokenizers against nine existing Arabic/Darija tokenizers. Top-left: fertility (lower = better). Top-right: cross-script disparity (lower = better). Bottom-left: exact match rate by script. Bottom-right: characters per token (higher = better). Our tokenizers achieve low fertility, balanced disparity, and $\geq$99\% exact match.}
+\caption{Comparison of our best tokenizers against ten existing Arabic/Darija tokenizers. Top-left: fertility (lower = better). Top-right: cross-script disparity (lower = better). Bottom-left: exact match rate by script. Bottom-right: characters per token (higher = better). Our tokenizers achieve low fertility, balanced disparity, and $\geq$99\% exact match.}
 \label{fig:comparison}
 \end{figure*}