shekar-ai/albert-base-v2-persian-zwnj-naab-mlm

ALBERT Persian (Naab, ZWNJ-aware)

A compact Persian ALBERT encoder pretrained with masked language modeling on the Naab corpus. Its SentencePiece vocabulary treats the zero-width non-joiner (ZWNJ, U+200C) as an explicit token, so Persian compounds such as می‌روم and کتاب‌ها survive tokenization intact instead of being split on an invisible character.

This is the shared backbone for the task-specific models in the Shekar toolkit: POS tagging, NER, dependency parsing, and sentiment analysis.

Architecture	ALBERT base — 12 layers, hidden 768, embedding 128, 12 heads
Parameters	11.5 M
Max sequence length	512
Vocabulary	32,004 SentencePiece unigram tokens (ZWNJ included)
Developed by	Ahmad Amirivojdan
License	MIT

Intended Use

• Masked-token prediction (fill-mask) in Persian text
• Sentence and phrase embeddings for clustering, retrieval, and semantic similarity
• Encoder to fine-tune for Persian classification, tagging, and parsing tasks

Usage

Shekar

Shekar exposes the encoder through ContextualEmbedder, which runs an ONNX export on CPU and returns a mask-weighted mean-pooled 768-dimensional sentence vector.

from shekar import ContextualEmbedder

embedder = ContextualEmbedder(model="albert")
vector = embedder("کتاب‌ها دریچه‌ای به جهان دانش هستند.")
print(vector.shape)  # (768,)

Transformers

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="shekar-ai/albert-base-v2-persian-zwnj-naab-mlm")

for prediction in fill_mask("این کتاب بسیار [MASK] است.")[:3]:
    print(prediction["token_str"], round(prediction["score"], 3))

مفید 0.204
آموزنده 0.072
عالی 0.044

For embeddings, load the encoder directly:

import torch
from transformers import AutoModel, AutoTokenizer

repo = "shekar-ai/albert-base-v2-persian-zwnj-naab-mlm"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModel.from_pretrained(repo).eval()

inputs = tokenizer("کتاب‌ها دریچه‌ای به جهان دانش هستند.", return_tensors="pt")
with torch.no_grad():
    hidden = model(**inputs).last_hidden_state

mask = inputs["attention_mask"].unsqueeze(-1)
embedding = (hidden * mask).sum(1) / mask.sum(1)  # (1, 768)

Training

Data. The Naab corpus (~130 GB of cleaned Persian text). Every document was normalized with Shekar's Normalizer before tokenization, then tokenized and concatenated into contiguous blocks of 512 tokens. 2% of the resulting blocks were held out for validation.

Objective. Masked language modeling with a 15% masking probability.

Hyperparameter	Value
Epochs	3
Batch size	32 per device
Learning rate	2e-5
Warmup steps	10,000
Weight decay	0.01
Block size	512 tokens

Evaluation

Masked-token prediction on 3,000 documents from the Naab test split (23,629 masked tokens, 15% masking, Shekar normalization applied):

Metric	Value
Pseudo-perplexity	23.8
Top-1 masked-token accuracy	49.1%

Limitations

At 11.5 M parameters this is a small encoder tuned for CPU-friendly deployment, not a knowledge-rich language model. It predicts frequent grammatical continuations reliably but is weak at factual cloze completion (پایتخت ایران [MASK] است. does not yield تهران). Use it as a representation model and a fine-tuning starting point rather than a knowledge source. Like the corpus it was trained on, it reflects the biases of Persian web and news text.

Citation

@article{Amirivojdan2025Shekar,
  author  = {Amirivojdan, Ahmad},
  title   = {{Shekar: A Python Toolkit for Persian Natural Language Processing}},
  journal = {Journal of Open Source Software},
  volume  = {10},
  number  = {114},
  pages   = {9128},
  year    = {2025},
  doi     = {10.21105/joss.09128},
  url     = {https://joss.theoj.org/papers/10.21105/joss.09128}
}