---
language:
- ur
license: mit
tags:
- urdu
- nastaliq
- bert
- character-level
- masked-language-modeling
- neobert
- low-resource
- abjad
datasets:
- statmt/cc100
metrics:
- perplexity
model-index:
- name: NastaliqBERT
  results:
  - task:
      type: fill-mask
    dataset:
      name: CC-100 Urdu
      type: statmt/cc100
    metrics:
    - type: perplexity
      value: 1.67
---

# NastaliqBERT

**Character-level NeoBERT encoder pretrained on unvocalized Urdu Nastaliq text.**

NastaliqBERT is the **first character-level pretrained encoder for Urdu** and the **first NeoBERT-architecture model for any South Asian language**. It achieves a masked language modelling perplexity of **1.67** on CC-100 Urdu.

## Model description

NastaliqBERT uses the NeoBERT architecture with RoPE positional embeddings, SwiGLU activation, and Pre-RMSNorm. It operates on a 116-token character vocabulary covering all Urdu Nastaliq letters (including final forms), ASCII, and special tokens — no subword tokenization.

| | |
|---|---|
| Architecture | NeoBERT (RoPE + SwiGLU + Pre-RMSNorm) |
| Vocabulary | 116 characters |
| Hidden size | 512 |
| Layers | 6 |
| Attention heads | 8 |
| Parameters | ~19.4M |
| Pretraining data | CC-100 Urdu (~8GB) |
| Perplexity | **1.67** |

## Why character-level for Urdu?

Urdu is written in Nastaliq a right-to-left abjad where short vowels are not written. Subword tokenizers fragment Nastaliq ligatures at arbitrary boundaries. Character-level modelling preserves:
- The aspiration digraph ھ (do-chashmi-he) as a distinct unit
- Final letter forms (ے vs ی, ں vs ن)
- Correct morphological boundaries

## Usage

```python
import torch, json, sys
sys.path.insert(0, 'src')
from model import BertCharUr, BertCharUrConfig
from tokenizer import CharTokenizer

tok = CharTokenizer()
with open('config.json') as f:
    cfg = BertCharUrConfig.from_dict(json.load(f))

model = BertCharUr(cfg)
state = torch.load('model.pt', map_location='cpu', weights_only=True)
model.load_state_dict(state['model_state'])
model.eval()

ids = tok.encode("مشاہیر", max_length=64)
input_ids = torch.tensor([ids])
mask = torch.ones_like(input_ids)

with torch.no_grad():
    hidden = model.get_encoder_output(input_ids, mask)
# hidden: (1, seq_len, 512)
```

## Downstream use

NastaliqBERT is used as the encoder in [UrduPhon](https://huggingface.co/mahwizzzz/UrduPhon), a G2P model for Urdu TTS that uses the Phonological Feature Decoder (PFD).

## Training

Pretrained for 100,000 steps on CC-100 Urdu (`statmt/cc100`, `name="ur"`) using:
- MLM with 15% mask rate (no NSP)
- AdamW optimizer, lr=1e-4, cosine decay
- Effective batch size 64 (32 × 2 grad accumulation)
- Mixed precision (bf16)
- FlashAttention-2 (via `F.scaled_dot_product_attention`)

## Citation

```bibtex
@misc{nastaliqbert2025,
  title        = {NastaliqBERT: A Character-Level NeoBERT Encoder for Urdu's Defective Abjad Script},
  author       = {Mahwiz},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/mahwizzzz/NastaliqBERT}},
}
```