Nawah-ASR-118M-v5

Arabic ASR with no third-party weights anywhere in the stack. A 49.86M Whisper-architecture encoder trained from scratch on our own 866 h, feeding a 50M Arabic LLM through the Qwen2Audio shell, finished with a sequence-level RL phase.

model	params	WER	CER
Nawah-ASR-118M-v5	118M	0.3358	0.1420
openai/whisper-large-v3	1,550M	0.4149	0.1245
openai/whisper-medium	769M	0.4776	0.1628
openai/whisper-small	244M	0.5739	0.2062

3,165 held-out clips, video-disjoint from training, every row scored with the same normalizer and the same manifest — the Whisper numbers were measured here, not quoted from elsewhere.

v5 has 13x fewer parameters than whisper-large-v3 and 0.079 lower WER. It also decodes ~50% faster than our own whisper-small-encoder variant on CPU (5.9x vs 3.9x realtime).

One honest asymmetry: large-v3 wins on CER (0.1245 vs 0.1420) while losing on WER. It recognises the acoustics at least as well, but writes them in Modern Standard Arabic orthography — so its characters land close and its words land wrong. v5 is trained on Egyptian transcripts and matches the target convention. If you need MSA output, large-v3 is the better starting point; if you need Egyptian as it is actually written, v5 is.

3,165 held-out clips, video-disjoint from training, one normalizer for every row. ## ⚠️ Loading

from load_model import load_v5, transcribe
model, tok, fe = load_v5("oddadmix/Nawah-ASR-118M-v5")
print(transcribe(model, tok, fe, wav))     # float32 mono @ 16 kHz

from_pretrained alone returns this model at the wrong frame rate, silently — see Frame rate below. load_v5 sets it and then asserts it.

---

Techniques

1. The encoder is ours, trained from scratch with CTC

whisper-small's encoder is 88M multilingual parameters, most of which are not doing Arabic. We trained a replacement on our own corpus:

• Shape: Whisper architecture, d576 / 12 layers / 9 heads / ffn 2304 = 49.86M (whisper-small: d768, 88.15M). The layer structure is kept identical to whisper's because Qwen2AudioEncoder loads whisper encoder weights by prefix remap — preserving the structure keeps that graft working.
• Objective: character CTC. An encoder alone has no loss, and CTC is data-efficient and yields a standalone WER, so the encoder's quality is known before it is ever grafted. 33 Arabic characters + blank, 100% charset coverage.
• Feasibility check first: whisper's conv stack outputs 50 Hz, so a median 3.0 s clip gives 151 frames against 30 characters — ratio 0.19, only 0.05% of clips infeasible. (CTC requires input frames ≥ target symbols; at a lower frame rate this would not have worked.)
• Variable-length training: WhisperEncoder hard-asserts exactly 3000 mel frames (30 s) and our median clip is 3.0 s, so ~87% of every forward pass would be silence. A subclass slices the positional embeddings to the batch's real length instead; the 1500 embeddings are kept intact so the exported encoder still drops into the 30 s-padded downstream model.
• Duration bucketing: whisper's encoder has no attention mask, so intra-batch padding is attended to as if it were audio. Tight length buckets keep that contamination small.
• Result: greedy CTC WER 0.4255 — versus whisper-small's 0.5739 end-to-end with a full seq2seq decoder. Note this advantage did not carry through to the grafted model: greedy-CTC quality and "features an LM decodes well" are different properties.

Training took 14 epochs. At 2 epochs it was still descending (0.6263) — stopping there would have produced a misleading "from-scratch encoders are worse" conclusion.

2. 50 Hz audio tokens, not 25 Hz

Qwen2Audio's encoder ends with a stride-2 average pool that halves Whisper's native 50 Hz. v5 removes it, doubling the temporal resolution the language model sees.

stage	rate
mel input	100 Hz
after conv2 (stride 2)	50 Hz
after avg_pooler (stride 2)	25 Hz — removed

Worth -0.011 WER / -0.009 CER in a controlled ablation (identical data, epochs and LRs; only the pooler changed). It costs ~2x the compute since the LM's sequence doubles. At 50 Hz the longest clip in the corpus is 897 audio tokens, well inside the 2048 context.

Frame rate is the sharp edge of this model. That pooler holds no parameters and is rebuilt by transformers on every load, so from_pretrained returns all 300 tensors byte-identical, with no warning, at 25 Hz — half the rate the model was trained at. config.json records audio_config.pool_stride: 1; stock transformers ignores it. Hence load_v5, which sets the rate and asserts 1500 tokens per 30 s before returning.

3. GRPO — sequence-level RL on top of supervised training

Cross-entropy optimises token likelihood. WER is a sequence-level metric and the two are not the same objective. GRPO samples K hypotheses per clip, scores each, and updates on the group-relative advantage. For ASR this is MWER (minimum-WER) training in a modern formulation.

• 3,000 steps, 8 clips x (6 sampled + greedy), T=0.6, lr 2e-6
• KL 0.05 against the frozen SFT policy
• Encoder frozen; LM + projector + lm_head train (68.5M)
• Worth -0.032 WER (-8.8% relative) over the identical model with cross-entropy alone

Three choices that turned out to be load-bearing:

The greedy hypothesis is in every group. Sampled-only groups scored 0.607 WER at T=0.4 against greedy's 0.368 — at every temperature tried, all samples were worse than what the model actually does at inference, so the policy would have been optimising a distribution it never uses.

The reward normalises orthography and blends in CER. -(0.7*WER + 0.3*CER) on text with hamza seat, ة/ه, ى/ي and diacritics folded. Raw WER against one reference punishes valid spelling variants, and gives no gradient between "one letter off" and "entirely wrong".

The reference is the dataset's own label, not a teacher's. An earlier design used Audar-ASR-V1-Turbo's transcripts (it scores 0.127 on this eval). That introduces a train/eval reference mismatch — Audar differs from these labels 12.7% of the time — and caps the student at the teacher. Audar's real contribution was measuring that the labels are sound.

The reward metric and the eval metric are deliberately different, so the gain transferred across a metric change rather than being optimised into the reported number.

4. Training pipeline

1. Encoder: CTC from scratch, 866 h, 14 epochs
2. Stage 1: projector only, 1 epoch (encoder and LM frozen)
3. Stage 2: + language model, 3 epochs, lr-proj 2e-4 / lr-lm 1e-4
4. GRPO: 3,000 steps

Data: 866 h / 867,849 clips of Arabic YouTube speech (MASC-derived), 3.2 h held out by video_id so no source recording straddles the split. The corpus is predominantly Modern Standard Arabic in register: only 4.6% of training clips and 2.8% of eval clips contain an Egyptian dialect marker (دي / بس / احنا / مش / عشان / كده …), against 25.2% for a conversational Egyptian corpus measured the same way.

Limitations

This model is strongest on Modern Standard Arabic. The reported 0.3358 is on MSA-register speech (see Training pipeline for the measurement). Colloquial dialect is the weaker case: on a conversational Egyptian corpus it scores around 0.55 WER, and closing that gap is active work rather than a solved problem. An earlier revision of this card claimed the reverse (that MSA was out of distribution); that was wrong.

Short-form (median 3.0 s) — segment long audio before transcribing. Clips under ~3 s remain the weakest case: with a median of 6 words, a single error is ~17% WER and the model tends to complete from its language prior rather than from what it heard. GRPO used clips ≤10 s (98.4% of corpus).