--- language: - el - en license: cc-by-4.0 library_name: nemo base_model: nvidia/canary-1b-v2 pipeline_tag: automatic-speech-recognition tags: - automatic-speech-recognition - speech-translation - speech - audio - NeMo - canary - FastConformer - greek - bilingual - multitask datasets: - google/fleurs - mozilla-foundation/common_voice_17_0 - librispeech_asr metrics: - wer - bleu model-index: - name: Sophea-Canary-ASR results: - task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: type: google/fleurs name: FLEURS (Greek) config: el_gr split: test metrics: - type: wer value: 12.69 name: Test WER (el) - task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: type: mozilla-foundation/common_voice_17_0 name: Common Voice 17.0 (Greek) config: el split: test metrics: - type: wer value: 2.90 name: Test WER (el) - task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: type: google/fleurs name: FLEURS (English) config: en_us split: test metrics: - type: wer value: 12.90 name: Test WER (en) - task: type: translation name: Speech Translation (el to en) dataset: type: google/fleurs name: FLEURS (Greek to English) config: el_gr split: test metrics: - type: bleu value: 24.13 name: Test BLEU (el->en) --- # Sophea-Canary-ASR A **bilingual (Greek + English), multitask** speech model fine-tuned from [`nvidia/canary-1b-v2`](https://huggingface.co/nvidia/canary-1b-v2) (1 B, offline attention-encoder-decoder, FastConformer). ## Note on Greek final sigma (inherited from `canary-1b-v2`) The base `nvidia/canary-1b-v2` SentencePiece tokenizer has **no token for the Greek word-final sigma `ς`** (it folds to ``), together with a handful of other Greek characters — this gap ships with the base canary model itself, not with this fine-tune. An earlier version of this checkpoint inherited that gap and emitted `⁇` in place of every word-final `ς`. The current weights fix it: the model was retrained on tokenizer-normalized Greek targets, so it now outputs the medial form `σ` everywhere and never ``. When serving, restore display orthography with a deterministic word-final `σ -> ς` rule in post-processing. ## Base model vs. this model How much did fine-tuning move each task? Both the base `nvidia/canary-1b-v2` and this model were scored on the **same held-out test manifests** with the **same validation-path harness** and **literal WER** (casing + punctuation), so these deltas are directly comparable. | Task | Test set | Base `canary-1b-v2` | **This model** | Δ | |---|---|---|---|---| | Greek ASR | FLEURS (el) | 30.93 % WER | **12.69 % WER** | **−18.2** | | Greek ASR | Common Voice 17 (el) | 37.30 % WER | **2.90 % WER** | **−34.4** | | English ASR | FLEURS (en) | 12.02 % WER | 12.90 % WER | +0.9 (retained) | | Greek->English | FLEURS (el->en) | BLEU 25.46 | BLEU 24.13 | −1.3 (retained) | **Reading it:** Greek fine-tuning is the whole story — FLEURS WER drops by more than half (30.9 -> 12.7) and Common Voice collapses from 37.3 to **2.9**. English ASR and Greek->English translation were **never trained** here; they only moved by ~1 point, i.e. they were *retained* rather than forgotten. Canary's `source_lang`/`target_lang` prompt-conditioning is what keeps the English and translation pathways intact while the Greek pathway is rebuilt. ![Multitask results](fig_multitask.png) ## What it does One model does **four jobs**: | Task | Test set | Score | |---|---|---| | Greek ASR | FLEURS (el) | **12.69 % WER** | | Greek ASR | Common Voice 17 (el) | **2.90 % WER** | | English ASR | FLEURS (en) | **12.90 % WER** | | Greek -> English speech translation | FLEURS (el->en) | **BLEU 24.13** | It also handles **medical and legal** Greek domains (in the fine-tuning mix). This is the **offline, high-accuracy** model of a two-model Greek ASR system; a separate streaming 0.6 B model serves the real-time path. ## Highlights - **Halves Greek WER vs the streaming baseline** (25.5 % -> 12.7 % FLEURS) and drives Common Voice to **2.9 %**. - **Bilingual for free.** Despite fine-tuning on ~360 h of mostly-Greek data, English ASR held at 12.9 % (base canary-1b-v2: 12.0 %) — Canary's `source_lang`/`target_lang` prompt-conditioning preserves the English pathway (no catastrophic forgetting). - **Translation for free.** Greek->English speech translation (BLEU 24.1) was *retained* from the base through ASR-only fine-tuning — the prompt selects the task, so the AST pathway survives. ## Usage Requires [NeMo](https://github.com/NVIDIA/NeMo) (`pip install nemo_toolkit[asr]`). ```python from nemo.collections.asr.models import EncDecMultiTaskModel from huggingface_hub import hf_hub_download ckpt = hf_hub_download("KIEFERSA/Sophea-Canary-ASR", "canary1bv2_el_stage3.nemo") model = EncDecMultiTaskModel.restore_from(ckpt) # Greek ASR print(model.transcribe(["greek_audio.wav"], source_lang="el", target_lang="el", pnc="yes")) # English ASR print(model.transcribe(["english_audio.wav"], source_lang="en", target_lang="en", pnc="yes")) # Greek -> English speech translation print(model.transcribe(["greek_audio.wav"], source_lang="el", target_lang="en", pnc="yes")) ``` **Prompt-sensitivity note.** This model is fine-tuned with the `canary2` prompt and `pnc=yes`. Pass `source_lang`/`target_lang` explicitly (as above) and keep `pnc=yes` for results matching the reported metrics. Audio should be **16 kHz mono**. ## Training - **Base:** `nvidia/canary-1b-v2` (1 B, offline AED, FastConformer encoder). - **Method:** full fine-tune via a custom NeMo launcher that *restores* the pretrained model and swaps the dataset (avoiding the tokenizer rebuild ..), with a 3-stage **warm-start chain** (each stage initialized from the previous endpoint): 1. **stage-1** — ~250 h Greek ASR (FLEURS + Common Voice + YODAS + TEDx + targeted TTS). 2. **stage-2** — + ~80 h clean Greek (medical, legal, parliamentary) -> 330 h. 3. **stage-3 (this model)** — + 30 h English ASR (LibriSpeech-clean) -> 360 h. - **Optimizer:** AdamW, cosine schedule, lr ~8e-6 (stage-3), bf16, Lhotse dynamic batching. - **Checkpoint selection:** best Greek `val_wer` (0.0577 at stage-3). ### Data | Source | Role | Notes | |---|---|---| | FLEURS (el / en) | Greek + English ASR | clean read speech | | Common Voice 17 (el) | Greek ASR | spontaneous, multi-speaker | | YODAS (el) | Greek ASR | YouTube subtitles, quality-filtered | | TEDx (el) | Greek ASR | real talks | | TTS-synthetic (el) | Greek ASR | Wikipedia/general domain, ~10 % of mix | | Medical / legal (el) | Greek ASR | domain coverage | | LibriSpeech (en) | English ASR | retention | ## Limitations - **Offline only.** Full-context AED — not for streaming/real-time use. - **Translation ceiling.** Greek->English BLEU (~24) is inherited from the base; an explicit AST fine-tune on machine-translated data did **not** improve it (you cannot out-train your labels). Real parallel data would be needed to push past this. - **Domain bias.** Strongest on read speech (FLEURS) and Common Voice; other domains (e.g. heavy dialect, far-field, overlapping speech) are untested. - **Common Voice over-specialization.** The validation mix is CV-heavy, so CV WER (2.9 %) is partly in-distribution; FLEURS (12.7 %) is the more conservative real-world estimate. ## License Released under **CC-BY-4.0**, inheriting the license of the base model [`nvidia/canary-1b-v2`](https://huggingface.co/nvidia/canary-1b-v2). ## Citation If you use this model, please credit this repository and the base model: ```bibtex @misc{kiefer2026canarygreek, title = {Sophea-Canary-ASR: a bilingual Greek+English multitask speech model}, author = {Kirouane, Ayoub}, year = {2026}, howpublished = {Hugging Face, KIEFERSA}, note = {Fine-tuned from nvidia/canary-1b-v2} } ```