--- language: - trp license: cc-by-nc-4.0 library_name: transformers pipeline_tag: automatic-speech-recognition tags: - automatic-speech-recognition - audio - speech - ctc - mms - low-resource - kokborok base_model: facebook/mms-1b-all datasets: - sulabhkatiyar/ne-asr-dataset-trp - sulabhkatiyar/ne-asr-dataset-trp-aug model-index: - name: ne-asr-trp results: - task: type: automatic-speech-recognition dataset: name: sulabhkatiyar/ne-asr-dataset-trp type: sulabhkatiyar/ne-asr-dataset-trp split: test metrics: - type: wer value: 70.73 name: Test WER - type: cer value: 26.46 name: Test CER --- # NE-ASR: Kokborok (trp) Romanized Kokborok CTC ASR adapter for [`facebook/mms-1b-all`](https://huggingface.co/facebook/mms-1b-all). ## Model description A per-language CTC **adapter** for the MMS-1B model. The ~1B-parameter base model is **frozen**; only the language adapter (`adapter_attn_dim=16`) and a fresh Latin-script CTC head are trained. Input is 16 kHz mono audio (up to 30 seconds). Output is a Romanized (Latin-script) transcript, lower-cased and NFC-normalized. - Base model: [`facebook/mms-1b-all`](https://huggingface.co/facebook/mms-1b-all) (frozen) - Adapter dimension: `adapter_attn_dim = 16` - Decoding: greedy CTC, no language model ## Datasets - Training / evaluation: [`sulabhkatiyar/ne-asr-dataset-trp`](https://huggingface.co/datasets/sulabhkatiyar/ne-asr-dataset-trp) - Augmented training: [`sulabhkatiyar/ne-asr-dataset-trp-aug`](https://huggingface.co/datasets/sulabhkatiyar/ne-asr-dataset-trp-aug) **Data source & attribution.** The training and evaluation data is derived from the **ARTPARK-IISc Vaani** project (https://vaani.iisc.ac.in/), released under **CC-BY-4.0**. Please retain that attribution when you use these models. ## Evaluation Held-out **test** split, scored **per-sample** with greedy CTC decoding (batch size 1, no language model). Reference processing: NFC + strip + lower. These are the honest per-sample numbers; they supersede any earlier aggregate figures. | split | n (utterances) | WER % | CER % | |-------|---------------:|------:|------:| | test | 279 | 70.73 | 26.46 | ## How to load Part of the [NE Speech AI V1](https://huggingface.co/collections/sulabhkatiyar/ne-speech-ai-v1-6a8efa7593551e32d63b876b) collection. **This repo hosts ONLY the per-language adapter and the CTC head — NOT the frozen ~1B `facebook/mms-1b-all` base weights.** As a result: - `Wav2Vec2ForCTC.from_pretrained("sulabhkatiyar/ne-asr-trp")` does **NOT** work — the repo hosts only the per-language adapter + CTC head, not the frozen ~1B base; the base must be pulled from `facebook/mms-1b-all`. - A naive `model.load_adapter(...)` onto stock `facebook/mms-1b-all` **shape-crashes** for the dim-64 languages (lus/grt) — the base must first be built at the right `adapter_attn_dim`. The snippet below handles this automatically by reading the `adapter.trp.meta.json` sidecar. The snippet below is fully **self-contained** — it needs only public packages and this repo (plus `facebook/mms-1b-all` for the frozen base), no private code. It reads the adapter dim from the meta sidecar, rebuilds `mms-1b-all` at that dim, loads the adapter + CTC head, and runs greedy CTC decoding. ```python # pip install "transformers>=5" huggingface_hub safetensors torch from transformers import (Wav2Vec2ForCTC, Wav2Vec2Config, Wav2Vec2CTCTokenizer, Wav2Vec2Processor, AutoFeatureExtractor) from huggingface_hub import hf_hub_download from huggingface_hub.utils import EntryNotFoundError from safetensors.torch import load_file import json, torch BASE = "facebook/mms-1b-all" def load_ne_asr(repo_id, iso): # Reads the per-language adapter dim from the repo's meta sidecar # (64 for lus/grt, 16 otherwise; falls back to 16 if absent). try: meta = json.load(open(hf_hub_download(repo_id, f"adapter.{iso}.meta.json"))) dim = int(meta.get("adapter_attn_dim", 16)) except EntryNotFoundError: dim = 16 tok = Wav2Vec2CTCTokenizer.from_pretrained(repo_id) feat = AutoFeatureExtractor.from_pretrained(BASE) # MMS's own feature extractor processor = Wav2Vec2Processor(feature_extractor=feat, tokenizer=tok) cfg = Wav2Vec2Config.from_pretrained(BASE) cfg.adapter_attn_dim = dim cfg.vocab_size = len(tok) cfg.pad_token_id = tok.pad_token_id cfg.ctc_zero_infinity = True model = Wav2Vec2ForCTC.from_pretrained(BASE, config=cfg, # frozen ~1B base (~3.6GB, once) ignore_mismatched_sizes=True) model.init_adapter_layers() state = load_file(hf_hub_download(repo_id, f"adapter.{iso}.safetensors")) model.load_state_dict(state, strict=False) # base from mms-1b-all; adapter+head from repo return model.eval(), processor # --- usage (replace with this repo's id + iso) --- model, processor = load_ne_asr("sulabhkatiyar/ne-asr-trp", "trp") import torchaudio wav, sr = torchaudio.load("example.wav") if sr != 16000: wav = torchaudio.functional.resample(wav, sr, 16000) inputs = processor(wav.squeeze().numpy(), sampling_rate=16000, return_tensors="pt") with torch.no_grad(): logits = model(inputs.input_values).logits pred = processor.batch_decode(logits.argmax(-1))[0] print(pred) ``` ## Limitations - The base `facebook/mms-1b-all` encoder (~1B parameters) is **frozen**; only a small per-language adapter (`adapter_attn_dim=16`) and a fresh Latin-script CTC head are trained. Capacity for this language is therefore limited. - Kokborok is a low-resource language; the training data is small and may not cover the full range of speakers, dialects, domains, and recording conditions. - Evaluation used **279** held-out test utterances (per-sample greedy CTC, no language model). ## Citation If you use this model, please cite the MMS base model: ```bibtex @article{pratap2023scaling, title = {Scaling Speech Technology to 1,000+ Languages}, author = {Pratap, Vineel and Tjandra, Andros and Shi, Bowen and Tomasello, Paden and Babu, Arun and Kundu, Sayani and Elkahky, Ali and Ni, Zhaoheng and Vyas, Apoorv and Fazel-Zarandi, Maryam and Baevski, Alexei and Adi, Yossi and Zhang, Xiaohui and Hsu, Wei-Ning and Conneau, Alexis and Auli, Michael}, journal = {arXiv preprint arXiv:2305.13516}, year = {2023} } ``` And the NE-ASR adapter release (placeholder; replace when the canonical publication is available): ```bibtex @misc{katiyar2026neasr, author = {Katiyar, Sulabh}, title = {NE-ASR: MMS-1B Adapters for Northeast Indian Languages}, year = {2026}, howpublished = {\url{https://huggingface.co/sulabhkatiyar/ne-asr-trp}}, note = {Placeholder citation; replace with the canonical publication when available.} } ``` ## License CC-BY-NC-4.0. This adapter is derived from [`facebook/mms-1b-all`](https://huggingface.co/facebook/mms-1b-all), released under CC-BY-NC 4.0. When you use this adapter you must comply with the MMS license terms. The underlying training and evaluation data is derived from the ARTPARK-IISc Vaani project (https://vaani.iisc.ac.in/), released under CC-BY-4.0; please also retain that attribution.