--- license: apache-2.0 pipeline_tag: audio-classification language: - zh - en tags: - keyword-spotting - wake-word - kws - zipformer - transducer - streaming - sherpa-onnx - robotics - IB-Robot - ascend - edge-deployment --- # Model Card for Sherpa-ONNX KWS Zipformer zh-en 3M (IB-Robot) A streaming Zipformer2 transducer keyword spotter (3M parameters, Chinese + English) packaged for the [IB-Robot](https://atomgit.com/openeuler/IB_Robot) framework and compiled for Ascend 310P1. This is the always-on stage: it runs on the idle robot and decides when to wake the rest of the pipeline (VAD → STT → dialogue → TTS). Keywords are **configured at run time** from a text file of token sequences — no retraining is needed to change the wake word. ## Deployments | deployment | backend | artifacts | |---|---|---| | `ascend_310p` | Ascend ACL (Ascend310P1) | `encoder.om`, `decoder.om`, `joiner.om` | | `ascend_310b` | Ascend ACL (Ascend310B1) | `encoder.om`, `decoder.om`, `joiner.om` | 310B board-side KWS | | `torch_cpu` | ONNX Runtime CPU | `assets/onnx/{encoder,decoder,joiner}.onnx` | Shared contract: `tensor_model/zipformer2_transducer_kws/keyword_spotting` (`host.kws.features [1,45,80] -> host.kws.logit [1,263]`), preprocessing `kaldi-fbank80-16khz-povey-snipedgesfalse-v1`, output `keyword-label-on-trigger-v1`, logical revision `sherpa-onnx-kws-zipformer-zh-en-3M@2025-12-20-chunk16-left64`. Both deployments are **stateful**: the encoder carries 38 cache tensors across chunks (`state_scope: stream`, `execution_structure: iterative`, `cancellation_granularity: checkpoint`). The manifest declares them as a single state bank `encoder.bank`, linking each `*_in` to the matching `*_out`. The contract is `orchestration_visibility: session`, not `executor`: the caller owns the loop. Each call runs one encoder → decoder → joiner pass over a single chunk; advancing the caches, shifting the feature buffer and running the beam search are the session's job — see `assets/reference_impl/live_kws.py`. ## Tensor contract **encoder** — 1 feature input + 38 state inputs, 1 output + 38 state outputs: ```text x float32 [1, 45, 80] 45 fbank frames, 80 mel bins encoder_out float32 [1, 8, 320] 8 output frames per chunk ``` The chunk is 45 frames in / 8 frames out and **advances by 32 frames** (`decode_chunk_len = 32`), i.e. each step consumes 320 ms of new audio while seeing 130 ms of right context. This is the `chunk-16-left-64` streaming variant. State tensors (all zeros at stream start, fed back verbatim), 6 encoder layers with `left_context = [64,32,16,8,16,32]`, `key_dim = [128,128,128,256,128,128]`, `value_dim = [48,48,48,96,48,48]`: ```text cached_key_{i} [left_context[i], 1, key_dim[i]] cached_nonlin_attn_{i} [1, 1, left_context[i], 96] cached_val1_{i} [left_context[i], 1, value_dim[i]] cached_val2_{i} [left_context[i], 1, value_dim[i]] cached_conv1_{i} [1, 128, 7] cached_conv2_{i} [1, 128, 7] embed_states [1, 128, 3, 19] processed_lens int64 [1] frames consumed so far, +32 per chunk ``` **decoder** — `y int64 [1, 2]` (context size 2) → `decoder_out float32 [1, 320]`. **joiner** — `encoder_out [1, 320]` + `decoder_out [1, 320]` → `logit [1, 263]`. ### Feature extraction Kaldi-compatible fbank: 16 kHz mono, 80 mel bins, 25 ms window / 10 ms shift, **povey window**, `snip_edges = false`, no dithering, no CMVN. `snip_edges = false` defines frame centres over the **whole signal** with mirrored padding, so features **cannot be computed per audio block independently** — doing so produces wrong frames at every block boundary. The reference implementation keeps a rolling PCM buffer, recomputes fbank over it, and emits only newly-stable frames, dropping the last 3 (their right context has not arrived yet). See `assets/reference_impl/live_kws.py`. ## Repository Structure - `inference_manifest.json` — deployment routing (schema v3) - `assets/adapter.json` — adapter identity (`zipformer2_transducer_kws` / `keyword_spotting`) - `assets/tokens.txt` — 263-entry token table, English ARPAbet phones plus pinyin syllables with tone marks (needed to encode keywords) - `assets/keywords.txt`, `assets/keywords_sensitive.txt` — example keyword sets (default and higher-sensitivity thresholds) - `assets/en.phone` — English lexicon required to encode **new** keywords - `assets/onnx/*.onnx` — upstream ONNX models (the `torch_cpu` deployment and the ATC compilation source) - `assets/pretrained-epoch-13-avg-2.pt` — the upstream PyTorch training checkpoint the ONNX graphs were exported from (13 MB, averaged weights) - `assets/reference_impl/np_fbank.py` — dependency-free numpy fbank - `assets/reference_impl/kws_decode.py` — context graph + transducer keyword beam search (`max_active_paths=4`, `num_trailing_blanks=1`, `unk_id=2`) - `assets/reference_impl/live_kws.py` — end-to-end streaming driver - `artifacts/ascend_310p/*.om` - `artifacts/ascend_310b/*.om` — 310B compiled trio ## Source Model The origin is the icefall training repository **[`pkufool/icefall-kws-zipformer-zh-en-3M-2025-12-20`](https://www.modelscope.cn/models/pkufool/icefall-kws-zipformer-zh-en-3M-2025-12-20)** on ModelScope (branch `master`, declared license Apache-2.0), checkpoint `epoch-13-avg-2`, streaming variant **`chunk-16-left-64`**, logical revision `sherpa-onnx-kws-zipformer-zh-en-3M@2025-12-20-chunk16-left64`. The widely-cited download is the sherpa-onnx GitHub release tarball, but that tarball **redistributes this repository's `onnx/` directory** — all three ONNX graphs we ship are byte-identical to the files there, which is how the origin was identified rather than assumed: | shipped as | ModelScope path | bytes | sha256 | |---|---|---:|---| | `assets/onnx/encoder.onnx` | `onnx/encoder-epoch-13-avg-2-chunk-16-left-64.onnx` | 11,976,272 | `540ff509ed89bd22afe04bf7049a54bb1c95c6d8a18742ea9691910cdb5f859e` | | `assets/onnx/decoder.onnx` | `onnx/decoder-epoch-13-avg-2-chunk-16-left-64.onnx` | 759,829 | `63a22dd60f40fff082ac3e09afa507f6787da36df76ded2fbe145fa233e22c21` | | `assets/onnx/joiner.onnx` | `onnx/joiner-epoch-13-avg-2-chunk-16-left-64.onnx` | 338,154 | `76f7a24ed0c08633af14b2ee377f747af880d3b65eeba2cd3f31f3380fb73e8d` | | `assets/tokens.txt` | `data/lang_phone/tokens.txt` | 1,928 | `2d3f32311f9b692b964da3c90e830258d3e78e013cb0c992dbfb15cd5a1a71b0` | | `assets/pretrained-epoch-13-avg-2.pt` | `checkpoint/pretrained-epoch-13-avg-2.pt` | 13,057,683 | `62188d073aaf7ec33f85a2c1e1f2a75751658743daeb203f17241ec3bf4ff6eb` | Only the encoder distinguishes the streaming variants — `decoder` and `joiner` are the same bytes for `chunk-8-left-64`, `chunk-16-left-64` and `chunk-16-left-128` in that repository. The variant is therefore pinned by the encoder hash above, and independently confirmed by matching the ONNX metadata (`T=45`, `decode_chunk_len=32`) against the streaming driver's constants rather than trusting a file name. `chunk-16` is the 320 ms-latency variant; the 160 ms `chunk-8` variant exists upstream and is not used here. **The training checkpoint is shipped** (`assets/pretrained-epoch-13-avg-2.pt`, the averaged weights the ONNX export was produced from), matching the convention in the rest of this organisation, where the upstream checkpoint lives under `assets/`. The repository also holds `checkpoint/epoch-13.pt` (51,959,636 bytes, full training state including optimizer); that is not needed for inference or re-export and is not shipped. `assets/en.phone` is **not** in the ModelScope repository — it exists only in the sherpa-onnx release tarball, so that remains the source for it: ```bash wget https://github.com/k2-fsa/sherpa-onnx/releases/download/kws-models/\ sherpa-onnx-kws-zipformer-zh-en-3M-2025-12-20.tar.bz2 # 32,885,699 bytes # sha256 68447f4fbc67e70eee3a93961f36e81e98f47aef73ce7e7ca00885c6cd3616a6 ``` | shipped as | upstream name in archive | bytes | sha256 | |---|---|---:|---| | `assets/en.phone` | `en.phone` | 3,330,061 | `f7000ec3a90544c0c7c16090d8951779c2b322e14dad5006290f498567d439ea` | The five files taken from that archive (three ONNX, `tokens.txt`, `en.phone`) were each verified byte-identical against a fresh download as well, so the two sources agree. The ONNX **export** is not byte-reproducible from the checkpoint here: the icefall export script is not part of this bundle. What is pinned is the exact checkpoint, the exact exported graphs, and the fact that the two come from the same upstream commit. Compiled on the board with ATC (`--soc_version=Ascend310P1`): | file | bytes | sha256 | |---|---:|---| | `artifacts/ascend_310p/encoder.om` | 62,669,509 | `dc95d61a66677b1e83b566bfc274d2092eba8fc8d4cfbf7c51beddfc0381c20d` | | `artifacts/ascend_310p/decoder.om` | 476,554 | `8b4e5a7b986427541ee1c8ced274ddd1d6141b9585be3717973d9db116e1c2eb` | | `artifacts/ascend_310p/joiner.om` | 216,152 | `76a21d460a27e7b6424722067365d5697a8c7168f47ab11a62a54939190e20e5` | The upstream repository declares Apache-2.0 but publishes no training-corpus description or dataset licence for this release. ## Validation Measured on a real Ascend 310P1 with the streaming driver, 8.5 s of 16 kHz audio containing two wake-word utterances: ```text chunks 26 mean compute 91 ms / chunk (OM inference + keyword decode) audio per chunk 320 ms RTF 0.29 detections 2.37 s, 5.25 s (both expected utterances, no false alarms) backlog 30–50 ms (stable — the pipeline keeps up) ``` RTF 0.29 leaves ~3× headroom, and the backlog stays flat rather than growing, which is the real test for a streaming model: it is consuming audio at least as fast as it arrives. ### OM vs ONNX, tensor level Continuous streaming comparison on the same 310P1: both backends run the full 45-frame-in / 32-frame-shift loop over the upstream `test_wavs`, **each keeping its own 38 cache tensors**, so any error accumulates exactly as it would in production. ONNX Runtime 1.23.2 CPU fp32 is the reference. | wav | chunks | encoder_out cosine (min) | encoder_out rel_l2 (mean) | worst state cosine | state cosine, last chunk | |---|---:|---:|---:|---:|---:| | `zh_1.wav` | 15 | 0.991565 | 0.021 | 0.989623 | 0.999934 | | `en_0.wav` | 20 | 0.998495 | 0.013 | 0.997338 | 0.999178 | | `en_1.wav` | 51 | 0.986783 | 0.017 | 0.956989 | 0.999128 | `decoder` cosine 0.99999991, `joiner` cosine 0.99999992, **joiner argmax agreement 1.00** — the OM and ONNX joiners pick the same token every time. `processed_lens` (the int64 frame counter carried through the state) is exactly equal on every chunk of every file. **The state does not drift.** The worst state cosine is reached mid-stream (chunk 1, 6 and 14 respectively) and the value *recovers* — after 51 chunks of continuous streaming, 16.7 s of audio, `en_1` is back at 0.9991. The error is per-chunk numerical noise, not accumulation, which is the failure mode that would actually matter for a model whose caches never reset. ### OM vs ONNX, end to end The number that decides whether the OM is usable is not a cosine, it is whether detections change. Same audio, same decoder, same context graph — only the backend differs: | wav | chunks | OM hits | ONNX hits | identical | |---|---:|---|---|---| | `en_0.wav` | 20 | `LIGHT_UP` @ 3.65 s | `LIGHT_UP` @ 3.65 s | yes | | `en_1.wav` | 51 | `LOVELY_CHILD` @ 6.21 s | `LOVELY_CHILD` @ 6.21 s | yes | | `zh_0/1/2.wav` | 17/15/13 | none | none | yes | | `zh_3.wav` | 24 | `文森特卡索` @ 2.37 s, `法国` @ 5.25 s | same | yes | | `zh_4.wav` | 13 | `蒋友伯` @ 1.73 s, `女儿` @ 3.65 s | same | yes | | `zh_5.wav` | 12 | `周望军` @ 1.41 s, `落实` @ 2.69 s | same | yes | | `zh_6.wav` | 10 | `朱丽楠` @ 1.41 s, `见面会` @ 3.01 s | same | yes | **9 files, 10 detections, zero disagreement — same keyword, same chunk.** The three files with no listed keyword produce no hits on either backend, so this covers false alarms as well as misses. The detections also match the upstream ground truth (`test_wavs/en_trans.txt`, `keywords_raw.txt`), so both backends are right, not merely consistent. ## Usage ```python from inference_manifest import load_inference_manifest validated = load_inference_manifest("models/sherpa-kws", "ascend_310p") ``` Per chunk: run the encoder on 45 fbank frames plus the carried state, feed the 8 output frames to the transducer keyword decoder, keep the returned state, and slide the feature buffer forward by 32 frames. A standalone run, reading raw 16 kHz mono s16le PCM from stdin: ```bash parec --device= --rate=16000 --channels=1 --format=s16le \ | python3 live_kws.py --model-dir . --tokens tokens.txt --keywords keywords.txt ``` ### Defining your own keywords The keyword file holds **token sequences**, not plain text, so a new wake word has to be encoded first. This model is bilingual and needs both the pinyin and the English lexicon: ```bash # keywords_raw.txt: one keyword per line, spaces replaced by underscores, # and the original text after an "@", e.g.: n i3 h ao3 @你好 sherpa-onnx-cli text2token \ --tokens tokens.txt \ --tokens-type phone+ppinyin \ --lexicon en.phone \ keywords_raw.txt keywords.txt ``` Each line then accepts optional per-keyword overrides: `:score` boosts the context-graph bonus, `#threshold` sets the detection threshold for that keyword alone. `keywords_sensitive.txt` is the same keyword set at a lower threshold — more hits, more false alarms. Tune per keyword rather than globally: a short wake word needs a higher threshold than a long one. No retraining is involved; changing the wake word is a text-file change. ## License Code and packaging: Apache-2.0. Model weights follow the upstream [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) / k2-fsa license (Apache-2.0). ## Citation @article{zipformer2023, title = {Zipformer: A faster and better encoder for automatic speech recognition}, author = {Yao, Zengwei and Guo, Liyong and Yang, Xiaoyu and Kang, Wei and Kuang, Fangjun and Yang, Yifan and Jin, Zengrui and Lin, Long and Povey, Daniel}, journal = {arXiv preprint arXiv:2310.11230}, year = {2023} } @software{sherpa_onnx, title = {sherpa-onnx: Real-time speech recognition and keyword spotting}, author = {{Next-gen Kaldi (k2-fsa)}}, url = {https://github.com/k2-fsa/sherpa-onnx}, license = {Apache-2.0} } @software{ib_robot, title = {IB-Robot: Intelligence Boom Robot}, url = {https://atomgit.com/openeuler/IB_Robot}, license = {Apache-2.0} }