---
license: cc-by-nc-sa-3.0
language:
  - es
pipeline_tag: automatic-speech-recognition
library_name: icefall
tags:
  - icefall
  - zipformer
  - streaming
  - phoneme-recognition
  - automatic-speech-recognition
datasets:
  - bookbot/common-voice-23-0-es-ipa-v1
  - bookbot/slr72-es-ipa-v1
---

# Pruned Stateless Zipformer RNN-T Streaming Robust ES v1

`zipformer-streaming-robust-es-v1` is Bookbot's causal Spanish pruned-stateless Zipformer RNN-T phoneme recognizer. It emits atomic, whitespace-interpretable IPA phone tokens rather than words. The canonical inference state is the exact cumulative averaged-model difference over epochs 76–80; its 639 tensors were checked against the production Icefall averaging path before the source checkpoints were pruned.

## License and permitted use

The model weights are published under **CC BY-NC-SA 3.0** because training included the [TinyVox](https://talkbank.org/phon/access/Derived/TinyVox.html) Spanish subset, which is derived from TalkBank/PhonBank data under that license and is restricted to non-commercial use. Do not use these weights in a commercial product or service. Follow the [TalkBank Ground Rules](https://talkbank.org/0share/rules.html), retain attribution, and share adaptations under compatible terms.

The included Icefall inference script retains its upstream Apache-2.0 source-code license. Common Voice and SLR72 retain their own source licenses. TinyVox audio, metadata, and utterance-level recognition output are **not** included in this repository or the companion public datasets.

## Training data

- [Common Voice 23.0 Spanish](https://mozilladatacollective.com/datasets/cmj8u3pe600fhnxxbqol7x8bg), CC0-1.0.
- [OpenSLR72 Colombian Spanish](https://www.openslr.org/72/), CC BY-SA 4.0.
- [TinyVox Spanish](https://talkbank.org/phon/access/Derived/TinyVox.html), CC BY-NC-SA 3.0; source-hosted only.

Versioned public training datasets with the exact v1 schema are:

- [`bookbot/common-voice-23-0-es-ipa-v1`](https://huggingface.co/datasets/bookbot/common-voice-23-0-es-ipa-v1)
- [`bookbot/slr72-es-ipa-v1`](https://huggingface.co/datasets/bookbot/slr72-es-ipa-v1)

Each row contains only `id`, `audio`, and the exact canonical `ipa_transcript` consumed by this model.

## Output inventory

`data/lang_phone/tokens.txt` is the authoritative symbol table. It contains 28 non-disambiguation entries, including `<eps>` and `<UNK>`, plus `#0` for graph disambiguation. Phones are atomic tokens; the composite affricate `t͡ʃ` is one token. The vocabulary differs from v0's older BabyGruut-era inventory and is not numerically compatible with v0 checkpoints.

## Streaming configuration

- Architecture: pruned stateless Zipformer RNN-T
- Causal streaming: yes
- Chunk size: 16 encoder frames
- Left context: 128 frames
- Encoder layers: `2,2,2,2,2,2`
- Feed-forward dimensions: `512,768,768,768,768,768`
- Encoder dimensions: `192,256,256,256,256,256`
- Encoder unmasked dimensions: `192,192,192,192,192,192`
- Decoder and joiner dimensions: 512
- Training hardware: 2 NVIDIA RTX 4090 GPUs

## Published artifacts

- `epoch-80.pt`: full resumable final training state.
- `evaluated-avg-76-80.pt`: canonical inference checkpoint used for every reported result.
- `jit_script_chunk_16_left_128.pt`: TorchScript export from that exact canonical checkpoint.
- `data/lang_phone/`: token, lexicon, and graph artifacts.
- `jit_pretrained_streaming.py`: streaming greedy inference example.
- `test_waves/common_voice_es_19696062.wav`: CC0 Common Voice 23.0 Spanish sample, resampled to 16 kHz.
- `streaming/evaluation-common-voice-23-0-es-slr72.json`: machine-readable corpus provenance and results.
- `model-config.json` and `SHA256SUMS`: reproducibility metadata.
- `tensorboard/` and `log/`: training metrics and logs.

The sherpa-onnx ORT conversion is published separately at [`bookbot/sherpa-onnx-ort-zipformer-streaming-robust-es-v1`](https://huggingface.co/bookbot/sherpa-onnx-ort-zipformer-streaming-robust-es-v1).

## Setup and usage

Install Icefall and its dependencies following the [Icefall installation guide](https://icefall.readthedocs.io/en/latest/installation/index.html). Bookbot uses its Icefall fork:

```bash
git clone https://github.com/bookbot-hive/icefall
cd icefall/egs/bookbot_es/ASR
```

### Download and run the pre-trained model

Clone this repository and run the included sample:

```bash
git lfs install
mkdir -p tmp
git clone \
  https://huggingface.co/bookbot/zipformer-streaming-robust-es-v1 \
  tmp/zipformer-streaming-robust-es-v1

python tmp/zipformer-streaming-robust-es-v1/jit_pretrained_streaming.py \
  --nn-model-filename tmp/zipformer-streaming-robust-es-v1/jit_script_chunk_16_left_128.pt \
  --tokens tmp/zipformer-streaming-robust-es-v1/data/lang_phone/tokens.txt \
  tmp/zipformer-streaming-robust-es-v1/test_waves/common_voice_es_19696062.wav
```

Observed greedy output for the included sample:

```text
xeneɾalmentelosimmigɾantessesitwaɾonenlapeɾifeɾjadelassjudades
```

The underlying token sequence exactly matches the production checkpoint's stored greedy hypothesis:

```text
x e n e ɾ a l m e n t e l o s i m m i g ɾ a n t e s s e s i t w a ɾ o n e n l a p e ɾ i f e ɾ j a d e l a s s j u d a d e s
```

Input audio must be mono 16 kHz WAV or another format supported by `torchaudio` with a 16 kHz sample rate.

## Evaluation

All values are **phoneme error rate (PER)**. Icefall's generic error writer and historical `wer-summary-*` filenames use the label WER, but the aligned units are whitespace-separated IPA phones.

Evaluation uses the exact epochs 76–80 averaged model, causal chunk size 16, left context 128, and 1,000 decode streams.

| Decoder | Common Voice 23.0 ES test | SLR72 test | TinyVox validation | TinyVox test |
|---|---:|---:|---:|---:|
| Greedy | 2.8076% | 1.5325% | 25.5180% | 33.7509% |
| Modified beam, 4 active paths | **2.6908%** | **1.3961%** | **24.5529%** | **32.9158%** |
| Fast beam, beam 8 / 4 contexts / 32 states / blank penalty 3.1 | 3.1927% | 1.5544% | 36.8720% | 45.7883% |
| Reference fast beam, beam 4 / 4 contexts / 32 states / blank penalty 0 | 74.9781% | 50.2461% | 85.2399% | 88.0634% |

Use **modified beam search with `num_active_paths=4`** for this model. The reference fast-beam configuration collapses through excessive blank/deletion bias. A blank penalty selected only on TinyVox validation rescues that collapse but remains worse than modified beam on every reported corpus.

### Evaluation limits

The recipe reuses the Common Voice and SLR72 official test splits for validation, so those two results are validation-overlapped rather than independent holdouts. TinyVox test is the independently held-out result; it was evaluated after the fast-beam setting had been selected on TinyVox validation. Common Voice dominates any combined aggregate, so per-corpus values are preferred.

## Checkpoint provenance

- Best recorded validation loss: `0.1675387544560873` at epoch 75.
- Training completed the configured 80 epochs; this completed run did not early-stop.
- Canonical averaged inference range: epochs 76–80.
- `epoch-80.pt` SHA-256: `e6704a4c639f94e7361fb07d6fac698d3ea0c67d8f3eecef4ad6c0423013b646`.
- `evaluated-avg-76-80.pt` SHA-256: `d5cde97d82c214da39c3d9decffa84eaf6b15869c7b6521d2f184a2e59ad30e5`.

See `SHA256SUMS` for every packaged binary and `model-config.json` for exact export settings.

## Training procedure

The historical run used Python 3.12, PyTorch 2.4.0+cu124, k2 1.24.4,
Lhotse 1.31.1, and two NVIDIA RTX 4090 GPUs. The retained checkpoint records
the clean Icefall branch `feature/spanish-phonemization-retrain` at
`4b746f3b`. That recipe revision is the authority for the historical run;
the current launcher may contain later checkpoint-retention and early-stopping
policy that did not produce these weights.

### Install Icefall

```bash
git clone https://github.com/bookbot-hive/icefall
cd icefall
export PYTHONPATH="$PWD:${PYTHONPATH:-}"
cd egs/bookbot_es/ASR
```

Install Icefall and its k2/Lhotse dependencies using the
[official installation guide](https://icefall.readthedocs.io/en/latest/installation/index.html).
The exact `4b746f3b` recipe revision must be available in your checkout for
historical data preparation. At publication time that feature revision was
recorded by the checkpoint but was not reachable from the public fork, so this
is an explicit source-reproducibility limit rather than a claim that current
`master` is identical.

### Prepare data

The two public companion datasets expose the accepted Common Voice and SLR72
rows, but exact training additionally requires:

- pinned full-source snapshots obtained from
  [Mozilla Data Collective](https://mozilladatacollective.com/datasets/cmj8u3pe600fhnxxbqol7x8bg)
  and [OpenSLR72](https://www.openslr.org/72/), identified by historical
  preparation revisions `591a99e9c5c9e308231746c7baa03eadfe670483` and
  `e7c9924b964eecbdcba1181add860367d3c7ca3a`, respectively;
- an existing verified Common Voice/SLR72 Lhotse fbank store;
- locally obtained TinyVox Spanish data and the licensed alignment lexicon.

Bookbot no longer hosts those full historical source snapshots. Exact
preparation therefore requires preserved local copies matching the recorded
revision and manifest hashes. The versioned IPA datasets above contain only
accepted final cuts and cannot replace the larger full-source inputs in this
historical preparation command.

TinyVox is not redistributed here. Obtain it from TalkBank and comply with
CC BY-NC-SA 3.0 and the TalkBank Ground Rules. With those prerequisites, the
historical preparation stages are:

```bash
python local/prepare_spanish_retrain.py \
  --existing-fbank-dir /path/to/existing/fbank \
  --output-data-dir data \
  --common-voice-source /path/to/common_voice_23_0_es \
  --slr72-source /path/to/slr72_dataset \
  --tinyvox-root /path/to/tinyvox_spanish \
  --tinyvox-alignment-lexicon local/tinyvox_alignment_lexicon.tsv

python local/compute_fbank_tinyvox.py

python local/prepare_lexicon.py \
  --tokens-file es-tokens.txt \
  --lang-dir data/lang_phone

python local/prepare_lang.py \
  --tokens-file es-tokens.txt \
  --lang-dir data/lang_phone

python local/validate_spanish_retrain.py \
  --fbank-dir data/fbank \
  --reports-dir data/reports \
  --tokens-file es-tokens.txt \
  --lang-tokens data/lang_phone/tokens.txt
```

Preparation is fail-closed: pinned source revisions and manifests, full audio
joins, the fixed phone inventory, and TinyVox split isolation must validate
before training starts.

### Train

This is the effective historical command recorded by `epoch-80.pt`. The
checkpoint confirms `keep_last_k=30`, seed 42, causal training chunk choices
`16,32,64,-1`, left-context choices `64,128,256,-1`, MUSAN mixing, and
SpecAugment. No early stopping was active in the run that produced v1.

```bash
export CUDA_VISIBLE_DEVICES="0,1"
export OMP_NUM_THREADS=1

python zipformer/train.py \
  --world-size 2 \
  --num-epochs 80 \
  --start-epoch 1 \
  --exp-dir zipformer/exp-spanish-atomic-phones \
  --manifest-dir data/fbank \
  --lang-dir data/lang_phone \
  --causal 1 \
  --chunk-size "16,32,64,-1" \
  --left-context-frames "64,128,256,-1" \
  --num-encoder-layers 2,2,2,2,2,2 \
  --feedforward-dim 512,768,768,768,768,768 \
  --encoder-dim 192,256,256,256,256,256 \
  --encoder-unmasked-dim 192,192,192,192,192,192 \
  --max-duration 1000 \
  --base-lr 0.04 \
  --keep-last-k 30 \
  --seed 42 \
  --enable-musan true \
  --enable-spec-aug true \
  --use-transducer true \
  --use-fp16 1
```

The run completed epoch 80. `evaluated-avg-76-80.pt` is the materialized
Icefall cumulative averaged-model difference for epochs 76–80, not a renamed
single-epoch checkpoint.

### Exporting to ONNX

The streaming exporter emits FP32 and dynamic-INT8 encoder, decoder, and
joiner ONNX files. Starting in `icefall/egs/bookbot_es/ASR`, expose the exact
materialized average through a scratch numeric checkpoint name:

```bash
native=tmp/zipformer-streaming-robust-es-v1
scratch="$(mktemp -d)"
ln -s "$(realpath "$native/evaluated-avg-76-80.pt")" \
  "$scratch/epoch-999.pt"

python zipformer/export-onnx-streaming.py \
  --tokens "$native/data/lang_phone/tokens.txt" \
  --epoch 999 \
  --avg 1 \
  --use-averaged-model false \
  --exp-dir "$scratch" \
  --num-encoder-layers 2,2,2,2,2,2 \
  --downsampling-factor 1,2,4,8,4,2 \
  --feedforward-dim 512,768,768,768,768,768 \
  --num-heads 4,4,4,8,4,4 \
  --encoder-dim 192,256,256,256,256,256 \
  --query-head-dim 32 \
  --value-head-dim 12 \
  --pos-head-dim 4 \
  --pos-dim 48 \
  --encoder-unmasked-dim 192,192,192,192,192,192 \
  --cnn-module-kernel 31,31,15,15,15,31 \
  --decoder-dim 512 \
  --joiner-dim 512 \
  --context-size 2 \
  --causal true \
  --chunk-size 16 \
  --left-context-frames 128 \
  --use-transducer true

for component in encoder decoder joiner; do
  mv "$scratch/${component}-epoch-999-avg-1-chunk-16-left-128.onnx" \
    "$scratch/${component}-evaluated-avg-76-80-chunk-16-left-128.onnx"
  mv "$scratch/${component}-epoch-999-avg-1-chunk-16-left-128.int8.onnx" \
    "$scratch/${component}-evaluated-avg-76-80-chunk-16-left-128.int8.onnx"
done
```

The exporter uses ONNX opset 13 and
`onnxruntime.quantization.quantize_dynamic` with QInt8 weights. It quantizes
encoder `MatMul`, decoder `MatMul`/`Gather`, and joiner `MatMul` operations.

### Converting ONNX to ORT

The published package used ONNX Runtime 1.23.2 with fixed optimization:

```bash
(
  cd "$scratch"
  python -m onnxruntime.tools.convert_onnx_models_to_ort \
    --optimization_style=Fixed \
    .
)
```

This produces six `.ort` files plus `required_operators.config`. Verify the
ORT files before publication, then remove the intermediate ONNX files and the
scratch checkpoint alias:

```bash
python -c 'import glob, sys, onnxruntime as ort; [ort.InferenceSession(p, providers=["CPUExecutionProvider"]) for p in glob.glob(sys.argv[1] + "/*.ort")]' "$scratch"
rm -f "$scratch"/*.onnx "$scratch"/epoch-999.pt
```

The final, validated ORT artifacts are published in
[`bookbot/sherpa-onnx-ort-zipformer-streaming-robust-es-v1`](https://huggingface.co/bookbot/sherpa-onnx-ort-zipformer-streaming-robust-es-v1).
`export-config.json` in that repository records the architecture, quantization
operators, conversion settings, and output names.

## Exporting the TorchScript model

The materialized average can be exposed to Icefall's numeric checkpoint loader through a temporary alias:

```bash
ln -s "$PWD/evaluated-avg-76-80.pt" /tmp/bookbot-v1-export/epoch-999.pt

python zipformer/export.py \
  --tokens data/lang_phone/tokens.txt \
  --use-averaged-model false \
  --epoch 999 --avg 1 \
  --exp-dir /tmp/bookbot-v1-export \
  --jit true --causal true \
  --chunk-size 16 --left-context-frames 128 \
  --num-encoder-layers 2,2,2,2,2,2 \
  --feedforward-dim 512,768,768,768,768,768 \
  --encoder-dim 192,256,256,256,256,256 \
  --encoder-unmasked-dim 192,192,192,192,192,192 \
  --use-transducer true
```

Remove the temporary alias after export. It is not a third retained checkpoint.

## Frameworks

- [Icefall](https://github.com/bookbot-hive/icefall)
- [k2](https://github.com/k2-fsa/k2)
- [Lhotse](https://github.com/lhotse-speech/lhotse)
- [PyTorch](https://pytorch.org/)

## Citations

For Common Voice, OpenSLR72, TinyVox, and TalkBank, follow the citation requirements on the linked source pages. In particular, SLR72 derives from Guevara-Rukoz et al., *Crowdsourcing Latin American Spanish for Low-Resource Text-to-Speech* (LREC 2020), and TinyVox is described by Charlot et al. (2025) and Lavechin, Bergelson, and Levy (2026).