Hugging Face's logo

musetric
/
whisper-large-v3-onnx

Automatic Speech Recognition
Transformers.js
ONNX
whisper
audio
webgpu
Model card Files
xet
 Community
whisper-large-v3-onnx
File size: 4,667 Bytes
228b203
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
---
license: apache-2.0
library_name: transformers.js
pipeline_tag: automatic-speech-recognition
base_model: openai/whisper-large-v3
tags:
  - audio
  - automatic-speech-recognition
  - whisper
  - onnx
  - transformers.js
  - webgpu
---

# Whisper large-v3 — word-timestamped ONNX (q4 / WebGPU)

ONNX export of [**openai/whisper-large-v3**][base] in the [transformers.js][tjs]
layout, with the **cross-attention alignment heads** baked into
`generation_config.json` so the decoder emits **word-level timestamps**
(`return_timestamps: 'word'`). Packaged for the
[`musetric`][musetric] `packages/ai` runtime
([`@huggingface/transformers`][tjs] on the **WebGPU** execution provider).

[onnx-community][oc] publishes such `_timestamped` variants for
[turbo][oc-turbo], [small][oc-small] and [tiny][oc-tiny], but not for
**large-v3** — hence this export. It is produced by [`scripts/onnx/whisper`][sc]
in [musetric-toolkit][toolkit] (optimum + the [transformers.js converter][conv]).

## Files

| file | role |
|---|---|
| `encoder_model_q4.onnx` | audio encoder (q4) |
| `decoder_model_merged_q4.onnx` | decoder with merged KV-cache (q4) |
| `config.json`, `generation_config.json`, `preprocessor_config.json` | model / feature-extractor config |
| `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json`, `normalizer.json` | tokenizer |

Only the **q4** graphs are shipped; the runtime loads `dtype: 'q4'` for both the
encoder and the merged decoder.

## How to use

```ts
import { pipeline } from '@huggingface/transformers';

const asr = await pipeline(
  'automatic-speech-recognition',
  'musetric/whisper-large-v3-onnx',
  {
    device: 'webgpu',
    subfolder: '',
    dtype: { encoder_model: 'q4', decoder_model_merged: 'q4' },
  },
);
const out = await asr(audio, { return_timestamps: 'word', chunk_length_s: 30 });
```

## Intended uses & limitations

**Intended:** client/edge speech-to-text with word timings via WebGPU through
[`@huggingface/transformers`][tjs] — e.g. lyric alignment in `musetric`.

**Limitations:**
- Requires a WebGPU adapter with `shader-f16`.
- q4 weight quantization trades a little accuracy for size/speed; for
  non-English audio quality varies (the `musetric` pipeline adds language
  forcing and loop guards on top).
- Word timestamps come from the whisper cross-attention heads, not a separate
  forced aligner.

## Source & lineage

Documented only as far as it is verifiable.

- **Base weights:** [`openai/whisper-large-v3`][base]
  @ `06f233fe06e710322aca913c1bc4249a0d71fce1`. Inference-only re-export in the
  transformers.js layout; no fine-tuning.
- **Alignment heads:** the `[layer, head]` pairs correlated with word timing come
  from [hollance's gist][heads], which covers tiny through large-v2. **large-v3
  is absent there**, so its heads are taken from the
  [`generation_config.json`][base-gc] of the base model itself.
- **Converter:** the [transformers.js converter][conv] plus
  [optimum][optimum], pinned to `transformers==4.42.4` / `optimum==1.21.3` /
  `torch==2.4.1` (transformers 4.43 adds a `cache_position` decoder input that
  `@huggingface/transformers` 4.2.0 does not feed).
- **Export tooling:** [`scripts/onnx/whisper`][sc] in [musetric-toolkit][toolkit].

## License

**Apache-2.0**, following the [model card][base] the weights are downloaded from.

Upstream is inconsistent here and it is worth knowing: the
[openai/whisper][gh] repository states that "Whisper's code and model weights are
released under the MIT License", while the Hugging Face model card these weights
are actually fetched from declares `apache-2.0`. This export follows the source
it downloads from; consult both before relying on either.

[base]: https://huggingface.co/openai/whisper-large-v3
[base-gc]: https://huggingface.co/openai/whisper-large-v3/blob/main/generation_config.json
[gh]: https://github.com/openai/whisper
[heads]: https://gist.github.com/hollance/42e32852f24243b748ae6bc1f985b13a
[tjs]: https://github.com/huggingface/transformers.js
[conv]: https://github.com/huggingface/transformers.js/blob/v3/scripts/convert.py
[optimum]: https://github.com/huggingface/optimum
[oc]: https://huggingface.co/onnx-community
[oc-turbo]: https://huggingface.co/onnx-community/whisper-large-v3-turbo_timestamped
[oc-small]: https://huggingface.co/onnx-community/whisper-small_timestamped
[oc-tiny]: https://huggingface.co/onnx-community/whisper-tiny_timestamped
[sc]: https://github.com/popelenkow/musetric-toolkit/tree/main/scripts/onnx/whisper
[toolkit]: https://github.com/popelenkow/musetric-toolkit
[musetric]: https://github.com/popelenkow/musetric