| --- |
| license: other |
| license_name: nvidia-software-and-model-evaluation-license |
| license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-and-model-evaluation-license/ |
| base_model: nvidia/Nemotron-3-Diarization-preview |
| library_name: fluidaudio |
| pipeline_tag: voice-activity-detection |
| tags: |
| - speaker-diarization |
| - streaming |
| - coreml |
| - apple-silicon |
| - ane |
| - sortformer |
| --- |
| |
| # Nemotron 3 Diarization — CoreML for Apple Silicon |
|
|
| CoreML conversion of NVIDIA's [Nemotron 3 Diarization preview](https://huggingface.co/nvidia/Nemotron-3-Diarization-preview) |
| (8-speaker streaming Sortformer, 100M params, 31-layer RoPE Transformer, 10 ms output |
| resolution), optimized for on-device inference on Apple platforms via |
| [FluidAudio](https://github.com/FluidInference/FluidAudio). |
|
|
| Runs fully on-device — Neural Engine or GPU — on macOS 14+ / iOS 17+. |
|
|
| ## Highlights (M5 Pro) |
|
|
| - **Reproduces NVIDIA's model-card accuracy on-device**: AMI MHM offline 9.28 DER |
| (card 9.30), verylow 10.13 (card 10.13 — exact), AliMeeting Far 9.83 (card 11.14), |
| AliMeeting Near 6.79 (card 7.25) — same forced-alignment references, collar 0 |
| - **8-speaker streaming at 1.04 s latency**, ~11 ms ANE per chunk (`fast` preset) |
| - **`fast128`: 9.59 DER at 1004x wall real-time** (card protocol, AMI MHM) — |
| every FluidAudio preset beats the card's published low-profile number |
| - **100% ANE-resident W8A8 split-graph builds** at half the weight footprint |
| (`c128-split-w8a8`: 9.68 DER, 566x wall, ANE only — GPU stays free) |
| - Beats streaming Sortformer v2 on all 16 AMI test meetings with double the |
| speaker capacity |
|
|
| ## Which model should I use? |
|
|
| | Model | Size | Audio chunk/call | Latency | DER¹ | Per-call cost | Pros | Cons | |
| |---|---|---|---|---|---|---|---| |
| | `low` | 190 MB | 0.72 s | 1.04 s | 9.90 | 28 ms ANE / 9.6 ms GPU | Best quality at real streaming latency; NVIDIA's reference config (card: 10.35) | Heaviest ANE draw per second of audio | |
| | `fast` | 190 MB | 0.72 s | 1.04 s | 10.65 | **10.9 ms ANE** / 6.1 ms GPU | ~3x cheaper than `low` at the same latency; leaves ANE room for concurrent ASR | Worst DER of the set (+0.75 vs `low`) | |
| | `fast32` | 190 MB | 2.56 s | 2.88 s | 9.94 | 12.5 ms ANE / 6.2 ms GPU | **Recommended default** — `low`-level quality at near-`fast` cost, 343x wall | 2.88 s latency; not for live-caption UX | |
| | `fast128` | 190 MB | 10.24 s | 10.56 s | **9.59** | 20.6 ms ANE / 7.5 ms GPU | Best streaming-lineup DER; 1004x wall | 10.5 s latency — near-live only | |
| | `offline` | 190 MB | 27.2 s | 30.4 s | **9.28** | 11.2 ms GPU only | Highest accuracy (exact card match); ~1680x wall | No ANE (compiler limit); 30 s latency | |
| | `s32_split_w8a8` | **95 MB** | 2.56 s | 2.88 s | 9.86 | **9.7 ms, 100% ANE** | Half size, zero GPU use — the iOS/battery pick | Split mode needs host-side pre-encode (FluidAudio handles it) | |
| | `c128_split_w8a8` | **95 MB** | 10.24 s | 10.56 s | 9.68 | **16.1 ms, 100% ANE** | Batch throughput (566x) without touching the GPU | Same split-mode dependency; 10.5 s latency | |
|
|
| ¹ Card protocol: AMI MHM test, forced-alignment refs, collar 0 (NVIDIA's published |
| numbers under this protocol: 9.30 offline / 10.35 low). Wall RTFx single-stream on |
| M5 Pro. Quick chooser: hard ~1 s latency -> `fast` (sharing ANE) or `low`; general |
| use -> `fast32`; latency-flexible quality -> `fast128`; Mac batch -> `offline`; |
| iPhone/iPad or GPU-busy -> the `split_w8a8` pair. |
|
|
| ## Presets |
|
|
| All presets share one checkpoint; they differ in streaming shape. Latency = |
| (chunk + right-context) x 80 ms. DER: **card protocol** — AMI MHM test, 16 meetings, |
| forced-alignment references (nttcslab-sp/diar-forced-alignment), collar 0, overlap |
| included. NVIDIA's card reports 9.30 (offline) / 10.35 (low) under this protocol. |
| Full tables incl. AMI SDM and AliMeeting in BENCHMARKS.md. |
|
|
| ### Monolithic (single model file, simplest integration) |
|
|
| | Preset | Latency | Audio/call | DER (card protocol) | ANE/call | Use case | |
| |---|---|---|---|---|---| |
| | `low` | 1.04 s | 0.72 s | 9.90 | 28 ms | card config (card: 10.35) | |
| | `fast` | 1.04 s | 0.72 s | 10.65 | **10.9 ms** | latency-critical + concurrent ASR | |
| | `fast32` | 2.88 s | 2.56 s | 9.94 | 12.5 ms | **recommended streaming default** | |
| | `fast128` | 10.56 s | 10.24 s | **9.59** | 20.6 ms | near-live throughput, 1004x wall | |
| | `offline` | 30.4 s | 27.2 s | 9.28 | GPU only | batch (card: 9.30; ~1680x wall) | |
|
|
| Other configurations (sub-second latency tiers, intermediate chunk sizes, weight-only |
| int8) were built and benchmarked but are not published — they are dominated by the |
| presets above. Open an issue if you need one; regeneration is scripted. |
|
|
| ### Split-graph (host pre-encode; 100% ANE-resident CoreML subgraph) |
|
|
| Feature stacking, the 1024→512 projection (`pre_encode_proj_t.bin`), state packing, |
| and masks run host-side; the model is the pure-fp transformer+head. FluidAudio |
| handles all of this via `splitGraph` presets. |
|
|
| | Preset | Latency | DER (card protocol) | ANE/call | Weights | |
| |---|---|---|---|---| |
| | `s32_split_w8a8` | 2.88 s | 9.86 | 9.7 ms | **95 MB** — streaming on ANE (iOS pick) | |
| | `c128_split_w8a8` | 10.56 s | 9.68 | **16.1 ms** | **95 MB** — recommended ANE batch | |
|
|
| ## Usage (FluidAudio, Swift) |
|
|
| ```swift |
| import FluidAudio |
| |
| let config = Nemotron3Config.fast32 // or .preset(named: "c128-split-w8a8")! |
| let models = try await Nemotron3Models.load( |
| config: config, |
| directory: modelsDirectoryURL // this repo's files |
| ) |
| let diarizer = Nemotron3Diarizer(config: config, models: models) |
| |
| let (probs, frames) = try diarizer.processComplete(audioSamples) // 16 kHz mono |
| let segments = Nemotron3Diarizer.segments(probabilities: probs, frameCount: frames) |
| // -> arrival-ordered speaker segments at 10 ms resolution, up to 8 speakers |
| ``` |
|
|
| Optional VAD gating for sparse audio (2x wall speedup at 55% speech density): |
|
|
| ```swift |
| let (probs, frames) = try diarizer.processComplete(audioSamples, speechMask: mask) |
| ``` |
|
|
| ## Files |
|
|
| - `monolithic/` — one `.mlmodelc` per preset (mel features in, predictions out) |
| - `split/` — W8A8 transformer graphs (`packed`/`attn_bias`/`output_mask` inputs) |
| - `learnable_sil_emb.bin` — learned silence embedding, 512 x fp32 (cache compression) |
| - `pre_encode_proj_t.bin` — FeatureStacking projection W^T, 1024x512 fp32 (split mode) |
|
|
| All models: fixed shapes, fp16 weights unless noted, `minimum_deployment_target` iOS 17 / |
| macOS 14. Conversion verified against the NeMo reference at ≤1e-4 single-chunk parity |
| and 99.995% closed-loop frame agreement on real audio. |
|
|
| ## Attribution & license |
|
|
| Converted from [nvidia/Nemotron-3-Diarization-preview](https://huggingface.co/nvidia/Nemotron-3-Diarization-preview) |
| and published with NVIDIA's permission; model weights remain subject to NVIDIA's |
| license terms (see `license_link` — currently the evaluation license, to be updated |
| at the model's general release). |
| Conversion pipeline, Swift runtime, and benchmarks by |
| [Fluid Inference](https://huggingface.co/FluidInference). |
|
|
| Reference: [Streaming Sortformer](https://arxiv.org/abs/2507.18446). |
|
|
