mweinbach/nemotron-3.5-asr-streaming-0.6b-coreai

Nemotron 3.5 ASR Streaming 0.6B for CoreAI

This repository contains a CoreAI conversion of nvidia/nemotron-3.5-asr-streaming-0.6b, portable source assets, and device-architecture-specific AOT artifacts for iOS 27+ and macOS 27+.

Use the companion Swift package: mweinbach/NemotronCoreAI.

Streaming behavior

The runtime preserves the model's cache-aware streaming contract rather than re-running whole utterances:

• incremental centered-STFT/log-mel frontend;
• persistent 24-layer encoder channel and time caches;
• persistent two-layer RNNT predictor hidden and cell state;
• greedy RNNT decoding with complete-prefix partial transcripts;
• packet-stable 44.1/48 kHz to 16 kHz resampling in Swift;
• explicit final padding and terminal sentinel at end of stream.

Source assets expose 80, 160, 320, 560, and 1120 ms encoder modes. AOT assets specialize the balanced 320 ms mode.

Artifact matrix

Platform	Architecture	Hardware	GPU AOT
macOS	h15d	M3 Ultra	yes
macOS	h16g	M4	yes
macOS	h17g	M5	yes
macOS	h17s	M5 Pro	yes
macOS	h17c	M5 Max	yes
iOS	h18p	A19 / A19 Pro family	yes

The Swift loader checks AIModel.deviceArchitectureName, platform, compute preference, and main.hash before loading an AOT asset. A missing or future architecture can use a source .aimodel and CoreAI's device cache instead.

The Python runtime applies the same GPU/ANE guard. It detects known macOS chip names or accepts an explicit architecture override, validates main.hash, and falls back from a missing or failed GPU AOT load to source specialization.

Artifacts under aot/macos/ and aot/ios/ are not interchangeable even when an architecture identifier appears on both platforms.

Compute choices

• GPU: Published AOT artifacts were compiled with --preferred-compute gpu and are the validated low-latency path.
• Automatic: Portable source assets work with CoreAI's default compute policy. On the tested M3 Ultra, the current graph resolves to the same MPSGraph/GPU implementation as explicit GPU.
• Neural Engine: Not falsely advertised as supported. With Xcode 27 beta, forced ANE inference aborts and AOT compilation fails verification on a rank-6 transpose generated around the cache-aware encoder. The Swift API reserves the ANE lane but refuses it unless a future ANE-authored artifact is present.

Download one target

Install the current Hugging Face CLI, then download only the manifest, runtime contract, and architecture needed by the device.

hf download mweinbach/nemotron-3.5-asr-streaming-0.6b-coreai \
  --local-dir ./NemotronModel \
  --include package-manifest.json \
  --include runtime-support.json \
  --include 'aot/macos/gpu/*h15d.aimodelc/**'

For iPhone 17, iPhone Air, iPhone 17 Pro, and iPhone 17 Pro Max, use:

hf download mweinbach/nemotron-3.5-asr-streaming-0.6b-coreai \
  --local-dir ./NemotronModel \
  --include package-manifest.json \
  --include runtime-support.json \
  --include 'aot/ios/gpu/*h18p.aimodelc/**'

For forward-compatible source specialization, download a source directory and pass its manifest key as sourceVariant:

hf download mweinbach/nemotron-3.5-asr-streaming-0.6b-coreai \
  --local-dir ./NemotronModel \
  --include package-manifest.json \
  --include runtime-support.json \
  --include 'deployment-lut-quality/**'

Swift example

import NemotronCoreAI

let session = try await NemotronCoreAI.loadSession(
    packageURL: modelDirectory,
    latencyMS: 320,
    computePreference: .gpu
)

try await session.beginPCMStream()
let partial = try await session.pushPCM(packet, sampleRate: 48_000)
let final = try await session.finishPCMStream()
print(partial.text, final.text)

Python service

The included FastAPI service exposes batch transcription and a stateful WebSocket streaming endpoint while sharing one serialized CoreAI dispatcher.

PYTHONPATH=src python -m nemotron_coreai.service . \
  --compute gpu --architecture h15d --host 127.0.0.1 --port 8000

--architecture is optional on a recognized macOS chip and useful for explicit deployment configuration. --compute automatic uses portable source specialization. --compute neural-engine is rejected before CoreAI load until the package contains an exact validated ANE AOT artifact.

Source variants

Manifest key	Approximate size	Intended use
fp16	1.19 GiB	full five-mode reference
fp16_320ms	1.19 GiB	AOT-compatible 320 ms fallback
lut_quality	663 MiB	LUT8 portable quality/size balance
lut_compact	538 MiB	LUT6 group-4 compact deployment
int8	664 MiB	compatibility comparison

See reports/ and docs/VALIDATION.md for numerical and streaming validation.

License and attribution

The model materials and compiled derivatives are distributed under OpenMDW-1.1. MODEL_LICENSE contains the complete retained license, and NOTICE retains NVIDIA's copyright and source attribution. Runtime and conversion software is MIT-licensed.