kdrkdrkdr/nemotron-3.5-asr-streaming-0.6b-w8a8

Nemotron 3.5 ASR Streaming 0.6B — W8A8 (pure-C runtime weights)

This repository hosts a W8A8 (packed Q8P int8) conversion of nvidia/nemotron-3.5-asr-streaming-0.6b, prepared for the dependency-free pure-C streaming inference runtime at github.com/kdrkdrkdr/nemotron-asr-streaming.c.

It is a single memory-mappable binary (nemotron-3.5-asr-streaming-0.6b-w8a8-linear.bin, ~0.62 GiB). Dense linear, LSTM, and vocabulary-classifier weights are stored as per-output-row int8 with 32-bit row scales, packed in four-row tiles with a 16-byte padded input stride (the runtime's "Q8P" format); the remaining tensors (layer norms, biases, depthwise convolutions, mel front-end, prediction embedding) stay float32. Activations are dynamically quantized to int8 at the typed dense call sites at inference time.

Usage

# 1. Build the runtime (macOS / Linux / Windows MSYS2+clang)
git clone https://github.com/kdrkdrkdr/nemotron-asr-streaming.c
cd nemotron-asr-streaming.c
make

# 2. Download this converted model
huggingface-cli download kdrkdrkdr/nemotron-3.5-asr-streaming-0.6b-w8a8 \
  nemotron-3.5-asr-streaming-0.6b-w8a8-linear.bin --local-dir .

# 3. Transcribe a WAV file
./nemotron_asr -m nemotron-3.5-asr-streaming-0.6b-w8a8-linear.bin \
  -i audio.wav -l auto --strip-tags

Live streaming from a microphone — pipe raw s16le 16 kHz mono into --stdin:

# macOS shown; Linux: `-f alsa -i default`, Windows: `-f dshow -i audio="<device>"`
ffmpeg -f avfoundation -i ":0" -ac 1 -ar 16000 -f s16le - | \
  ./nemotron_asr -m nemotron-3.5-asr-streaming-0.6b-w8a8-linear.bin \
    --stdin -l auto --strip-tags --att-right 1

--att-right sets the streaming latency (1 = 160 ms, default 3 = 320 ms). See the runtime repository for full options, per-OS capture flags, the Windows pipe caveat, and architecture notes.

Model facts

• Architecture: 24-layer cache-aware FastConformer encoder + RNN-T (2-layer LSTM prediction network) decoder, d_model=1024, 8 attention heads.
• Audio: 16 kHz mono; 128 log-mel bins (n_fft=512, win=400, hop=160).
• Vocabulary: 13087 tokens (+ blank); multilingual language prompts.
• File format: little-endian tensor stream, magic NM35ASR, version 1.

Provenance & license

• Base model: nvidia/nemotron-3.5-asr-streaming-0.6b by NVIDIA. All model capabilities, training, and intellectual property belong to the original authors.
• This repository redistributes a format-converted and int8-quantized copy of those weights for CPU inference. No retraining or fine-tuning was performed.
• License: OpenMDW 1.1, the same license as the base model. This redistribution retains that license and attribution. Please review the base model card for intended use, limitations, bias, safety, and privacy notes.

Conversion was produced with convert_nemo.py (included in this repository) run as python convert_nemo.py <model>.nemo --w8a8-linear-weights.