Danny-Dasilva/neutts-2e-onnx

NeuTTS-2E — ONNX

Original model: neuphonic/neutts-2e by Neuphonic.

Community ONNX export (FP32 + dynamic-int8) of NeuTTS-2E, Neuphonic's on-device emotional English TTS model (~236M params, Qwen3-architecture backbone + NeuCodec).

• Speakers: emily, paul, sophie, steven (fixed, pre-encoded references ship with the neutts package)
• Emotions: angry, disgusted, fearful, happy, neutral, sad, surprised
• Output: 24 kHz audio, watermarked by default with Perth

This is an unofficial conversion, not affiliated with Neuphonic. Official releases: torch (original), q8 gguf, q4 gguf.

Files

File	Size	Fidelity vs torch fp32	Notes
model.onnx	~1.4 GB	max logits diff < 1e-4 (measured 3.8e-05), argmax-identical	FP32 export with KV cache — recommended for quality
model_int8.onnx	~350 MB	logits diverge measurably (per-step top-50 overlap ≈ 50–80%)	Dynamic int8 — ~1.7× faster; transcription-accurate in testing but not bit-comparable to fp32; listen before production use
onnx_infer.py	—	—	Self-contained inference example (generation loop + NeuCodec decode)

A 4-bit MatMulNBits variant was also built and tested but is not shipped: it decoded at the same speed as int8 (45.1 vs 45.3 tok/s) while being 1.7× larger and less accurate (argmax agreement 14/21 on teacher-forced steps), with no configuration (fp32/int8 lm_head, block size 32) beating int8 on any axis.

Benchmarks

Measured with bench_onnx.py on an idle AMD Ryzen Threadripper 7970X (32C/64T), onnxruntime 1.27.0 (CPU EP), prompt 454 tokens, 250 decode steps. Decode timing includes top-k/top-p/min-p sampling over the 217k vocab; it excludes prefill (shown separately) and NeuCodec decode (~0.2 s per 5 s of audio, ~4%). NeuCodec produces 50 frames/s of audio, so ×RT = tok/s ÷ 50 (higher is better).

Model	Threads	Prefill	Decode	Speed vs real-time
model.onnx (fp32)	4	564 ms	27.5 tok/s	0.55×
model.onnx (fp32)	default (64)	480 ms	14.9 tok/s	0.30×
model_int8.onnx	3	367 ms	45.3 tok/s	0.91×
model_int8.onnx	4	357 ms	40.6 tok/s	0.81×
model_int8.onnx	default (64)	343 ms	18.6 tok/s	0.37×

Set intra_op_num_threads to 3–4. Decode throughput peaks there and collapses at higher counts (the ONNX Runtime default — all cores — is 2–3× slower than the peak for this small-hidden-size model). onnx_infer.py defaults to 4 threads (--threads).

Conversion details

Exported with optimum-cli export onnx --task text-generation-with-past (optimum 2.1.0 / optimum-onnx 0.1.0, fp32). Export validated against the torch model: fp32 prefill logits max abs diff 3.8e-05 with identical argmax, and 21 teacher-forced decode steps with 21/21 argmax agreement and 10/10 top-10 overlap. Note the neutts library runs the original backbone in bfloat16, which itself deviates from the fp32 reference by up to 0.136 in logit space — the FP32 ONNX model is numerically closer to the fp32 weights than the official pipeline. The int8 model is a dynamic-range quantization (onnxruntime.quantization.quantize_dynamic, QInt8 weights).

Because ONNX cannot tie weights, the shared embedding/LM-head matrix of the original model is materialized twice in the FP32 graph — that is why the FP32 file is larger than the original safetensors. One config change was made for export tooling compatibility: the transformers-5 rope_parameters block was flattened back to the legacy rope_theta: 10000.0 key.

The neutts library does not support ONNX backbones directly (only ONNX codec decoders), so this repo ships onnx_infer.py, which reproduces the library's prompt format exactly (verified token-for-token against neutts' _apply_chat_template, including the emotion-token path and quote normalization).

Generation defaults

onnx_infer.py samples with temperature 1.0, top-k 50, top-p 0.95, min-p 0.05. The official torch backend uses pure top-k 50; the official GGUF path additionally inherits llama.cpp's default top-p 0.95 / min-p 0.05, and in A/B testing that chain scored slightly better (mean +0.21 PESQ over 4 matched pairs), so it is the default here — pass --top-p 1.0 --min-p 0 for exact torch-backend behavior. min_new_tokens=50 forces ≥1 s of audio even for very short inputs (matching the library), and the 2048-token context bounds a single generation to roughly 30 s of audio. As with the original model, output is stochastic: occasional bad draws (a slurred word, a trailing artifact) happen at any precision — re-run with a different --seed if a take sounds off.

Usage

pip install neutts onnxruntime soundfile
hf download Danny-Dasilva/neutts-2e-onnx --local-dir neutts-2e-onnx
cd neutts-2e-onnx

python onnx_infer.py --model model.onnx --tokenizer . \
    --text "I can't believe it's finally here!" \
    --speaker emily --emotion happy --out test.wav

NeuCodec weights are fetched from the Hub on first run. Pre-encoded speaker references are loaded from the installed neutts package. Use --model model_int8.onnx for the faster int8 variant and --codec neuphonic/neucodec-onnx-decoder to run the codec decoder under ONNX Runtime as well (torch is still required to load the .pt speaker references and by the neucodec/neutts packages).

Voice samples

Generated with the FP32 model.onnx in this repo (default sampler above, seed 1234 — except paul/sophie neutral, re-rolled with seed 4321), decoded with the torch NeuCodec decoder, Perth-watermarked. All four voices are included, each with a neutral sample and one emotional sample, plus two samples from model_int8.onnx for quality comparison. Every sample below was verified to transcribe exactly back to its input text with Whisper (base.en).

Model	Voice	Emotion	Text	Audio
fp32	emily	neutral	The quick brown fox jumps over the lazy dog, and honestly, it never gets old.
fp32	emily	happy	I can't believe it, this is the best news I've heard all year!
fp32	paul	neutral	The quick brown fox jumps over the lazy dog, and honestly, it never gets old.
fp32	paul	angry	How many times do I have to say it? This is completely unacceptable!
fp32	sophie	neutral	The quick brown fox jumps over the lazy dog, and honestly, it never gets old.
fp32	sophie	sad	I really thought things would turn out differently this time.
fp32	steven	neutral	The quick brown fox jumps over the lazy dog, and honestly, it never gets old.
fp32	steven	surprised	Wait, what? I had absolutely no idea that was even possible!
int8	emily	neutral	The quick brown fox jumps over the lazy dog, and honestly, it never gets old.
int8	sophie	sad	I really thought things would turn out differently this time.

License

Distributed under the NeuTTS Open License v1.0 (free research use; commercial use limited to entities under $5M annual revenue — see LICENSE). Modification notice per the license: this repository contains an ONNX export (and an int8-quantized variant) of the original safetensors weights; conversion changes are described above.