ultravox-qwen3.6-27b-v2

A speech-language model for phone-call voice agents, built on the Ultravox architecture: a frozen Qwen 3.6-27B LLM that consumes audio directly through a Whisper-large-v3-turbo encoder + projection adapter (LoRA r=16 on the encoder; knowledge-distillation training against the text LLM as teacher).

Trained by QuantumDesk for receptionist/sales phone agents: heavy real-telephony and entity-dense data rather than multilingual benchmark balance. This is the production successor to ultravox-qwen3.6-27b-base (the early English-only research checkpoint) — use this one for real voice-agent work.

Benchmarks (native inference, greedy, n=100 per set)

Metric	v2	ultravox-v0_6-gemma-3-27b (same rig)
LibriSpeech test-clean WER	2.08%	1.90%
Switchboard (real phone calls) WER	9.67%	21.2%
Phone-number digit accuracy (phone-quality audio)	97.0%	98.5%
Person-name accuracy	78.8%	60.6%
Spelled-email accuracy	53.2%	50.6%
Business-name accuracy	91.2%	73.5%

On conversational phone audio — the target domain — this model roughly halves the word error rate of general-purpose Ultravox models, while matching digit accuracy and substantially exceeding name recognition.

Conversational use

v2 responds natively to audio (not transcription-only). It is trained on this exact user turn shape:

[
  {"role": "system", "content": "<your agent persona>"},
  {"role": "user", "content": [
    {"type": "text", "text": "Respond conversationally to what the user just said. Keep replies short — one or two natural sentences. Never use markdown, code, symbols, or headers. Never invent quotes. If you didn't catch what they said, ask them to repeat. "},
    {"type": "input_audio", "input_audio": {"data": "<b64 wav>", "format": "wav"}}
  ]}
]

For transcription use: "Repeat the following text, without any explanation: " + audio. A persona system prompt is strongly recommended for conversational use.

Training

Two phases from scratch on 2×H200:

1. Phase 1B (~35K steps, eff. batch 16): English ASR corpora + real telephone conversations, with telephony augmentation (8kHz resample, AMR/μ-law compression, band-pass, pink noise, gain).
2. Phase 2 (15K steps): 150K synthetic entity-dense utterances (phone numbers, names, spelled emails, addresses, dates/times across 31 TTS voices) + 40K spoken-dialogue response pairs + anti-forgetting garnish from phase-1 corpora.

Serving

Works with vLLM (--trust-remote-code). Note: Qwen 3.6's hybrid linear attention currently loses some accuracy under vLLM relative to native transformers inference; for maximum quality use HF transformers, or vLLM with --no-enable-chunked-prefill and default (fp32) mamba cache dtypes.

Known limitations

• Short utterances degrade sharply. The benchmarks above are sentence-length inputs. On isolated sub-second turns — "Yes", "Hello", "Okay", single-word backchannels — the model frequently mistranscribes or hallucinates (observed in live deployment; the training mix is dominated by sentence-length corpora). If you use this in a live agent, buffer very short VAD segments with surrounding context, or route turns under ~1.5 s through a fallback. A v3 addressing this with short-turn training data is in progress.
• English only — degraded quality on other languages and heavily accented English.
• Response style is trained to the prompt shape shown above; deviating from it (especially for conversational use) reduces quality.

Apache 2.0. Built with the Ultravox training framework by Fixie.ai.