license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: audio-text-to-text
base_model:
- Qwen/Qwen3.6-27B
- openai/whisper-large-v3-turbo
tags:
- audio
- speech
- speech-language-model
- ultravox
- qwen
- telephony
- voice-agent
- automatic-speech-recognition
ultravox-qwen3.6-27b-v2
A speech-language model for phone-call voice agents, built on the Ultravox architecture: a frozen Qwen 3.6-27B LLM that consumes audio directly through a Whisper-large-v3-turbo encoder + projection adapter (LoRA r=16 on the encoder; knowledge-distillation training against the text LLM as teacher).
Trained by QuantumDesk for receptionist/sales phone agents: heavy real-telephony and entity-dense data rather than multilingual benchmark balance. This is the production successor to ultravox-qwen3.6-27b-base (the early English-only research checkpoint) — use this one for real voice-agent work.
Benchmarks (native inference, greedy, n=100 per set)
| Metric | v2 | ultravox-v0_6-gemma-3-27b (same rig) |
|---|---|---|
| LibriSpeech test-clean WER | 2.08% | 1.90% |
| Switchboard (real phone calls) WER | 9.67% | 21.2% |
| Phone-number digit accuracy (phone-quality audio) | 97.0% | 98.5% |
| Person-name accuracy | 78.8% | 60.6% |
| Spelled-email accuracy | 53.2% | 50.6% |
| Business-name accuracy | 91.2% | 73.5% |
On conversational phone audio — the target domain — this model roughly halves the word error rate of general-purpose Ultravox models, while matching digit accuracy and substantially exceeding name recognition.
Conversational use
v2 responds natively to audio (not transcription-only). It is trained on this exact user turn shape:
[
{"role": "system", "content": "<your agent persona>"},
{"role": "user", "content": [
{"type": "text", "text": "Respond conversationally to what the user just said. Keep replies short — one or two natural sentences. Never use markdown, code, symbols, or headers. Never invent quotes. If you didn't catch what they said, ask them to repeat. "},
{"type": "input_audio", "input_audio": {"data": "<b64 wav>", "format": "wav"}}
]}
]
For transcription use: "Repeat the following text, without any explanation: " + audio.
A persona system prompt is strongly recommended for conversational use.
Training
Two phases from scratch on 2×H200:
- Phase 1B (~35K steps, eff. batch 16): English ASR corpora + real telephone conversations, with telephony augmentation (8kHz resample, AMR/μ-law compression, band-pass, pink noise, gain).
- Phase 2 (15K steps): 150K synthetic entity-dense utterances (phone numbers, names, spelled emails, addresses, dates/times across 31 TTS voices) + 40K spoken-dialogue response pairs + anti-forgetting garnish from phase-1 corpora.
Serving
Works with vLLM (--trust-remote-code). Note: Qwen 3.6's hybrid linear attention
currently loses some accuracy under vLLM relative to native transformers inference; for
maximum quality use HF transformers, or vLLM with --no-enable-chunked-prefill and
default (fp32) mamba cache dtypes.
Known limitations
- Short utterances degrade sharply. The benchmarks above are sentence-length inputs. On isolated sub-second turns — "Yes", "Hello", "Okay", single-word backchannels — the model frequently mistranscribes or hallucinates (observed in live deployment; the training mix is dominated by sentence-length corpora). If you use this in a live agent, buffer very short VAD segments with surrounding context, or route turns under ~1.5 s through a fallback. A v3 addressing this with short-turn training data is in progress.
- English only — degraded quality on other languages and heavily accented English.
- Response style is trained to the prompt shape shown above; deviating from it (especially for conversational use) reduces quality.
Apache 2.0. Built with the Ultravox training framework by Fixie.ai.