---
license: cc-by-nc-4.0
language:
  - en
library_name: ctrlspeech
pipeline_tag: text-to-speech
tags:
  - text-to-speech
  - speech-synthesis
  - prosody-control
  - controllable-tts
  - flow-matching
---

# CtrlSpeech

Checkpoints for **CtrlSpeech** — expressive speech synthesis with coarse-to-fine
latent control. Code, CLI and interactive demo:
**https://github.com/zszheng147/CtrlSpeech**

CtrlSpeech is a zero-shot TTS model you can steer after generating. It exposes
the pitch contour, loudness contour and phoneme boundaries of any utterance,
lets you edit one of them, and resynthesises following the edit while leaving
the others alone. Prosody is conditioned **per phoneme token**, so a single word
can be stretched while the rest of the sentence keeps its timing.

## Contents

| Path | Params | Prosody control | Notes |
|---|---|---|---|
| `control-600m/` | 692 M | pitch · loudness · duration | Default |
| `control-150m/` | 150 M | pitch · loudness · duration | Smaller |
| `base-600m/` | 689 M | — | Ablation baseline, no control |
| `base-150m/` | 148 M | — | Ablation baseline, no control |
| `svae/` | — | — | SVAE (DAC + BigVGAN) 16 kHz codec, 40 Hz latents |
| `shared/campplus.onnx` | — | — | CAM++ speaker embedding |
| `shared/vocab.json` | — | — | Phoneme vocabulary (ARPAbet) |

Each model folder holds `model.safetensors`, a training-free `config.yaml`, and
the `qwen_config.json` describing the AR backbone architecture. The fine-tuned
backbone weights are inside the checkpoint; no Qwen weights are redistributed
and none are downloaded at load time.

## Usage

```bash
pip install git+https://github.com/zszheng147/CtrlSpeech
```

```python
from ctrlspeech import CtrlSpeech, shift_pitch_semitones

tts = CtrlSpeech.from_pretrained("control-600m")     # downloads on first use
baseline = tts.from_audio("clip.wav", "the transcript of that clip")
tts.regenerate(baseline, pitch=shift_pitch_semitones(baseline.gen_f0, 5)) \
   .save("higher.wav")
```

Duration editing and adopting your own recording require
[Montreal Forced Aligner](https://montreal-forced-aligner.readthedocs.io) with
the `english_us_arpa` acoustic model and dictionary.

## Control representation

Analysis runs at 100 frames/s (16 kHz, hop 160).

- **Pitch** — 128 mel-spaced bins; bin 0 is unvoiced.
- **Loudness** — 64 A-weighted dB bins, about 1.05 bins per dB.
- **Duration** — per-phoneme frame counts, at most 191 frames (1.91 s) per
  phoneme; an edited timeline is capped at 2001 frames (20 s).