---
license: cc0-1.0
base_model: unsloth/orpheus-3b-0.1-ft
tags:
- text-to-speech
- tts
- orpheus
- swahili
- kiswahili
- snac
- peft
- lora
- unsloth
language:
- sw
- en
datasets:
- rlabz/swa_lug_tts
pipeline_tag: text-to-speech
---

# quantum_tts

`quantum_tts` is a LoRA fine-tune of [Orpheus-TTS](https://github.com/canopyai/Orpheus-TTS) (`unsloth/orpheus-3b-0.1-ft`), adapted to a single Kiswahili voice using SNAC-tokenized speech data. At inference time the LoRA adapter is combined with the base model via PEFT, which preserves the base model's original capabilities alongside the fine-tuned voice — including the ability to perform **voice characteristic transfer to English text**, despite no English training data being used for this speaker.

## Model Details

- **Base model:** [`unsloth/orpheus-3b-0.1-ft`](https://huggingface.co/unsloth/orpheus-3b-0.1-ft) (Llama-3B backbone)
- **Fine-tuning method:** LoRA adapter via [Unsloth](https://github.com/unslothai/unsloth) + [PEFT](https://github.com/huggingface/peft)
- **Audio codec:** [SNAC](https://github.com/hubertsiuzdak/snac) (`hubertsiuzdak/snac_24khz`)
- **Language(s):** Kiswahili (training), with emergent English voice-transfer capability at inference
- **License:** CC0 1.0 (inherited from training data)
- **Training data:** [`rlabz/swa_lug_tts`](https://huggingface.co/datasets/rlabz/swa_lug_tts), tokenized into Orpheus's interleaved text/SNAC-code format

## Intended Use

`quantum_tts` is intended for single-voice Kiswahili text-to-speech synthesis, and experimentally for cross-lingual voice transfer to English text using the same speaker's voice characteristics. Because the adapter is combined with the base model at inference (rather than replacing it), the model retains the general TTS capabilities of the Orpheus base — this is what enables it to speak English text in the fine-tuned voice even though English audio was never part of training.

This model is not intended as a general-purpose multi-speaker or multi-language TTS system.

## How to Get Started

### Load the model

```python
from unsloth import FastLanguageModel
from peft import PeftModel

# Base model
base_model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/orpheus-3b-0.1-ft",
    max_seq_length=4096,   # choose any for long context
    dtype=None,            # None = auto detection
    # load_in_4bit=True,   # set True to reduce memory usage
)

# Attach the quantum_tts LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    "rlabz/quantum_tts",  # remove this step to run the base model only
)
```

### Run inference

```python
import gc
import torch
from IPython.display import display, Audio
from snac import SNAC

snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").to("cuda")

gc.collect()
torch.cuda.empty_cache()

FastLanguageModel.for_inference(model)
snac_model.to("cpu")  # free GPU for LLM generation

prompts_ = [(f"{chosen_voice}: " + p) if chosen_voice else p for p in prompts]

# Tokenize and wrap with Orpheus's special tokens
all_input_ids = [tokenizer(p, return_tensors="pt").input_ids for p in prompts_]

start_token = torch.tensor([[128259]], dtype=torch.int64)          # start of human
end_tokens  = torch.tensor([[128009, 128260]], dtype=torch.int64)  # EOT, end of human

all_modified_input_ids = [
    torch.cat([start_token, ids, end_tokens], dim=1) for ids in all_input_ids
]

# Pad batch
max_length = max(t.shape[1] for t in all_modified_input_ids)
all_padded_tensors, all_attention_masks = [], []
for modified in all_modified_input_ids:
    padding = max_length - modified.shape[1]
    padded = torch.cat([torch.full((1, padding), 128263, dtype=torch.int64), modified], dim=1)
    mask = torch.cat(
        [torch.zeros((1, padding), dtype=torch.int64),
         torch.ones((1, modified.shape[1]), dtype=torch.int64)], dim=1
    )
    all_padded_tensors.append(padded)
    all_attention_masks.append(mask)

input_ids = torch.cat(all_padded_tensors, dim=0).to("cuda")
attention_mask = torch.cat(all_attention_masks, dim=0).to("cuda")

# Generate
with torch.inference_mode():
    generated_ids = model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        max_new_tokens=1200,
        do_sample=True,
        temperature=0.6,
        top_p=0.95,
        repetition_penalty=1.1,
        num_return_sequences=1,
        eos_token_id=128258,
        use_cache=True,
    )

# Extract SNAC audio tokens, then decode with SNAC to get waveforms
# (see full decoding + redistribute_codes logic in the repo's inference script)
```

> The snippet above is abbreviated for readability — the full inference script (token extraction, `redistribute_codes`, and SNAC decoding into playable audio) is included in this project [`notebook`](https://colab.research.google.com/drive/1aoYk1uJPekUJZHRaulSw1RxY0Sgk5Y12?usp=sharing).

## Training Details

### Training Data

Fine-tuned on the SNAC-tokenized version of [`rlabz/swa_lug_tts`](https://huggingface.co/datasets/rlabz/swa_lug_tts), filtered to a single Kiswahili speaker and formatted into Orpheus's interleaved `input_ids` / `labels` / `attention_mask` layout (text tokens + SNAC audio codes, per the [Orpheus fine-tuning data format](https://github.com/canopyai/Orpheus-TTS)).

### Training Procedure

Fine-tuned with Hugging Face `Trainer` on top of an [Unsloth](https://github.com/unslothai/unsloth)-loaded Orpheus base model, using LoRA adapters (via PEFT) rather than full fine-tuning.

**Training hyperparameters:**

| Hyperparameter | Value |
|---|---|
| Per-device train batch size | 1 |
| Gradient accumulation steps | 8 |
| Effective batch size | 8 |
| Learning rate | 3.4e-5 |
| LR scheduler | cosine |
| Warmup steps | 50 |
| Max steps | 600 |
| Optimizer | adamw_8bit |
| Weight decay | 0.001 |
| Eval strategy | steps (every 100) |
| Save strategy | steps (every 100) |
| Load best model at end | Yes |
| Metric direction | greater_is_better=False (lower eval loss is better) |
| Seed | 3407 |

```python
from transformers import TrainingArguments, Trainer, DataCollatorForSeq2Seq

trainer = Trainer(
    model=model,
    data_collator=data_collator,
    train_dataset=dataset,
    eval_dataset=val_dataset,
    args=TrainingArguments(
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=3.5e-6,
        warmup_steps=50,
        max_steps=600,
        logging_steps=100,
        eval_strategy="steps",
        eval_steps=100,
        save_steps=100,
        load_best_model_at_end=True,
        greater_is_better=False,
        optim="adamw_8bit",
        weight_decay=0.001,
        lr_scheduler_type="cosine",
        seed=3407,
        output_dir="outputs",
        report_to="none",
    ),
)
```

### Compute Infrastructure

Trained via Unsloth-accelerated fine-tuning on the Orpheus-3B backbone.

## Notable Capability: Cross-Lingual Voice Transfer

Because inference combines the LoRA adapter with the unmodified base model (rather than serving a fully-merged fine-tuned checkpoint), `quantum_tts` retains the base Orpheus model's broader multilingual TTS ability. In practice this means the fine-tuned voice can be prompted with **English text** and will speak it in the same voice characteristics learned from the Kiswahili fine-tuning data — even though the training set contained no English audio for this speaker. This is an emergent property of the PEFT-based inference setup rather than something explicitly trained for, so quality and consistency on English text should be evaluated case by case.

## Acknowledgements

- [Orpheus-TTS](https://github.com/canopyai/Orpheus-TTS) by Canopy Labs
- [SNAC](https://github.com/hubertsiuzdak/snac) neural audio codec
- [Unsloth](https://github.com/unslothai/unsloth) for efficient fine-tuning
- Training data derived from [`rlabz/swa_lug_tts`](https://huggingface.co/datasets/rlabz/swa_lug_tts)
---