laion/voice-acting-adaln-speaker-r8-step850

AdaLN-Zero Speaker Conditioning — Run 8 Step 850

Best-performing speaker conditioning checkpoint for the voice-acting audio DiT model.

Architecture

• Base model: the voice-acting audio-only DiT (3.29B params, frozen)
• LoRA merge: Run 7 LoRA rank-64 adapter (288 layer pairs) merged into base DiT
• AdaLN-Zero: SpeakerAdaLNZero network (~440M params, fp32)
  • speaker_dim=2176, audio_dim=2048, num_blocks=48, bottleneck_dim=512
  • 9 AdaLN parameters per block (6 base MSA+FFN + 3 cross-attn QK)

Training

• Approach: Freeze LoRA-merged DiT, train only AdaLN-Zero
• LR: 7e-5 with cosine decay (min_lr_ratio=0.1)
• Epochs: 6 (1,020 steps, effective batch=128)
• Data: 21,734 samples (voice-acting best_of_25 + podcast + emolia)
• Step 850 selected: Best UTMOS across all evaluation variants

Evaluation (step850 vs Run 5 AdaLN e5 baseline)

Variant	ΔMOS	ΔUTMOS	ΔSpkSim
Raw	−0.054	+0.064	−0.003
Sidon	+0.031	+0.129	+0.001
VC→Sidon	+0.013	+0.079	+0.007

Files

• speaker_adaln_step850.pt — AdaLN-Zero weights (fp32, 1.7 GB)
• merged_dit.safetensors — LoRA-merged DiT checkpoint (bf16, 6.2 GB)
• training_args.json — Full training configuration
• metrics.jsonl — Per-step training metrics (1,020 steps)

Usage

from speaker_adaln import SpeakerAdaLNZero, install_speaker_adaln_hooks

# Load AdaLN-Zero
adaln = SpeakerAdaLNZero(
    speaker_dim=2176, audio_dim=2048,
    num_blocks=48, num_adaln_params=9, bottleneck_dim=512
)
adaln.load_state_dict(torch.load("speaker_adaln_step850.pt"))

# Load merged DiT
model = build_audio_only_model("merged_dit.safetensors", device, dtype)

# Install hooks for inference
handles = install_speaker_adaln_hooks(model, adaln, speaker_embedding)

License

CC BY 4.0.

---

This checkpoint conditions a voice-acting AI model.