---
license: other
library_name: pytorch
datasets:
- bghira/minimax-music3-rvq-reverse-distillation
tags:
- audio
- music
- minimax-music-3
- rvq
- reference-audio
- comfyui
- diffusers
---

# Open RVQ Encoders for MiniMax Music 3

## What this release does

MiniMax Music 3 normally creates its internal music codes from text and lyrics. The original audio-to-code encoder was not released. These models add an approximate path from a reference recording back to those codes.

The useful result is **0.8748 mean condition-replay cosine** on 130 held-out generated tracks. In plain English: after a reference track is reduced to eight integers per 25 Hz frame and replayed through the official language-model path, v4 recovers most of the conditioning signal that the diffusion model would have received from the original codes.

This is not 87% exact-token accuracy. It is not an audio-quality score. The RVQ code space is redundant: different code tuples can lead to nearly the same downstream condition. Exact token labels therefore understate functional accuracy.

The project started with a 41M-parameter, single-GPU community proof at **0.6633** replay cosine. The SimpleTuner experiments moved through width and music-feature alignment, then reached **0.8748** by making the acoustic codebooks autoregressive across RVQ depth.

Use **v4** unless reproducing an experiment.

## Files

| File | Parameters | Experiment | Replay cosine |
|---|---:|---|---:|
| `minimax_music3_rvq_encoder_v1_41m_independent_heads.safetensors` | 40,978,944 | Baseline; eight independent heads | 0.7624* |
| `minimax_music3_rvq_encoder_v2_155m_wide_independent_heads.safetensors` | 154,736,064 | Wider shared encoder | 0.7698 |
| `minimax_music3_rvq_encoder_v3_155m_mert_aligned_independent_heads.safetensors` | 154,736,064 | v2 plus training-only MERT alignment | 0.7703 |
| `minimax_music3_rvq_encoder_v4_169m_autoregressive_depth_recommended.safetensors` | 169,008,576 | Causal acoustic decoder across codebook depth | **0.8748** |

Each weight file has a same-named `.json` configuration file in [`encoders/`](encoders/).

\* The v1 replay evaluation used the final checkpoint. This collection ships the v1 card-recommended step-17,500 checkpoint. V2-v4 replay scores and shipped checkpoints are final checkpoints.

![Condition replay comparison](assets/condition-embedding-replay-comparison.png)

Metric: per-track mean cosine between replayed and stored condition embeddings. Evaluation set: 130 exact-alignment records from the held-out split. True sampled codes score 0.9999 as the pipeline control. Release provenance is in [`experiment-summary.json`](experiment-summary.json). Full summary statistics and quantiles are in [`evaluation/condition-replay-aggregate.json`](evaluation/condition-replay-aggregate.json).

## Experiment arc

### Community proof

`Serveurperso` trained a 41M encoder on one GPU and demonstrated that exact token agreement was not required. Their checkpoint reached 0.6633 condition-replay cosine. Their implementation independently identified the 345-latent stitched hop, the extra warm-up code row, and the final partial-chunk interpolation rule. The SimpleTuner dataset and trainer work independently derived the same alignment contract.

That result changed the question from "can this work?" to "which errors matter downstream?"

### v1: working baseline

- 41M parameters.
- DAV latents as input.
- One semantic head and seven independent acoustic heads.
- Teacher top-50 KL plus hard cross-entropy.
- μP initialization and μTransfer shape metadata.
- DDP training on the reverse-distillation corpus.
- 0.7624 replay cosine at the evaluated final checkpoint.

Detailed card: [`SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1`](https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1).

### v2: width

- Width increased from 512 to 1,088.
- Parameters increased from 41M to 155M.
- Polynomial decay with warm-up.
- Replay improved from 0.7624 to 0.7698.

Width helped token metrics. It barely moved the downstream replay metric. Capacity was not the main limit.

Detailed card: [`SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2`](https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2).

### v3: MERT alignment

- Same exported architecture as v2.
- Frozen MERT features used only as a training target.
- Mid-layer cosine alignment loss annealed to zero.
- No MERT weights or projection are present in the exported encoder.
- Replay improved from 0.7698 to 0.7703.

The measured gain was too small to justify an inference dependency or a larger MERT experiment on this corpus.

Detailed card: [`SimpleTuner/open-rvq-encoder-minimax-music3-155m-v3`](https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3-155m-v3).

### v4: autoregressive depth

- Shared encoder kept at v2 width.
- The seven acoustic heads were replaced by a causal decoder.
- Codebook `k` is conditioned on the semantic code and acoustic codebooks `< k`.
- Parameters increased to 169M.
- Replay improved from 0.7703 to 0.8748.

Free-running exact acoustic top-1 decreased slightly. Teacher-forced acoustic top-1 increased to 18.42%. Replay improved by 0.1045. This is the central result: code-tuple compatibility matters more than matching one sampled token tuple.

Detailed card: [`SimpleTuner/open-rvq-encoder-minimax-music3-169m-v4`](https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3-169m-v4).

## ComfyUI

Requirements:

- current ComfyUI with native MiniMax Music 3 support;
- official ComfyUI MiniMax diffusion model, text encoder, and DAV decoder;
- original `dav.pth`, because the Comfy DAV file is decoder-only;
- this repository cloned with Git LFS.

```bash
git lfs install
git clone https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3

cd ComfyUI
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt huggingface_hub

.venv/bin/hf download Comfy-Org/MiniMax-Music-3 \
  diffusion_models/minimax_music3_dit_fp16.safetensors \
  text_encoders/minimax_music3_text_encoder_pruned_int8_convrot.safetensors \
  vae/minimax_music3_dav.safetensors \
  --local-dir models

.venv/bin/hf download MiniMaxAI/MiniMax-Music3 dav.pth \
  --local-dir models/vae

ln -s /path/to/open-rvq-encoder-minimax-music3/comfyui_open_rvq \
  custom_nodes/comfyui_open_rvq
```

Restart ComfyUI. Load [`comfyui_workflow_example.json`](comfyui_workflow_example.json). Upload a reference audio file. Select v4 in **MiniMax Music3 RVQ Reference Encoder Loader**.

The node package reads the encoder files directly from this clone. They can instead be placed in:

```text
ComfyUI/models/minimax_music3_rvq_encoders/
```

The tested graph used ComfyUI 0.33.0, one NVIDIA L40S, the pruned int8 text encoder, the fp16 diffusion model, a 32.28-second held-out reference, five Euler steps, and the v4 encoder. It completed and produced a full-length non-silent stereo FLAC. Use 30 steps for normal output.

## Diffusers

The MiniMax Music 3 Diffusers integration is currently installed from its integration commit:

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install \
  git+https://github.com/huggingface/diffusers@dafe3733fcfdbf3c48915fe77be3aef65b5d6a2d \
  transformers accelerate safetensors huggingface_hub soundfile torchaudio
```

Download [`minimax_music3_reference_adapter.py`](minimax_music3_reference_adapter.py) into the project, then:

```python
import soundfile as sf
import torch
from diffusers import ModularPipeline

from minimax_music3_reference_adapter import (
    MiniMaxMusic3ReferenceAdapter,
    install_diffusers_reference_adapter,
)

install_diffusers_reference_adapter()

pipe = ModularPipeline.from_pretrained(
    "MiniMaxAI/MiniMax-Music3",
    torch_dtype=torch.bfloat16,
).to("cuda")
adapter = MiniMaxMusic3ReferenceAdapter.from_pretrained()

audio, sample_rate = sf.read("reference.flac", always_2d=True)
waveform = torch.from_numpy(audio.T).float()
generator = torch.Generator(device="cpu").manual_seed(42)

frame_hiddens, predicted_codes = adapter.encode_reference(
    pipe,
    waveform,
    sample_rate,
    prompt="rock",
    lyrics="[instrumental]",
    generator=generator,
    device="cuda",
)

result = pipe(
    frame_hiddens=frame_hiddens,
    generator=torch.Generator(device="cpu").manual_seed(42),
    num_inference_steps=30,
    output_type="pt",
)
```

The patch only adds a precomputed-`frame_hiddens` bypass to the modular pipeline. It does not replace MiniMax model code.

## Limits

- These are approximate encoders, not the original MiniMax encoder.
- No original encoder weights or source were used.
- Training data is synthetic MiniMax Music 3 output, not MiniMax's training set.
- Real-audio generalization is not established.
- Context is 128 frames, or 5.12 seconds. There is no cross-window encoder state.
- Reference replay still needs the official MiniMax language model and RVQ depth decoder.
- v4 uses greedy code selection. Other decoding strategies remain untested.

## Credits

- [`bghira`](https://huggingface.co/bghira): reverse-distillation dataset, independent alignment derivation, SimpleTuner trainer, compute, v1-v4 runs, and replay experiments.
- [`marduk191`](https://huggingface.co/marduk191): WAV/code samples, early mel-encoder proof, corpus-scale observations, and encoder experiments.
- [`scragnog`](https://huggingface.co/scragnog): SimpleTuner calibration, relative-weight analysis, caption-cache and rollout-seam findings, and GGML interoperability tests.
- [`Serveurperso`](https://huggingface.co/Serveurperso): independent 41M encoder, single-GPU proof, replay evaluation, and independent stitched-timeline findings.
- [`dernet`](https://huggingface.co/dernet): RVQ supervision, internal-alignment, and tokenizer reverse-engineering analysis.

## License

`minimax_music3_reference_adapter.py` and `comfyui_open_rvq/` are Apache-2.0 licensed. See [`LICENSE-CODE`](LICENSE-CODE).

The encoder weights were trained on outputs from MiniMax Music 3. Use of the weights, official model components, and generated output remains subject to the applicable MiniMax Music 3 terms. The MERT license also applies to the v3 training provenance; no MERT weights are distributed here.