---
library_name: pytorch
datasets:
- bghira/minimax-music3-rvq-reverse-distillation
tags:
- audio
- music
- rvq
- reverse-distillation
- minimax-music-3
- mup
- autoregressive
---

# Open RVQ Encoder for MiniMax Music 3, 169M, v4

## Status

- Training complete: 17,660 optimizer steps.
- Recommended checkpoint: `final`.
- `checkpoint-17500` has the best free-running acoustic top-1 by 0.009 percentage point.
- Not an official MiniMax model.
- Not the original MiniMax Music 3 RVQ encoder.
- No original encoder weights or source code were used.
- V4 was initialized from scratch. No v1, v2, or v3 weights were loaded.
- Real-audio generalization is not established.
- A packaged `from_pretrained` loader is not present yet.

## Result

V4 replaces the seven independent acoustic readouts with a causal decoder across RVQ depth.

At step 17,500, relative to v3:

| Metric | v3 | v4 free-running | Change |
|---|---:|---:|---:|
| semantic top-1 | 43.03% | **43.16%** | +0.13 pp |
| semantic top-5 | 80.48% | **80.51%** | +0.03 pp |
| acoustic top-1 | **7.65%** | 7.30% | -0.35 pp |
| acoustic top-5 | **22.02%** | 19.88% | -2.14 pp |

V4 teacher-forced acoustic accuracy at the same step:

| Metric | Free-running | Teacher-forced |
|---|---:|---:|
| acoustic top-1 | 7.30% | 18.42% |
| acoustic top-5 | 19.88% | 44.54% |

The decoder learned strong conditional distributions. Greedy error propagation lowers exact agreement with the one sampled label sequence.

That exact-match result is not the primary outcome. The RVQ code space is redundant. Multiple code tuples can produce the same downstream conditioning.

Condition-embedding replay through the official LM and RVQ depth decoder:

| Model | Mean cosine | Change from v3 |
|---|---:|---:|
| v3 | 0.770259 | - |
| v4 | **0.874796** | **+0.104537** |

V4 is substantially more functionally correct by the downstream metric. Exact sampled-token accuracy is a weak proxy for that result.

V4 loss is conditional on true previous codebooks. It is not directly comparable to the independent-head v1-v3 loss.

## Objective

Approximate the missing audio-to-RVQ path used by MiniMax Music 3.

```text
44.1 kHz waveform
  -> frozen DAV / Flow-VAE encoder
  -> 128-channel DAV latents
  -> shared audio encoder
  -> semantic distribution
  -> causal RVQ-depth decoder
  -> 7 acoustic distributions
```

| Head | Role | Vocabulary |
|---:|---|---:|
| 0 | semantic | 16,384 |
| 1-7 | acoustic | 1,024 each |

The downstream path replays selected codes through the MiniMax Music 3 language model, official RVQ depth decoder, condition encoder, diffusion transformer, and DAV decoder.

## Architecture

Exported parameters: **169,008,576**.

| Component | Configuration | Parameters |
|---|---|---:|
| DAV latent input stem | Conv1d, 128 -> 1,088, kernel 7 | 975,936 |
| Local residual stack | 3 blocks, dilations 1/3/9 | 14,217,984 |
| Position embedding | learned, 128 x 1,088 | 139,264 |
| Shared Transformer | 8 pre-norm layers, width 1,088, 17 heads, FFN 4,352 | 113,752,576 |
| Final normalization | LayerNorm(1,088) | 2,176 |
| Semantic readout | `mup.MuReadout`, 1,088 -> 16,384 | 17,842,176 |
| Causal depth decoder | width 512, 8 heads, 2 layers, FFN 2,048 | 22,078,464 |

Shared audio processing:

1. Apply the convolutional stem and residual stack at DAV latent rate.
2. Average-pool exact DAV spans into 25 Hz frames.
3. Add learned frame positions.
4. Apply eight bidirectional Transformer layers.
5. Produce the semantic distribution.

Context: 128 frames, or 5.12 seconds. There is no cross-window state.

The per-sample pool matrix preserves stitched-chunk alignment. It is not a fixed-ratio resampler.

## Causal Depth Decoder

The depth decoder operates independently for each audio frame.

Configuration:

- width: 512;
- layers: 2;
- attention heads: 8;
- head dimension: 64;
- FFN multiplier: 4;
- dropout: 0.1;
- learned position for depth positions 0 through 7;
- separate prior-token embedding per codebook;
- separate 1,024-way output head per acoustic codebook.

Sequence for acoustic book `k`:

```text
[projected audio context, semantic code, acoustic code 1, ..., acoustic code k-1]
```

Training uses true earlier codes. Inference uses argmax predictions from earlier books.

The causal mask prevents access to later codebooks. The decoder is autoregressive across RVQ depth, not across time.

## Architecture Selection

| Version | Parameters | Change |
|---|---:|---|
| v1 | 40,978,944 | 512-wide baseline |
| v2 | 154,736,064 | shared width increased to 1,088 |
| v3 | 154,736,064 | v2 plus training-only MERT alignment |
| v4 | 169,008,576 | v2 shared encoder plus causal depth decoder |

V3 showed only a small gain from MERT alignment. V2 and v3 showed a monotonic top-1 decline across acoustic codebook depth. V4 tests whether conditioning book `k` on books `< k` removes that decline.

The teacher-forced result confirms the conditional-dependence diagnosis. The free-running result exposes a second problem: training and inference histories differ.

## Initialization and muTransfer

Package: [`microsoft/mup`](https://github.com/microsoft/mup).

Shared encoder shape family:

| Model | Width | Heads | Head dimension |
|---|---:|---:|---:|
| base | 128 | 2 | 64 |
| delta | 256 | 4 | 64 |
| target | 1,088 | 17 | 64 |

Depth topology is present in base, delta, and target shape models. Depth width remains fixed at 512.

Initialization order:

1. Construct target, base, and delta models with the depth decoder attached.
2. Call `mup.set_base_shapes` before optimizer construction.
3. Construct `mup.MuAdamW` after infshapes are attached.
4. Save depth-scoped base-shape metadata with each checkpoint.

V1-v3 encoder-only shape files are rejected for v4.

Semantic readout:

- `mup.MuReadout`;
- output multiplier 1.0;
- zero initialized;
- initial semantic distribution uniform.

The fixed-width depth decoder uses standard linear layers except for the 1,088 -> 512 context projection, which is a nonzero-initialized `mup.MuReadout`.

Shared attention score scale: `8 / head_dim`. At head dimension 64 this is `1/8`.

Seed: 42, device-specific under DDP.

## Data

Dataset: [`bghira/minimax-music3-rvq-reverse-distillation`](https://huggingface.co/datasets/bghira/minimax-music3-rvq-reverse-distillation).

Run snapshot:

- 2,972 one-track ZIP shards;
- 2,837 training records;
- 135 holdout records before exact-alignment filtering;
- approximately 178 GB;
- synthetic tracks generated by MiniMax Music 3;
- not MiniMax's original training set.

Trainer inputs:

- waveform audio;
- sampled RVQ codes;
- teacher top-50 token IDs and logits;
- exact chunk-stitching metadata.

Waveforms are re-encoded with [`SimpleTuner/MiniMax-Music-3-Encoder`](https://huggingface.co/SimpleTuner/MiniMax-Music-3-Encoder). DAV latents are cached once. Window reads use `safetensors.safe_open(...).get_slice(...)`.

Stored flow-VAE latents in the dataset are not consumed.

## Timeline Alignment

DAV hop: 512 samples at 44.1 kHz.

Frame center:

```text
((latent_start + latent_end) / 2) * 512 / 44100 seconds
```

Stitched rollout rules:

- rollout window: 200 semantic frames;
- rollout hop: 100 semantic frames;
- full stitched hop: 345 DAV latents;
- later chunks begin ownership 25 semantic frames after nominal start;
- code row 0 is warm-up;
- semantic frame `i` uses code row `i + 1`;
- the final partial chunk uses its own integer latent length;
- records without exact `chunk_stitching` metadata are excluded.

## Loss

```text
loss = mean(CE_head_0 ... CE_head_7)
     + 0.25 * mean(KL_head_0 ... KL_head_7)
```

For heads 1 through 7, logits are teacher-forced on true previous codebooks during training and loss evaluation.

Hard targets:

- cross-entropy against sampled RVQ codes;
- equal weight for all eight heads;
- padding target `-100`.

Soft targets:

- teacher top-k 50;
- temperature 1.0;
- Hinton `T^2` scaling;
- teacher renormalized over valid stored IDs;
- student full-vocabulary log-softmax gathered at teacher IDs;
- no student top-k renormalization;
- invalid, EOS, and out-of-vocabulary IDs excluded.

## Training

| Setting | Value |
|---|---:|
| Hardware | 4 x NVIDIA L40S |
| Distribution | PyTorch DDP through Accelerate |
| Precision | bfloat16 mixed precision |
| Epochs | 20 |
| Optimizer steps | 17,660 |
| Batch per rank | 16 |
| Global batch | 64 |
| Gradient accumulation | 1 |
| Optimizer | `mup.MuAdamW` |
| Learning rate | 3e-4 |
| Weight decay | 0.01 |
| LR schedule | polynomial, power 1.0 |
| Linear warmup | 500 steps |
| Final learning rate | 1e-7 |
| Gradient norm limit | 1.0 |
| Train crop | random 128-frame window |
| Validation crop | deterministic 128-frame windows |
| Validation interval | 500 steps |
| Checkpoint interval | 500 steps |

The learning-rate multiplier warms linearly for 500 steps, then decays linearly to the final learning rate. It does not restart or reheat.

Training metrics: [Weights & Biases](https://wandb.ai/bghira/simpletuner-rvq-encoder/runs/dxfjxfps).

Representative command:

```bash
accelerate launch --num_processes 4 scripts/train_minimax_music_rvq_encoder.py \
  --dataset_repo_id bghira/minimax-music3-rvq-reverse-distillation \
  --pretrained_vae_model_name_or_path SimpleTuner/MiniMax-Music-3-Encoder \
  --latent_cache_dir cache/vae/minimaxmusic-rvq-encoder \
  --output_dir output/minimaxmusic-rvq-encoder-169m-v4 \
  --d_model 1088 --layers 8 --heads 17 --ff_mult 4 \
  --depth_decoder --depth_decoder_dim 512 --depth_decoder_layers 2 \
  --depth_decoder_heads 8 --depth_decoder_ff_mult 4 \
  --mup --mup_readout_zero_init --optimizer torch-adamw \
  --learning_rate 3e-4 --lr_scheduler polynomial --lr_warmup_steps 500 \
  --lr_end 1e-7 --lr_power 1.0 --weight_decay 0.01 \
  --teacher_kl_weight 0.25 --teacher_kl_temperature 1.0 \
  --train_batch_size 16 --validation_batch_size 16 --num_train_epochs 20 \
  --mixed_precision bf16 --require_exact_alignment \
  --report_to wandb --tracker_project_name simpletuner-rvq-encoder \
  --push_to_hub SimpleTuner/open-rvq-encoder-minimax-music3-169m-v4
```

## Checkpoint Format

| File | Contents |
|---|---|
| `rvq_encoder.safetensors` | exported encoder and depth-decoder state dictionary |
| `rvq_encoder_config.json` | architecture and muP configuration |
| `mup_base_shapes.bsh` | exported depth-topology muP base shapes |
| `mup_base_shapes.bsh.meta.json` | shape scope and depth configuration |

Loading currently requires `RVQEncoderConfig` and `MiniMaxMusicRVQEncoder` from `scripts/train_minimax_music_rvq_encoder.py`.

## Evaluation

Protocol:

- exact-alignment holdout;
- 130 tracks;
- 2,768 deterministic windows;
- all 35 numbered checkpoints and `final`;
- four-rank evaluation;
- free-running argmax metrics;
- teacher-forced depth metrics reported separately.

Final checkpoint:

| Metric | Free-running | Teacher-forced |
|---|---:|---:|
| semantic top-1 | 43.17% | 43.17% |
| semantic top-5 | 80.51% | 80.51% |
| acoustic top-1 | 7.29% | 18.42% |
| acoustic top-5 | 19.88% | 44.55% |

Conditional final loss:

| Metric | Value |
|---|---:|
| total | 3.987436 |
| hard CE | 3.514642 |
| teacher KL before 0.25 weighting | 1.891180 |

Top-k accuracy measures exact token inclusion. It does not measure perceptual code equivalence.

### Matched Comparison at Step 17,500

![V1, v2, v3, and v4 comparison](evaluation/v4/v1-v2-v3-v4-comparison.png)

Machine-readable comparison: [`comparison-metrics.json`](evaluation/v4/comparison-metrics.json).

## Condition-Embedding Replay Comparison

Protocol:

- 130 exact-alignment holdout tracks;
- each final checkpoint predicts free-running argmax RVQ codes from cached DAV latents;
- predicted codes are teacher-forced through the official language model and official RVQ depth decoder;
- hidden states pass through the official condition encoder with recorded chunk stitching;
- reconstructed condition embeddings are compared with stored condition embeddings;
- metric: per-track mean cosine over stitched condition-latent frames;
- true sampled codes provide the replay control.

| Model | Parameters | Mean cosine | Standard deviation | 5th-95th percentile |
|---|---:|---:|---:|---:|
| Serveurperso v1 | 40,978,944 | 0.663329 | 0.022175 | 0.628052-0.696328 |
| SimpleTuner v1 | 40,978,944 | 0.762442 | 0.019550 | 0.734519-0.790450 |
| SimpleTuner v2 | 154,736,064 | 0.769841 | 0.019063 | 0.742991-0.798636 |
| SimpleTuner v3 | 154,736,064 | 0.770259 | 0.019274 | 0.741585-0.800492 |
| SimpleTuner v4 | 169,008,576 | **0.874796** | 0.015662 | 0.845350-0.893839 |
| True-code control | - | 0.999907 | 0.000150 | 0.999821-0.999950 |

![Condition-embedding replay comparison](evaluation/replay-comparison/condition-embedding-replay-comparison.png)

V4 adds 0.104537 mean cosine over v3. Its 5th-percentile track exceeds the best v3 track in this holdout. The depth decoder therefore improves code-tuple compatibility even though exact agreement with the sampled tokens declines slightly.

This test stops before diffusion and DAV decode. It is not an STFT, waveform, lyric-identity, or listening score.

Data: [`combined-aggregate.json`](evaluation/replay-comparison/combined-aggregate.json), [`provenance.json`](evaluation/replay-comparison/provenance.json), and [`raw per-record metrics`](evaluation/replay-comparison/raw-metrics-simpletuner-v4.json).

## Limitations

- 5.12-second temporal context.
- No cross-window state.
- Synthetic model-output training domain.
- Real audio remains out of distribution.
- Teacher-forced depth training has a large exact-token exposure gap under greedy decoding.
- Exact-token accuracy does not identify all functionally equivalent RVQ tuples.
- Conditional v4 loss is not comparable to v1-v3 loss.
- No beam search, scheduled sampling, or differentiable depth sampling was tested.
- Diffusion render and audio-domain evaluation remain pending.
- No stable packaged loading API.
- Use is subject to MiniMax Music 3 and dataset terms.

## Discussion and Attribution

Primary discussion: [MiniMaxAI/MiniMax-Music3 discussion #10](https://huggingface.co/MiniMaxAI/MiniMax-Music3/discussions/10).

Attribution covers public discussion, measurements, datasets, and independent experiments. It does not imply shared authorship.

- [`bghira`](https://huggingface.co/bghira): SimpleTuner experiments, trace extraction, teacher distributions, alignment records, corpus publication, and training runs.
- [`marduk191`](https://huggingface.co/marduk191): WAV/code samples, early mel encoder proof, corpus-scale observations, and encoder experiments.
- [`scragnog`](https://huggingface.co/scragnog): SimpleTuner calibration, relative-weight analysis, caption-cache and rollout-seam findings, and GGML interoperability tests.
- [`Serveurperso`](https://huggingface.co/Serveurperso): independent encoder, corpus generator, replay evaluation, and stitched-timeline findings.
- [`dernet`](https://huggingface.co/dernet): RVQ supervision, internal-alignment, and tokenizer reverse-engineering analysis.

Related artifacts:

- [`SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1`](https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1)
- [`SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2`](https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2)
- [`SimpleTuner/open-rvq-encoder-minimax-music3-155m-v3`](https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3-155m-v3)
- [`marduk191/Minmax_music3_experiments`](https://huggingface.co/marduk191/Minmax_music3_experiments/tree/main/corpus)
- [`ServeurpersoCom/minimaxmusic.cpp` proof commit](https://github.com/ServeurpersoCom/minimaxmusic.cpp/commit/d19efe9f94e41ac4c900aa30d56fe90c8dac7ef1)

<!-- simpletuner-rvq-evaluation-start -->
## Offline Checkpoint Evaluation

Exact-alignment holdout: 130 tracks, 2,768 windows.

| Selection | Checkpoint | Step | Loss | Semantic top-1 | Semantic top-5 | Acoustic top-1 | Acoustic top-5 |
|---|---|---:|---:|---:|---:|---:|---:|
| lowest loss; best semantic top-1; final | `final` | 17,660 | 3.987436 | 0.4317 | 0.8051 | 0.0729 | 0.1988 |
| best acoustic top-1 | `checkpoint-17500` | 17,500 | 3.988227 | 0.4316 | 0.8051 | 0.0730 | 0.1988 |

Top-k accuracy measures exact token inclusion. It does not measure perceptual code equivalence.

### Checkpoint Loss

![Checkpoint Loss](evaluation/v4/checkpoint-loss.png)

### Checkpoint Accuracy

![Checkpoint Accuracy](evaluation/v4/checkpoint-accuracy.png)

### Codebook Top-1

![Codebook Top-1](evaluation/v4/codebook-top1.png)

### Depth Teacher-Forcing Gap

![Depth Teacher-Forcing Gap](evaluation/v4/depth-teacher-forcing-gap.png)

### Training History

![Training History](evaluation/v4/training-history.png)

Full data: [`checkpoint-metrics.csv`](evaluation/v4/checkpoint-metrics.csv), [`evaluation-metrics.json`](evaluation/v4/evaluation-metrics.json), and [`comparison-metrics.json`](evaluation/v4/comparison-metrics.json).
<!-- simpletuner-rvq-evaluation-end -->