File size: 16,181 Bytes

---
library_name: pytorch
datasets:
- bghira/minimax-music3-rvq-reverse-distillation
tags:
- audio
- music
- rvq
- reverse-distillation
- minimax-music-3
- mup
- mert
---

# Open RVQ Encoder for MiniMax Music 3, 155M, v3

## Status

- Training complete: 17,660 optimizer steps.
- Recommended checkpoint: `final`.
- Not an official MiniMax model.
- Not the original MiniMax Music 3 RVQ encoder.
- No original encoder weights or source code were used.
- V3 was initialized from scratch. No v1 or v2 weights were loaded.
- Real-audio generalization is not established.
- MERT was used only as a frozen training teacher. MERT weights are not included.
- A packaged `from_pretrained` loader is not present yet.

## Result

V3 adds MERT representation alignment to the v2 architecture.

At the matched step 17,500, relative to v2:

- loss: 5.264569 -> 5.260236;
- semantic top-1: 42.86% -> 43.03%;
- semantic top-5: 80.17% -> 80.48%;
- acoustic top-1: 7.62% -> 7.65%;
- acoustic top-5: 21.98% -> 22.02%.

The MERT objective converged. Holdout MERT cosine similarity reached 0.762. RVQ accuracy changed only slightly. MERT alignment is not the main missing component for this architecture and corpus.

## Objective

Approximate the missing audio-to-RVQ path used by MiniMax Music 3.

```text
44.1 kHz waveform
  -> frozen DAV / Flow-VAE encoder
  -> 128-channel DAV latents
  -> this encoder
  -> 8 RVQ distributions per 25 Hz frame
  -> 1 semantic code + 7 acoustic codes
```

| Head | Role | Vocabulary |
|---:|---|---:|
| 0 | semantic | 16,384 |
| 1-7 | acoustic | 1,024 each |

Argmax gives a discrete code stream. The downstream path replays those codes through the MiniMax Music 3 LM, condition encoder, diffusion transformer, and DAV decoder.

## Architecture

Exported encoder parameters: **154,736,064**.

Training-only MERT projection parameters: **835,584**.

Total trainable parameters during v3 training: **155,571,648**.

| Component | Configuration | Parameters |
|---|---|---:|
| DAV latent input stem | Conv1d, 128 -> 1,088, kernel 7 | 975,936 |
| Local residual stack | 3 blocks, dilations 1/3/9, GroupNorm, kernel-3 and kernel-1 convolutions | 14,217,984 |
| Position embedding | learned, 128 x 1,088 | 139,264 |
| Transformer | 8 pre-norm layers, width 1,088, 17 heads, FFN 4,352, GELU, dropout 0.1 | 113,752,576 |
| Final normalization | LayerNorm(1,088) | 2,176 |
| RVQ readouts | 8 independent `mup.MuReadout` heads | 25,648,128 |
| MERT projection | training-only `mup.MuReadout`, 1,088 -> 768, no bias | 835,584 |

Processing:

1. Apply the convolutional stem and residual stack at DAV latent rate.
2. Average-pool exact DAV spans into 25 Hz frames.
3. Add learned positions.
4. Apply eight bidirectional Transformer encoder layers.
5. Apply final LayerNorm.
6. Produce eight independent RVQ distributions.

Context: 128 frames, or 5.12 seconds. There is no cross-window state.

The per-sample pool matrix preserves stitched-chunk alignment. It is not a fixed-ratio resampler.

## Architecture Selection

| Version | Exported parameters | Change |
|---|---:|---|
| v1 | 40,978,944 | 512-wide baseline |
| v2 | 154,736,064 | width increased to 1,088 |
| v3 | 154,736,064 | v2 encoder plus training-only MERT alignment |

V3 keeps the v2 encoder unchanged. This isolates the MERT auxiliary objective.

The encoder still predicts the seven acoustic books independently. Head `k` does not receive selected codes from heads `< k`. The per-head results show a strong accuracy decline with codebook depth. V4 addresses that separately with a causal depth decoder.

## Initialization and muTransfer

Package: [`microsoft/mup`](https://github.com/microsoft/mup).

Encoder shape family:

| Model | Width | Heads | Head dimension |
|---|---:|---:|---:|
| base | 128 | 2 | 64 |
| delta | 256 | 4 | 64 |
| target | 1,088 | 17 | 64 |

Initialization order:

1. Construct target, base, and delta training wrappers.
2. Attach wrapper-level base shapes with `mup.set_base_shapes`.
3. Construct `mup.MuAdamW` after infshapes are attached.
4. Save wrapper and exported-encoder base-shape files.

The wrapper-level shape family includes the MERT projection. Encoder-only v1/v2 base-shape files are incompatible with the v3 training wrapper.

RVQ readouts:

- `mup.MuReadout`;
- output multiplier 1.0;
- zero initialized;
- initial output distributions uniform within each vocabulary.

MERT projection:

- `mup.MuReadout`, 1,088 -> 768;
- no bias;
- nonzero initialization required for cosine loss;
- removed from exported encoder checkpoints.

Attention score scale is `8 / head_dim`. At head dimension 64 this equals standard `1/sqrt(64)` scaling.

Seed: 42, device-specific under DDP.

## Data

Dataset: [`bghira/minimax-music3-rvq-reverse-distillation`](https://huggingface.co/datasets/bghira/minimax-music3-rvq-reverse-distillation).

Run snapshot:

- 2,972 one-track ZIP shards;
- 2,837 training records;
- 135 holdout records before exact-alignment filtering;
- approximately 178 GB;
- synthetic tracks generated by MiniMax Music 3;
- not MiniMax's original training set.

Trainer inputs:

- waveform audio;
- sampled RVQ codes;
- teacher top-50 token IDs and logits;
- exact chunk-stitching metadata.

Waveforms are re-encoded with [`SimpleTuner/MiniMax-Music-3-Encoder`](https://huggingface.co/SimpleTuner/MiniMax-Music-3-Encoder). DAV latents are cached once. Window reads use `safetensors.safe_open(...).get_slice(...)`.

Stored flow-VAE latents in the dataset are not consumed.

## Timeline Alignment

DAV hop: 512 samples at 44.1 kHz.

Frame center:

```text
((latent_start + latent_end) / 2) * 512 / 44100 seconds
```

Stitched rollout rules:

- rollout window: 200 semantic frames;
- rollout hop: 100 semantic frames;
- full stitched hop: 345 DAV latents;
- later chunks begin ownership 25 semantic frames after nominal start;
- code row 0 is warm-up;
- semantic frame `i` uses code row `i + 1`;
- the final partial chunk uses its own integer latent length;
- records without exact `chunk_stitching` metadata are excluded.

MERT features are linearly interpolated onto these exact DAV frame centers. A naive 3:1 reshape is not used.

## MERT Alignment

Teacher: [`m-a-p/MERT-v1-95M`](https://huggingface.co/m-a-p/MERT-v1-95M).

Pinned revision: `12af15fef9d0ac838c3f475bfbbf26d2060dd4f5`.

| Setting | Value |
|---|---:|
| Teacher layer | 9 |
| Student capture layer | 4, zero-based |
| Teacher hidden size | 768 |
| Teacher sample rate | 24 kHz |
| Teacher feature rate | 75 Hz |
| Chunk length | 5 seconds |
| Chunk overlap | 1 second |
| Cache dtype | bfloat16 |
| Initial alignment weight | 0.5 |
| Constant phase | 0% through 70% of training |
| Linear decay | 70% through 90% |
| Disabled weight | final 10% |

MERT sidecars are generated before training. Cache metadata records the model, revision, hidden layers, chunk geometry, dtype, emitted frame count, and alignment version `dav512-mert75-center-v1`.

The projection forward remains active after its scheduled weight reaches zero. This preserves DDP parameter participation. The projection is not exported.

MERT-v1-95M is published under CC-BY-NC-4.0. This repository does not redistribute MERT weights. Users remain responsible for applicable model, dataset, and teacher terms.

## Loss

```text
reported_loss = mean(CE_head_0 ... CE_head_7)
              + 0.25 * mean(KL_head_0 ... KL_head_7)

optimization_loss = reported_loss
                  + scheduled_MERT_weight * cosine_alignment_loss
```

Hard targets:

- cross-entropy against sampled RVQ codes;
- equal weight for all eight heads;
- padding target `-100`.

Soft targets:

- teacher top-k 50;
- temperature 1.0;
- Hinton `T^2` scaling;
- teacher renormalized over valid stored IDs;
- student full-vocabulary log-softmax gathered at teacher IDs;
- no student top-k renormalization;
- invalid, EOS, and out-of-vocabulary IDs excluded;
- frames with no valid teacher IDs skipped for KL.

MERT target:

- cosine distance between projected student layer 4 and frozen MERT layer 9;
- mean over batch and frames.

Reported loss excludes MERT. V1, v2, and v3 loss curves therefore remain directly comparable.

## Training

| Setting | Value |
|---|---:|
| Hardware | 4 x NVIDIA L40S |
| Distribution | PyTorch DDP through Accelerate |
| Precision | bfloat16 mixed precision |
| Epochs | 20 |
| Optimizer steps | 17,660 |
| Batch per rank | 16 |
| Global batch | 64 |
| Gradient accumulation | 1 |
| Optimizer | `mup.MuAdamW` |
| Learning rate | 3e-4 |
| Weight decay | 0.01 |
| LR schedule | polynomial, power 1.0 |
| Linear warmup | 500 steps |
| Final learning rate | 1e-7 |
| Gradient norm limit | 1.0 |
| Train crop | random 128-frame window |
| Validation crop | deterministic 128-frame windows |
| Validation interval | 500 steps |
| Checkpoint interval | 500 steps |

The learning-rate multiplier warms linearly for 500 steps, then decays linearly to the final learning rate. It does not restart or reheat.

Training metrics: [Weights & Biases](https://wandb.ai/bghira/simpletuner-rvq-encoder/runs/tap14a1y).

## Checkpoint Format

Each exported checkpoint contains:

| File | Contents |
|---|---|
| `rvq_encoder.safetensors` | exported encoder state dictionary |
| `rvq_encoder_config.json` | architecture and muP configuration |
| `mup_base_shapes.bsh` | exported-encoder muP base shapes |

The training-only MERT projection and MERT teacher are not included.

Loading currently requires `RVQEncoderConfig` and `MiniMaxMusicRVQEncoder` from `scripts/train_minimax_music_rvq_encoder.py`.

## Evaluation

Protocol:

- exact-alignment holdout;
- 130 tracks;
- 2,768 deterministic windows;
- all 35 numbered checkpoints and `final`;
- four-rank evaluation; no distributed-sampler padding;
- exact-token top-1 and top-5.

Recommended `final` result:

| Metric | Value |
|---|---:|
| loss | 5.259917 |
| hard CE | 4.629320 |
| teacher KL before 0.25 weighting | 2.522392 |
| semantic top-1 | 43.03% |
| semantic top-5 | 80.49% |
| acoustic top-1 | 7.66% |
| acoustic top-5 | 22.03% |

Top-k accuracy measures exact token inclusion. It does not measure perceptual code equivalence.

### Matched Comparison at Step 17,500

| Metric | v1, 41M | v2, 155M | v3, 155M + MERT | v3 vs v2 |
|---|---:|---:|---:|---:|
| loss | 5.337856 | 5.264569 | **5.260236** | -0.004334 |
| semantic top-1 | 41.03% | 42.86% | **43.03%** | +0.17 pp |
| semantic top-5 | 78.38% | 80.17% | **80.48%** | +0.31 pp |
| acoustic top-1 | 7.17% | 7.62% | **7.65%** | +0.03 pp |
| acoustic top-5 | 20.94% | 21.98% | **22.02%** | +0.04 pp |

![V1, v2, and v3 comparison](evaluation/v3/v1-v2-v3-comparison.png)

Interpretation:

- v2's width increase produced the main gain over v1;
- v3 learned the MERT alignment target;
- v3 wins every listed aggregate metric over v2;
- the v3-v2 differences are small;
- MERT alignment did not remove the acoustic codebook-depth gradient;
- causal conditioning across acoustic books is the next structural test.

Machine-readable comparison: [`comparison-metrics.json`](evaluation/v3/comparison-metrics.json).

## Condition-Embedding Replay Comparison

Protocol:

- 130 exact-alignment holdout tracks;
- each final checkpoint predicts argmax RVQ codes from cached DAV latents;
- predicted codes are teacher-forced through the official language model and RVQ depth decoder;
- hidden states pass through the official condition encoder with recorded chunk stitching;
- reconstructed condition embeddings are compared with stored condition embeddings;
- metric: per-track mean cosine over stitched condition-latent frames;
- true sampled codes provide the replay control.

| Model | Parameters | Mean cosine | Standard deviation | 5th-95th percentile |
|---|---:|---:|---:|---:|
| Serveurperso v1 | 40,978,944 | 0.663329 | 0.022175 | 0.628052-0.696328 |
| SimpleTuner v1 | 40,978,944 | 0.762442 | 0.019550 | 0.734519-0.790450 |
| SimpleTuner v2 | 154,736,064 | 0.769841 | 0.019063 | 0.742991-0.798636 |
| SimpleTuner v3 | 154,736,064 | **0.770259** | 0.019274 | 0.741585-0.800492 |
| True-code control | - | 0.999907 | - | - |

![Condition-embedding replay comparison](evaluation/replay-comparison/condition-embedding-replay-comparison.png)

V1 exceeds the independent Serveurperso checkpoint by 0.099114 mean cosine. V2 adds 0.007399 over v1. V3 adds 0.000418 over v2. The MERT gain remains small downstream.

This test stops before diffusion and DAV decode. It is not an STFT, waveform, lyric-identity, or listening score.

Data: [`combined-aggregate.json`](evaluation/replay-comparison/combined-aggregate.json), [`provenance.json`](evaluation/replay-comparison/provenance.json), and [`raw per-record metrics`](evaluation/replay-comparison/raw-metrics-simpletuner-v3.json).

## Limitations

- 5.12-second context.
- No cross-window state.
- Synthetic model-output training domain.
- Real audio remains out of distribution.
- Teacher top-k uncertainty is from LM rollout, not an audio-conditioned posterior.
- Exact-token accuracy understates perceptual equivalence.
- Semantic CE can dominate early because its vocabulary is larger.
- Acoustic heads are independent despite residual-codebook dependence.
- Diffusion render and audio-domain evaluation remain pending.
- No stable packaged loading API.
- Use is subject to MiniMax Music 3, dataset, and MERT terms.

## Discussion and Attribution

Primary discussion: [MiniMaxAI/MiniMax-Music3 discussion #10](https://huggingface.co/MiniMaxAI/MiniMax-Music3/discussions/10).

Attribution covers public discussion, measurements, datasets, and independent experiments. It does not imply shared authorship.

- [`bghira`](https://huggingface.co/bghira): SimpleTuner experiments, trace extraction, teacher distributions, alignment records, corpus publication, and training runs.
- [`marduk191`](https://huggingface.co/marduk191): WAV/code samples, early mel encoder proof, corpus-scale observations, and encoder experiments.
- [`scragnog`](https://huggingface.co/scragnog): SimpleTuner calibration, relative-weight analysis, caption-cache and rollout-seam findings, and GGML interoperability tests.
- [`Serveurperso`](https://huggingface.co/Serveurperso): independent encoder, corpus generator, replay evaluation, and stitched-timeline findings.
- [`dernet`](https://huggingface.co/dernet): RVQ supervision, internal-alignment, and tokenizer reverse-engineering analysis.

Additional artifacts:

- [`SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1`](https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1)
- [`SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2`](https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2)
- [`marduk191/Minmax_music3_experiments`](https://huggingface.co/marduk191/Minmax_music3_experiments/tree/main/corpus)
- [`ServeurpersoCom/minimaxmusic.cpp` proof commit](https://github.com/ServeurpersoCom/minimaxmusic.cpp/commit/d19efe9f94e41ac4c900aa30d56fe90c8dac7ef1)

<!-- simpletuner-rvq-evaluation-start -->
## Offline Checkpoint Evaluation

Exact-alignment holdout: 130 tracks, 2,768 windows.

| Selection | Checkpoint | Step | Loss | Semantic top-1 | Semantic top-5 | Acoustic top-1 | Acoustic top-5 |
|---|---|---:|---:|---:|---:|---:|---:|
| best semantic top-1 | `checkpoint-17500` | 17,500 | 5.260236 | 0.4303 | 0.8048 | 0.0765 | 0.2202 |
| lowest loss; best semantic top-5; best acoustic top-1; best acoustic top-5; final | `final` | 17,660 | 5.259917 | 0.4303 | 0.8049 | 0.0766 | 0.2203 |

Top-k accuracy measures exact token inclusion. It does not measure perceptual code equivalence.

### Checkpoint Loss

![Checkpoint Loss](evaluation/v3/checkpoint-loss.png)

### Checkpoint Accuracy

![Checkpoint Accuracy](evaluation/v3/checkpoint-accuracy.png)

### Codebook Top1

![Codebook Top1](evaluation/v3/codebook-top1.png)

### Training History

![Training History](evaluation/v3/training-history.png)

### MERT Alignment

![MERT Alignment](evaluation/v3/mert-alignment.png)

Full data: [`checkpoint-metrics.csv`](evaluation/v3/checkpoint-metrics.csv), [`evaluation-metrics.json`](evaluation/v3/evaluation-metrics.json), [`comparison-metrics.json`](evaluation/v3/comparison-metrics.json).
<!-- simpletuner-rvq-evaluation-end -->