README.md

metadata

library_name: pytorch
datasets:
  - bghira/minimax-music3-rvq-reverse-distillation
tags:
  - audio
  - music
  - rvq
  - reverse-distillation
  - minimax-music-3
  - mup
  - mert

Open RVQ Encoder for MiniMax Music 3, 155M, v3

Status

• Training complete: 17,660 optimizer steps.
• Recommended checkpoint: final.
• Not an official MiniMax model.
• Not the original MiniMax Music 3 RVQ encoder.
• No original encoder weights or source code were used.
• V3 was initialized from scratch. No v1 or v2 weights were loaded.
• Real-audio generalization is not established.
• MERT was used only as a frozen training teacher. MERT weights are not included.
• A packaged from_pretrained loader is not present yet.

Result

V3 adds MERT representation alignment to the v2 architecture.

At the matched step 17,500, relative to v2:

• loss: 5.264569 -> 5.260236;
• semantic top-1: 42.86% -> 43.03%;
• semantic top-5: 80.17% -> 80.48%;
• acoustic top-1: 7.62% -> 7.65%;
• acoustic top-5: 21.98% -> 22.02%.

The MERT objective converged. Holdout MERT cosine similarity reached 0.762. RVQ accuracy changed only slightly. MERT alignment is not the main missing component for this architecture and corpus.

Objective

Approximate the missing audio-to-RVQ path used by MiniMax Music 3.

44.1 kHz waveform
  -> frozen DAV / Flow-VAE encoder
  -> 128-channel DAV latents
  -> this encoder
  -> 8 RVQ distributions per 25 Hz frame
  -> 1 semantic code + 7 acoustic codes

Head	Role	Vocabulary
0	semantic	16,384
1-7	acoustic	1,024 each

Argmax gives a discrete code stream. The downstream path replays those codes through the MiniMax Music 3 LM, condition encoder, diffusion transformer, and DAV decoder.

Architecture

Exported encoder parameters: 154,736,064.

Training-only MERT projection parameters: 835,584.

Total trainable parameters during v3 training: 155,571,648.

Component	Configuration	Parameters
DAV latent input stem	Conv1d, 128 -> 1,088, kernel 7	975,936
Local residual stack	3 blocks, dilations 1/3/9, GroupNorm, kernel-3 and kernel-1 convolutions	14,217,984
Position embedding	learned, 128 x 1,088	139,264
Transformer	8 pre-norm layers, width 1,088, 17 heads, FFN 4,352, GELU, dropout 0.1	113,752,576
Final normalization	LayerNorm(1,088)	2,176
RVQ readouts	8 independent mup.MuReadout heads	25,648,128
MERT projection	training-only mup.MuReadout, 1,088 -> 768, no bias	835,584

Processing:

1. Apply the convolutional stem and residual stack at DAV latent rate.
2. Average-pool exact DAV spans into 25 Hz frames.
3. Add learned positions.
4. Apply eight bidirectional Transformer encoder layers.
5. Apply final LayerNorm.
6. Produce eight independent RVQ distributions.

Context: 128 frames, or 5.12 seconds. There is no cross-window state.

The per-sample pool matrix preserves stitched-chunk alignment. It is not a fixed-ratio resampler.

Architecture Selection

Version	Exported parameters	Change
v1	40,978,944	512-wide baseline
v2	154,736,064	width increased to 1,088
v3	154,736,064	v2 encoder plus training-only MERT alignment

V3 keeps the v2 encoder unchanged. This isolates the MERT auxiliary objective.

The encoder still predicts the seven acoustic books independently. Head k does not receive selected codes from heads < k. The per-head results show a strong accuracy decline with codebook depth. V4 addresses that separately with a causal depth decoder.

Initialization and muTransfer

Package: microsoft/mup.

Encoder shape family:

Model	Width	Heads	Head dimension
base	128	2	64
delta	256	4	64
target	1,088	17	64

Initialization order:

1. Construct target, base, and delta training wrappers.
2. Attach wrapper-level base shapes with mup.set_base_shapes.
3. Construct mup.MuAdamW after infshapes are attached.
4. Save wrapper and exported-encoder base-shape files.

The wrapper-level shape family includes the MERT projection. Encoder-only v1/v2 base-shape files are incompatible with the v3 training wrapper.

RVQ readouts:

• mup.MuReadout;
• output multiplier 1.0;
• zero initialized;
• initial output distributions uniform within each vocabulary.

MERT projection:

• mup.MuReadout, 1,088 -> 768;
• no bias;
• nonzero initialization required for cosine loss;
• removed from exported encoder checkpoints.

Attention score scale is 8 / head_dim. At head dimension 64 this equals standard 1/sqrt(64) scaling.

Seed: 42, device-specific under DDP.

Data

Dataset: bghira/minimax-music3-rvq-reverse-distillation.

Run snapshot:

• 2,972 one-track ZIP shards;
• 2,837 training records;
• 135 holdout records before exact-alignment filtering;
• approximately 178 GB;
• synthetic tracks generated by MiniMax Music 3;
• not MiniMax's original training set.

Trainer inputs:

• waveform audio;
• sampled RVQ codes;
• teacher top-50 token IDs and logits;
• exact chunk-stitching metadata.

Waveforms are re-encoded with SimpleTuner/MiniMax-Music-3-Encoder. DAV latents are cached once. Window reads use safetensors.safe_open(...).get_slice(...).

Stored flow-VAE latents in the dataset are not consumed.

Timeline Alignment

DAV hop: 512 samples at 44.1 kHz.

Frame center:

((latent_start + latent_end) / 2) * 512 / 44100 seconds

Stitched rollout rules:

• rollout window: 200 semantic frames;
• rollout hop: 100 semantic frames;
• full stitched hop: 345 DAV latents;
• later chunks begin ownership 25 semantic frames after nominal start;
• code row 0 is warm-up;
• semantic frame i uses code row i + 1;
• the final partial chunk uses its own integer latent length;
• records without exact chunk_stitching metadata are excluded.

MERT features are linearly interpolated onto these exact DAV frame centers. A naive 3:1 reshape is not used.

MERT Alignment

Teacher: m-a-p/MERT-v1-95M.

Pinned revision: 12af15fef9d0ac838c3f475bfbbf26d2060dd4f5.

Setting	Value
Teacher layer	9
Student capture layer	4, zero-based
Teacher hidden size	768
Teacher sample rate	24 kHz
Teacher feature rate	75 Hz
Chunk length	5 seconds
Chunk overlap	1 second
Cache dtype	bfloat16
Initial alignment weight	0.5
Constant phase	0% through 70% of training
Linear decay	70% through 90%
Disabled weight	final 10%

MERT sidecars are generated before training. Cache metadata records the model, revision, hidden layers, chunk geometry, dtype, emitted frame count, and alignment version dav512-mert75-center-v1.

The projection forward remains active after its scheduled weight reaches zero. This preserves DDP parameter participation. The projection is not exported.

MERT-v1-95M is published under CC-BY-NC-4.0. This repository does not redistribute MERT weights. Users remain responsible for applicable model, dataset, and teacher terms.

Loss

reported_loss = mean(CE_head_0 ... CE_head_7)
              + 0.25 * mean(KL_head_0 ... KL_head_7)

optimization_loss = reported_loss
                  + scheduled_MERT_weight * cosine_alignment_loss

Hard targets:

• cross-entropy against sampled RVQ codes;
• equal weight for all eight heads;
• padding target -100.

Soft targets:

• teacher top-k 50;
• temperature 1.0;
• Hinton T^2 scaling;
• teacher renormalized over valid stored IDs;
• student full-vocabulary log-softmax gathered at teacher IDs;
• no student top-k renormalization;
• invalid, EOS, and out-of-vocabulary IDs excluded;
• frames with no valid teacher IDs skipped for KL.

MERT target:

• cosine distance between projected student layer 4 and frozen MERT layer 9;
• mean over batch and frames.

Reported loss excludes MERT. V1, v2, and v3 loss curves therefore remain directly comparable.

Training

Setting	Value
Hardware	4 x NVIDIA L40S
Distribution	PyTorch DDP through Accelerate
Precision	bfloat16 mixed precision
Epochs	20
Optimizer steps	17,660
Batch per rank	16
Global batch	64
Gradient accumulation	1
Optimizer	mup.MuAdamW
Learning rate	3e-4
Weight decay	0.01
LR schedule	polynomial, power 1.0
Linear warmup	500 steps
Final learning rate	1e-7
Gradient norm limit	1.0
Train crop	random 128-frame window
Validation crop	deterministic 128-frame windows
Validation interval	500 steps
Checkpoint interval	500 steps

The learning-rate multiplier warms linearly for 500 steps, then decays linearly to the final learning rate. It does not restart or reheat.

Training metrics: Weights & Biases.

Checkpoint Format

Each exported checkpoint contains:

File	Contents
rvq_encoder.safetensors	exported encoder state dictionary
rvq_encoder_config.json	architecture and muP configuration
mup_base_shapes.bsh	exported-encoder muP base shapes

The training-only MERT projection and MERT teacher are not included.

Loading currently requires RVQEncoderConfig and MiniMaxMusicRVQEncoder from scripts/train_minimax_music_rvq_encoder.py.

Evaluation

Protocol:

• exact-alignment holdout;
• 130 tracks;
• 2,768 deterministic windows;
• all 35 numbered checkpoints and final;
• four-rank evaluation; no distributed-sampler padding;
• exact-token top-1 and top-5.

Recommended final result:

Metric	Value
loss	5.259917
hard CE	4.629320
teacher KL before 0.25 weighting	2.522392
semantic top-1	43.03%
semantic top-5	80.49%
acoustic top-1	7.66%
acoustic top-5	22.03%

Top-k accuracy measures exact token inclusion. It does not measure perceptual code equivalence.

Matched Comparison at Step 17,500

Metric	v1, 41M	v2, 155M	v3, 155M + MERT	v3 vs v2
loss	5.337856	5.264569	5.260236	-0.004334
semantic top-1	41.03%	42.86%	43.03%	+0.17 pp
semantic top-5	78.38%	80.17%	80.48%	+0.31 pp
acoustic top-1	7.17%	7.62%	7.65%	+0.03 pp
acoustic top-5	20.94%	21.98%	22.02%	+0.04 pp

Interpretation:

• v2's width increase produced the main gain over v1;
• v3 learned the MERT alignment target;
• v3 wins every listed aggregate metric over v2;
• the v3-v2 differences are small;
• MERT alignment did not remove the acoustic codebook-depth gradient;
• causal conditioning across acoustic books is the next structural test.

Machine-readable comparison: comparison-metrics.json.

Condition-Embedding Replay Comparison

Protocol:

• 130 exact-alignment holdout tracks;
• each final checkpoint predicts argmax RVQ codes from cached DAV latents;
• predicted codes are teacher-forced through the official language model and RVQ depth decoder;
• hidden states pass through the official condition encoder with recorded chunk stitching;
• reconstructed condition embeddings are compared with stored condition embeddings;
• metric: per-track mean cosine over stitched condition-latent frames;
• true sampled codes provide the replay control.

Model	Parameters	Mean cosine	Standard deviation	5th-95th percentile
Serveurperso v1	40,978,944	0.663329	0.022175	0.628052-0.696328
SimpleTuner v1	40,978,944	0.762442	0.019550	0.734519-0.790450
SimpleTuner v2	154,736,064	0.769841	0.019063	0.742991-0.798636
SimpleTuner v3	154,736,064	0.770259	0.019274	0.741585-0.800492
True-code control	-	0.999907	-	-

V1 exceeds the independent Serveurperso checkpoint by 0.099114 mean cosine. V2 adds 0.007399 over v1. V3 adds 0.000418 over v2. The MERT gain remains small downstream.

This test stops before diffusion and DAV decode. It is not an STFT, waveform, lyric-identity, or listening score.

Data: combined-aggregate.json, provenance.json, and raw per-record metrics.

Limitations

• 5.12-second context.
• No cross-window state.
• Synthetic model-output training domain.
• Real audio remains out of distribution.
• Teacher top-k uncertainty is from LM rollout, not an audio-conditioned posterior.
• Exact-token accuracy understates perceptual equivalence.
• Semantic CE can dominate early because its vocabulary is larger.
• Acoustic heads are independent despite residual-codebook dependence.
• Diffusion render and audio-domain evaluation remain pending.
• No stable packaged loading API.
• Use is subject to MiniMax Music 3, dataset, and MERT terms.

Discussion and Attribution

Primary discussion: MiniMaxAI/MiniMax-Music3 discussion #10.

Attribution covers public discussion, measurements, datasets, and independent experiments. It does not imply shared authorship.

• bghira: SimpleTuner experiments, trace extraction, teacher distributions, alignment records, corpus publication, and training runs.
• marduk191: WAV/code samples, early mel encoder proof, corpus-scale observations, and encoder experiments.
• scragnog: SimpleTuner calibration, relative-weight analysis, caption-cache and rollout-seam findings, and GGML interoperability tests.
• Serveurperso: independent encoder, corpus generator, replay evaluation, and stitched-timeline findings.
• dernet: RVQ supervision, internal-alignment, and tokenizer reverse-engineering analysis.

Additional artifacts:

• SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1
• SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2
• marduk191/Minmax_music3_experiments
• ServeurpersoCom/minimaxmusic.cpp proof commit

Offline Checkpoint Evaluation

Exact-alignment holdout: 130 tracks, 2,768 windows.

Selection	Checkpoint	Step	Loss	Semantic top-1	Semantic top-5	Acoustic top-1	Acoustic top-5
best semantic top-1	checkpoint-17500	17,500	5.260236	0.4303	0.8048	0.0765	0.2202
lowest loss; best semantic top-5; best acoustic top-1; best acoustic top-5; final	final	17,660	5.259917	0.4303	0.8049	0.0766	0.2203

Top-k accuracy measures exact token inclusion. It does not measure perceptual code equivalence.

Checkpoint Loss

Checkpoint Accuracy

Codebook Top1

Training History

MERT Alignment

Full data: checkpoint-metrics.csv, evaluation-metrics.json, comparison-metrics.json.