SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2

Open RVQ Encoder for MiniMax Music 3, 155M, v2

Status

• V2 training is complete.
• Recommended checkpoint: final at step 17,660 for lowest loss and best aggregate top-5 accuracy.
• checkpoint-17500 has the best aggregate semantic and acoustic top-1 accuracy by less than 0.01 percentage point.
• Not an official MiniMax model.
• Not the original MiniMax Music 3 RVQ encoder.
• No original encoder weights or source code were used.
• No v1 checkpoint weights were loaded.
• Real-audio generalization is not established.
• A packaged from_pretrained loader is not present yet.

Objective

Approximate the missing audio-to-RVQ path used by MiniMax Music 3.

Input path:

44.1 kHz waveform
  -> frozen DAV / Flow-VAE encoder
  -> 128-channel DAV latents
  -> this encoder
  -> 8 RVQ distributions per 25 Hz frame
  -> 1 semantic code + 7 acoustic codes

Output vocabularies:

Head	Role	Vocabulary
0	semantic	16,384
1-7	acoustic	1,024 each

The model predicts code distributions. Argmax produces a discrete code stream. The intended downstream test replays those codes through the MiniMax Music 3 LM, condition encoder, diffusion transformer, and DAV decoder.

Architecture

Exact trainable parameter count: 154,736,064.

Component	Configuration	Parameters
DAV latent input stem	Conv1d, 128 -> 1,088, kernel 7	975,936
Local residual stack	3 blocks, dilations 1/3/9, GroupNorm, kernel-3 + kernel-1 convolutions	14,217,984
Position embedding	learned, 128 x 1,088	139,264
Transformer	8 pre-norm layers, width 1,088, 17 heads, FFN 4,352, GELU, dropout 0.1	113,752,576
Final normalization	LayerNorm(1,088)	2,176
RVQ readouts	8 independent mup.MuReadout heads	25,648,128

Processing order:

1. Apply the convolutional stem and residual stack at DAV latent rate.
2. Average-pool exact DAV latent spans into 25 Hz semantic frames.
3. Add learned positions.
4. Apply eight bidirectional Transformer encoder layers.
5. Apply final LayerNorm.
6. Produce one logit tensor per RVQ codebook.

The pool matrix is supplied with each sample. It is not a fixed-ratio resampler. This preserves stitched-chunk alignment.

Context: 128 semantic frames = 5.12 seconds. There is no cross-window state.

Architecture Selection

• V1 established the 512-wide, 8-layer baseline at 40,978,944 parameters.
• V2 tests width while keeping depth, receptive field, context, loss, data, batch size, and output schema fixed.
• Width 1,024 produces 138,557,440 parameters and does not clear the requested 150M threshold.
• Width 1,088 is the smallest 64-aligned width above 150M with the v1 depth and FFN multiplier.
• Seventeen attention heads preserve the 64-dimensional head size used by the base, delta, and v1 models.
• The parameter count is 3.78 times v1.
• The run tests whether v1 was capacity-limited. It does not test deeper context, autoregressive codebook-depth prediction, or MERT alignment.

Initialization and μTransfer

Package: microsoft/mup.

Shape family:

Model	Width	Heads	Head dimension
base	128	2	64
delta	256	4	64
v1	512	8	64
v2 target	1,088	17	64

Initialization sequence:

1. Construct target, base, and delta models.
2. Call mup.set_base_shapes(target, base, delta=delta).
3. Delete base and delta models.
4. Construct mup.MuAdamW after infshapes are attached.
5. Save mup_base_shapes.bsh with each exported checkpoint.

Readouts:

• All eight output layers are mup.MuReadout.
• output_mult = 1.0.
• readout_zero_init = true.
• Readout weights and biases start at zero.
• Initial output distributions are uniform within each vocabulary.

Attention:

• Score scale: attention_multiplier / head_dim.
• attention_multiplier = 8.0.
• Target scale: 8 / 64 = 1/8, equal to standard 1/sqrt(64) scaling.
• Head dimension remains 64 across base, delta, v1, and v2 widths.

Other parameters:

• Learned positions use Normal(0, 0.02).
• Convolution, attention, FFN, and normalization modules use their PyTorch initializers before μP shape metadata is attached.
• Seed: 42, device-specific under DDP.

V2 is randomly initialized. V1 supplies the width-family design and training calibration only. μTransfer here means width-aware parametrization and optimizer scaling, not weight transfer from v1.

Data

Dataset: bghira/minimax-music3-rvq-reverse-distillation.

Run-launch snapshot:

• 2,972 one-track ZIP shards.
• 2,837 training records.
• 135 holdout records.
• Approximately 178 GB.
• Synthetic tracks generated by MiniMax Music 3.
• This is not MiniMax's original training set.

Fields consumed by this trainer:

• waveform audio;
• sampled RVQ codes;
• teacher top-50 token IDs;
• teacher top-50 logits;
• exact chunk-stitching metadata.

The corpus also contains stored flow-VAE latents. This trainer does not consume them. It re-encodes waveform audio with SimpleTuner/MiniMax-Music-3-Encoder and caches DAV latents once.

Cached windows use safetensors.safe_open(...).get_slice(...). Full-track latent tensors are not loaded for each crop.

Alignment

Nominal DAV ratio: 441 / 128 = 3.4453125 latents per semantic frame.

The actual stitched timeline is not a global multiplication by that ratio.

• Autoregressive rollout window: 200 semantic frames.
• Rollout hop: 100 semantic frames.
• Full stitched hop: 345 DAV latents.
• Later chunks begin ownership 25 semantic frames after their nominal start.
• Code row 0 is warm-up/priming.
• Semantic frame i is supervised by code row i + 1.
• The final partial chunk uses its own integer latent length.
• Per-shard chunk_stitching bounds define the pool spans.
• Training uses exact-alignment mode. Records without chunk_stitching metadata are excluded.

These rules prevent cumulative label drift and training across incorrectly assigned rollout seams.

Objective Function

loss = mean(CE_head_0 ... CE_head_7)
     + 0.25 * mean(KL_head_0 ... KL_head_7)

Hard targets:

• Cross-entropy against sampled RVQ codes.
• Equal weight for all eight heads.
• Padding target: -100.

Soft targets:

• Teacher top-k: 50.
• Temperature: 1.0.
• Hinton T^2 scaling.
• Teacher distribution is renormalized over valid stored top-50 IDs.
• Student uses full-vocabulary log-softmax, then gathers the teacher IDs.
• Student probabilities are not renormalized over the top-50 subset.
• Negative, EOS, and out-of-vocabulary teacher IDs are excluded.
• Remaining teacher mass is renormalized after exclusion.
• Frames with no valid teacher IDs are skipped for KL.

The teacher logits come from LM predictions before audio-conditioned encoder output is available. Their uncertainty is useful but is not identical to an audio-conditioned posterior. This is why KL weight is 0.25 rather than 1.0.

Equal head averaging is simple but imperfect. The semantic head has a much larger vocabulary and can dominate early CE. Per-head weighting is a possible follow-up.

Training Run

Setting	Value
Hardware	4 x NVIDIA L40S
Distribution	PyTorch DDP through Accelerate
Precision	bfloat16 mixed precision
Epochs	20
Optimizer steps	17,660
Batch per rank	16
Global batch	64
Gradient accumulation	1
Optimizer	mup.MuAdamW
Learning rate	3e-4
Weight decay	0.01
LR schedule	polynomial
Linear warmup	500 global steps
Final learning rate	1e-7
Polynomial power	1.0
Gradient norm limit	1.0
Train crop	random 128-frame window
Validation crop	deterministic 128-frame windows
Validation interval	500 steps
Checkpoint interval	500 steps

The learning-rate multiplier warms linearly for 500 optimizer steps. It then decays with polynomial power 1.0 to the configured final learning rate. Unlike v1's limited-data cosine schedule, v2 does not restart or reheat.

Training metrics: Weights & Biases run.

Representative command:

torchrun --standalone --nproc_per_node=4 scripts/train_minimax_music_rvq_encoder.py \
  --dataset_repo_id bghira/minimax-music3-rvq-reverse-distillation \
  --pretrained_vae_model_name_or_path SimpleTuner/MiniMax-Music-3-Encoder \
  --latent_cache_dir cache/vae/minimaxmusic-rvq-encoder \
  --output_dir output/minimaxmusic-rvq-encoder-155m-v2 \
  --require_exact_alignment \
  --num_train_epochs 20 \
  --train_batch_size 16 \
  --mixed_precision bf16 \
  --optimizer torch-adamw \
  --learning_rate 3e-4 \
  --weight_decay 0.01 \
  --lr_scheduler polynomial \
  --lr_warmup_steps 500 \
  --lr_end 1e-7 \
  --lr_power 1.0 \
  --teacher_kl_weight 0.25 \
  --teacher_kl_temperature 1.0 \
  --window_frames 128 \
  --window_stride 128 \
  --d_model 1088 \
  --layers 8 \
  --heads 17 \
  --ff_mult 4 \
  --dropout 0.1 \
  --mup \
  --mup_base_d_model 128 \
  --mup_delta_d_model 256 \
  --mup_readout_zero_init \
  --checkpointing_steps 500 \
  --validation_steps 500 \
  --report_to wandb \
  --tracker_project_name simpletuner-rvq-encoder \
  --tracker_run_name minimax-music3-155m-v2 \
  --push_to_hub SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2

Checkpoint Format

Each exported checkpoint contains:

File	Contents
rvq_encoder.safetensors	model state dictionary
rvq_encoder_config.json	architecture and μP configuration
mup_base_shapes.bsh	μP base-shape metadata

Trainer state, optimizer state, local paths, and credentials are not uploaded to this model repository.

Loading currently requires the matching RVQEncoderConfig and MiniMaxMusicRVQEncoder definitions from scripts/train_minimax_music_rvq_encoder.py.

Evaluation

Exact-alignment holdout:

• 130 tracks.
• 2,768 deterministic windows.
• Every numbered checkpoint and final evaluated over the full holdout.
• Top-k accuracy measures exact token inclusion, not perceptual code equivalence.

Recommended final result:

Metric	Value
loss	5.264375
hard CE	4.633118
teacher KL before 0.25 weighting	2.525017
semantic top-1	42.86%
semantic top-5	80.18%
acoustic top-1	7.62%
acoustic top-5	21.98%

Machine-readable raw statistics: training-stats.json. It contains 883 training records and 36 full-holdout checkpoint evaluations. The JSON includes metric definitions and provenance.

V1 Comparison

Matched checkpoint: step 17,500 for both models.

Holdout metric	v1, 41M	v2, 155M	v2 delta	Result
loss	5.337856	5.264569	-0.073287	v2 wins
semantic top-1	41.03%	42.86%	+1.84 pp	v2 wins
semantic top-5	78.38%	80.17%	+1.79 pp	v2 wins
acoustic top-1	7.17%	7.62%	+0.45 pp	v2 wins
acoustic top-5	20.94%	21.98%	+1.04 pp	v2 wins

V2 loses none of the aggregate metrics above. The gain is consistent but modest relative to the 3.78x parameter increase. Width was mildly limiting. It was not the main ceiling.

The acoustic depth gradient remains. Early acoustic heads are substantially more accurate than later heads. Independent readouts do not condition head k on selected heads < k. Wider hidden states do not remove that structural limitation.

Required end-to-end acceptance test:

1. Encode held-out waveform to DAV latents.
2. Predict eight codes per frame.
3. Replay predicted codes through the official LM path.
4. Compare replayed condition embeddings with stored condition embeddings.
5. Run the condition encoder, diffusion transformer, and DAV decoder.
6. Compare reconstructed audio and lyric identity with the source generation.

Condition-embedding replay is implemented in a separate offline harness. It is not part of the trainer. Token top-1 is insufficient because multiple code sequences can be perceptually equivalent.

Prior independent evidence from Serveurperso:

• held-out STFT similarity: 0.83 to 0.87;
• exact-code replay STFT similarity: 0.998;
• acoustic exact-token match: 3% to 6%;
• same music and lyrics remained identifiable after predicted-code replay;
• a 550-track corpus overfit by epoch 17.

Those numbers are from a separate encoder and training stack. They are not results for this checkpoint.

Condition-Embedding Replay Comparison

Protocol:

• 130 exact-alignment holdout tracks;
• each final checkpoint predicts argmax RVQ codes from cached DAV latents;
• predicted codes are teacher-forced through the official language model and RVQ depth decoder;
• hidden states pass through the official condition encoder with recorded chunk stitching;
• reconstructed condition embeddings are compared with stored condition embeddings;
• metric: per-track mean cosine over stitched condition-latent frames;
• true sampled codes provide the replay control.

Model	Parameters	Mean cosine	Standard deviation	5th-95th percentile
Serveurperso v1	40,978,944	0.663329	0.022175	0.628052-0.696328
SimpleTuner v1	40,978,944	0.762442	0.019550	0.734519-0.790450
SimpleTuner v2	154,736,064	0.769841	0.019063	0.742991-0.798636
SimpleTuner v3	154,736,064	0.770259	0.019274	0.741585-0.800492
True-code control	-	0.999907	-	-

V1 exceeds the independent Serveurperso checkpoint by 0.099114 mean cosine. V2 adds 0.007399 over v1. V3 adds 0.000418 over v2. The MERT gain remains small downstream.

This test stops before diffusion and DAV decode. It is not an STFT, waveform, lyric-identity, or listening score.

Data: combined-aggregate.json, provenance.json, and raw per-record metrics.

Limitations

• 5.12-second encoder context.
• No cross-window memory.
• Synthetic model-output training domain.
• Real audio is out of distribution until demonstrated otherwise.
• Teacher uncertainty is from the LM rollout, not an audio-conditioned teacher encoder.
• Exact token accuracy understates perceptual equivalence.
• Semantic CE may dominate acoustic CE early.
• Acoustic heads are predicted independently despite residual-codebook depth dependence.
• Diffusion render and audio-domain evaluation remain pending.
• Loading is not packaged as a stable library API.
• Use is subject to the MiniMax Music 3 model terms and the reverse-distillation dataset terms.

Discussion and Experimental Inputs

Primary discussion: MiniMaxAI/MiniMax-Music3 discussion #10, "Is the model trainable?".

Attribution below is for public discussion, measurements, datasets, and independent experiments. It does not imply shared authorship of this implementation.

• bghira: ran the SimpleTuner training experiments; extracted sampled codes, teacher distributions, and alignment records; published the reverse-distillation corpus; organized this compatible-encoder run.
• marduk191: published WAV/code samples; built an early mel-based encoder proof of concept; reported small-corpus and real-audio limits; tested additional encoder variants.
• scragnog: calibrated HOT-Step CPP training against SimpleTuner; reported relative-weight-movement and loss measurements; identified structured-caption cache behavior and conditioning-rollout seam effects; confirmed SimpleTuner LoRA export interoperability with GGML.
• Serveurperso: independently built a 41M encoder, corpus generator, loader, and replay evaluation stack; demonstrated viable predicted-code replay; identified the stitched-hop, warm-up-row, and final-partial-chunk alignment rules.
• dernet: explained why inference-time internal alignment does not provide target-derived alignment during training; clarified the role of RVQ token supervision; contributed tokenizer reverse-engineering analysis.

Additional public artifacts:

• SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1
• marduk191/Minmax_music3_experiments corpus
• ServeurpersoCom/minimaxmusic.cpp encoder proof commit

Offline Checkpoint Evaluation

Exact-alignment holdout: 130 tracks, 2,768 windows.

Selection	Checkpoint	Step	Loss	Semantic top-1	Semantic top-5	Acoustic top-1	Acoustic top-5
best semantic top-1; best acoustic top-1	checkpoint-17500	17,500	5.264569	0.4286	0.8017	0.0762	0.2198
lowest loss; best semantic top-5; best acoustic top-5; final	final	17,660	5.264375	0.4286	0.8018	0.0762	0.2198

Top-k accuracy measures exact token inclusion. It does not measure perceptual code equivalence.

Checkpoint Loss

Checkpoint Accuracy

Codebook Top1

Training History

Full data: checkpoint-metrics.csv, evaluation-metrics.json, comparison-metrics.json.