SimpleTuner/open-rvq-encoder-minimax-music3-169m-v4

Open RVQ Encoder for MiniMax Music 3, 169M, v4

Status

• Training complete: 17,660 optimizer steps.
• Recommended checkpoint: final.
• checkpoint-17500 has the best free-running acoustic top-1 by 0.009 percentage point.
• Not an official MiniMax model.
• Not the original MiniMax Music 3 RVQ encoder.
• No original encoder weights or source code were used.
• V4 was initialized from scratch. No v1, v2, or v3 weights were loaded.
• Real-audio generalization is not established.
• A packaged from_pretrained loader is not present yet.

Result

V4 replaces the seven independent acoustic readouts with a causal decoder across RVQ depth.

At step 17,500, relative to v3:

Metric	v3	v4 free-running	Change
semantic top-1	43.03%	43.16%	+0.13 pp
semantic top-5	80.48%	80.51%	+0.03 pp
acoustic top-1	7.65%	7.30%	-0.35 pp
acoustic top-5	22.02%	19.88%	-2.14 pp

V4 teacher-forced acoustic accuracy at the same step:

Metric	Free-running	Teacher-forced
acoustic top-1	7.30%	18.42%
acoustic top-5	19.88%	44.54%

The decoder learned strong conditional distributions. Greedy error propagation lowers exact agreement with the one sampled label sequence.

That exact-match result is not the primary outcome. The RVQ code space is redundant. Multiple code tuples can produce the same downstream conditioning.

Condition-embedding replay through the official LM and RVQ depth decoder:

Model	Mean cosine	Change from v3
v3	0.770259	-
v4	0.874796	+0.104537

V4 is substantially more functionally correct by the downstream metric. Exact sampled-token accuracy is a weak proxy for that result.

V4 loss is conditional on true previous codebooks. It is not directly comparable to the independent-head v1-v3 loss.

Objective

Approximate the missing audio-to-RVQ path used by MiniMax Music 3.

44.1 kHz waveform
  -> frozen DAV / Flow-VAE encoder
  -> 128-channel DAV latents
  -> shared audio encoder
  -> semantic distribution
  -> causal RVQ-depth decoder
  -> 7 acoustic distributions

Head	Role	Vocabulary
0	semantic	16,384
1-7	acoustic	1,024 each

The downstream path replays selected codes through the MiniMax Music 3 language model, official RVQ depth decoder, condition encoder, diffusion transformer, and DAV decoder.

Architecture

Exported parameters: 169,008,576.

Component	Configuration	Parameters
DAV latent input stem	Conv1d, 128 -> 1,088, kernel 7	975,936
Local residual stack	3 blocks, dilations 1/3/9	14,217,984
Position embedding	learned, 128 x 1,088	139,264
Shared Transformer	8 pre-norm layers, width 1,088, 17 heads, FFN 4,352	113,752,576
Final normalization	LayerNorm(1,088)	2,176
Semantic readout	mup.MuReadout, 1,088 -> 16,384	17,842,176
Causal depth decoder	width 512, 8 heads, 2 layers, FFN 2,048	22,078,464

Shared audio processing:

1. Apply the convolutional stem and residual stack at DAV latent rate.
2. Average-pool exact DAV spans into 25 Hz frames.
3. Add learned frame positions.
4. Apply eight bidirectional Transformer layers.
5. Produce the semantic distribution.

Context: 128 frames, or 5.12 seconds. There is no cross-window state.

The per-sample pool matrix preserves stitched-chunk alignment. It is not a fixed-ratio resampler.

Causal Depth Decoder

The depth decoder operates independently for each audio frame.

Configuration:

• width: 512;
• layers: 2;
• attention heads: 8;
• head dimension: 64;
• FFN multiplier: 4;
• dropout: 0.1;
• learned position for depth positions 0 through 7;
• separate prior-token embedding per codebook;
• separate 1,024-way output head per acoustic codebook.

Sequence for acoustic book k:

[projected audio context, semantic code, acoustic code 1, ..., acoustic code k-1]

Training uses true earlier codes. Inference uses argmax predictions from earlier books.

The causal mask prevents access to later codebooks. The decoder is autoregressive across RVQ depth, not across time.

Architecture Selection

Version	Parameters	Change
v1	40,978,944	512-wide baseline
v2	154,736,064	shared width increased to 1,088
v3	154,736,064	v2 plus training-only MERT alignment
v4	169,008,576	v2 shared encoder plus causal depth decoder

V3 showed only a small gain from MERT alignment. V2 and v3 showed a monotonic top-1 decline across acoustic codebook depth. V4 tests whether conditioning book k on books < k removes that decline.

The teacher-forced result confirms the conditional-dependence diagnosis. The free-running result exposes a second problem: training and inference histories differ.

Initialization and muTransfer

Package: microsoft/mup.

Shared encoder shape family:

Model	Width	Heads	Head dimension
base	128	2	64
delta	256	4	64
target	1,088	17	64

Depth topology is present in base, delta, and target shape models. Depth width remains fixed at 512.

Initialization order:

1. Construct target, base, and delta models with the depth decoder attached.
2. Call mup.set_base_shapes before optimizer construction.
3. Construct mup.MuAdamW after infshapes are attached.
4. Save depth-scoped base-shape metadata with each checkpoint.

V1-v3 encoder-only shape files are rejected for v4.

Semantic readout:

• mup.MuReadout;
• output multiplier 1.0;
• zero initialized;
• initial semantic distribution uniform.

The fixed-width depth decoder uses standard linear layers except for the 1,088 -> 512 context projection, which is a nonzero-initialized mup.MuReadout.

Shared attention score scale: 8 / head_dim. At head dimension 64 this is 1/8.

Seed: 42, device-specific under DDP.

Data

Dataset: bghira/minimax-music3-rvq-reverse-distillation.

Run snapshot:

• 2,972 one-track ZIP shards;
• 2,837 training records;
• 135 holdout records before exact-alignment filtering;
• approximately 178 GB;
• synthetic tracks generated by MiniMax Music 3;
• not MiniMax's original training set.

Trainer inputs:

• waveform audio;
• sampled RVQ codes;
• teacher top-50 token IDs and logits;
• exact chunk-stitching metadata.

Waveforms are re-encoded with SimpleTuner/MiniMax-Music-3-Encoder. DAV latents are cached once. Window reads use safetensors.safe_open(...).get_slice(...).

Stored flow-VAE latents in the dataset are not consumed.

Timeline Alignment

DAV hop: 512 samples at 44.1 kHz.

Frame center:

((latent_start + latent_end) / 2) * 512 / 44100 seconds

Stitched rollout rules:

• rollout window: 200 semantic frames;
• rollout hop: 100 semantic frames;
• full stitched hop: 345 DAV latents;
• later chunks begin ownership 25 semantic frames after nominal start;
• code row 0 is warm-up;
• semantic frame i uses code row i + 1;
• the final partial chunk uses its own integer latent length;
• records without exact chunk_stitching metadata are excluded.

Loss

loss = mean(CE_head_0 ... CE_head_7)
     + 0.25 * mean(KL_head_0 ... KL_head_7)

For heads 1 through 7, logits are teacher-forced on true previous codebooks during training and loss evaluation.

Hard targets:

• cross-entropy against sampled RVQ codes;
• equal weight for all eight heads;
• padding target -100.

Soft targets:

• teacher top-k 50;
• temperature 1.0;
• Hinton T^2 scaling;
• teacher renormalized over valid stored IDs;
• student full-vocabulary log-softmax gathered at teacher IDs;
• no student top-k renormalization;
• invalid, EOS, and out-of-vocabulary IDs excluded.

Training

Setting	Value
Hardware	4 x NVIDIA L40S
Distribution	PyTorch DDP through Accelerate
Precision	bfloat16 mixed precision
Epochs	20
Optimizer steps	17,660
Batch per rank	16
Global batch	64
Gradient accumulation	1
Optimizer	mup.MuAdamW
Learning rate	3e-4
Weight decay	0.01
LR schedule	polynomial, power 1.0
Linear warmup	500 steps
Final learning rate	1e-7
Gradient norm limit	1.0
Train crop	random 128-frame window
Validation crop	deterministic 128-frame windows
Validation interval	500 steps
Checkpoint interval	500 steps

The learning-rate multiplier warms linearly for 500 steps, then decays linearly to the final learning rate. It does not restart or reheat.

Training metrics: Weights & Biases.

Representative command:

accelerate launch --num_processes 4 scripts/train_minimax_music_rvq_encoder.py \
  --dataset_repo_id bghira/minimax-music3-rvq-reverse-distillation \
  --pretrained_vae_model_name_or_path SimpleTuner/MiniMax-Music-3-Encoder \
  --latent_cache_dir cache/vae/minimaxmusic-rvq-encoder \
  --output_dir output/minimaxmusic-rvq-encoder-169m-v4 \
  --d_model 1088 --layers 8 --heads 17 --ff_mult 4 \
  --depth_decoder --depth_decoder_dim 512 --depth_decoder_layers 2 \
  --depth_decoder_heads 8 --depth_decoder_ff_mult 4 \
  --mup --mup_readout_zero_init --optimizer torch-adamw \
  --learning_rate 3e-4 --lr_scheduler polynomial --lr_warmup_steps 500 \
  --lr_end 1e-7 --lr_power 1.0 --weight_decay 0.01 \
  --teacher_kl_weight 0.25 --teacher_kl_temperature 1.0 \
  --train_batch_size 16 --validation_batch_size 16 --num_train_epochs 20 \
  --mixed_precision bf16 --require_exact_alignment \
  --report_to wandb --tracker_project_name simpletuner-rvq-encoder \
  --push_to_hub SimpleTuner/open-rvq-encoder-minimax-music3-169m-v4

Checkpoint Format

File	Contents
rvq_encoder.safetensors	exported encoder and depth-decoder state dictionary
rvq_encoder_config.json	architecture and muP configuration
mup_base_shapes.bsh	exported depth-topology muP base shapes
mup_base_shapes.bsh.meta.json	shape scope and depth configuration

Loading currently requires RVQEncoderConfig and MiniMaxMusicRVQEncoder from scripts/train_minimax_music_rvq_encoder.py.

Evaluation

Protocol:

• exact-alignment holdout;
• 130 tracks;
• 2,768 deterministic windows;
• all 35 numbered checkpoints and final;
• four-rank evaluation;
• free-running argmax metrics;
• teacher-forced depth metrics reported separately.

Final checkpoint:

Metric	Free-running	Teacher-forced
semantic top-1	43.17%	43.17%
semantic top-5	80.51%	80.51%
acoustic top-1	7.29%	18.42%
acoustic top-5	19.88%	44.55%

Conditional final loss:

Metric	Value
total	3.987436
hard CE	3.514642
teacher KL before 0.25 weighting	1.891180

Top-k accuracy measures exact token inclusion. It does not measure perceptual code equivalence.

Matched Comparison at Step 17,500

Machine-readable comparison: comparison-metrics.json.

Condition-Embedding Replay Comparison

Protocol:

• 130 exact-alignment holdout tracks;
• each final checkpoint predicts free-running argmax RVQ codes from cached DAV latents;
• predicted codes are teacher-forced through the official language model and official RVQ depth decoder;
• hidden states pass through the official condition encoder with recorded chunk stitching;
• reconstructed condition embeddings are compared with stored condition embeddings;
• metric: per-track mean cosine over stitched condition-latent frames;
• true sampled codes provide the replay control.

Model	Parameters	Mean cosine	Standard deviation	5th-95th percentile
Serveurperso v1	40,978,944	0.663329	0.022175	0.628052-0.696328
SimpleTuner v1	40,978,944	0.762442	0.019550	0.734519-0.790450
SimpleTuner v2	154,736,064	0.769841	0.019063	0.742991-0.798636
SimpleTuner v3	154,736,064	0.770259	0.019274	0.741585-0.800492
SimpleTuner v4	169,008,576	0.874796	0.015662	0.845350-0.893839
True-code control	-	0.999907	0.000150	0.999821-0.999950

V4 adds 0.104537 mean cosine over v3. Its 5th-percentile track exceeds the best v3 track in this holdout. The depth decoder therefore improves code-tuple compatibility even though exact agreement with the sampled tokens declines slightly.

This test stops before diffusion and DAV decode. It is not an STFT, waveform, lyric-identity, or listening score.

Data: combined-aggregate.json, provenance.json, and raw per-record metrics.

Limitations

• 5.12-second temporal context.
• No cross-window state.
• Synthetic model-output training domain.
• Real audio remains out of distribution.
• Teacher-forced depth training has a large exact-token exposure gap under greedy decoding.
• Exact-token accuracy does not identify all functionally equivalent RVQ tuples.
• Conditional v4 loss is not comparable to v1-v3 loss.
• No beam search, scheduled sampling, or differentiable depth sampling was tested.
• Diffusion render and audio-domain evaluation remain pending.
• No stable packaged loading API.
• Use is subject to MiniMax Music 3 and dataset terms.

Discussion and Attribution

Primary discussion: MiniMaxAI/MiniMax-Music3 discussion #10.

Attribution covers public discussion, measurements, datasets, and independent experiments. It does not imply shared authorship.

• bghira: SimpleTuner experiments, trace extraction, teacher distributions, alignment records, corpus publication, and training runs.
• marduk191: WAV/code samples, early mel encoder proof, corpus-scale observations, and encoder experiments.
• scragnog: SimpleTuner calibration, relative-weight analysis, caption-cache and rollout-seam findings, and GGML interoperability tests.
• Serveurperso: independent encoder, corpus generator, replay evaluation, and stitched-timeline findings.
• dernet: RVQ supervision, internal-alignment, and tokenizer reverse-engineering analysis.

Related artifacts:

• SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1
• SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2
• SimpleTuner/open-rvq-encoder-minimax-music3-155m-v3
• marduk191/Minmax_music3_experiments
• ServeurpersoCom/minimaxmusic.cpp proof commit

Offline Checkpoint Evaluation

Exact-alignment holdout: 130 tracks, 2,768 windows.

Selection	Checkpoint	Step	Loss	Semantic top-1	Semantic top-5	Acoustic top-1	Acoustic top-5
lowest loss; best semantic top-1; final	final	17,660	3.987436	0.4317	0.8051	0.0729	0.1988
best acoustic top-1	checkpoint-17500	17,500	3.988227	0.4316	0.8051	0.0730	0.1988

Top-k accuracy measures exact token inclusion. It does not measure perceptual code equivalence.

Checkpoint Loss

Checkpoint Accuracy

Codebook Top-1

Depth Teacher-Forcing Gap

Training History

Full data: checkpoint-metrics.csv, evaluation-metrics.json, and comparison-metrics.json.