Document v3 architecture and evaluation

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+evaluation/v3/checkpoint-accuracy.png filter=lfs diff=lfs merge=lfs -text
+evaluation/v3/codebook-top1.png filter=lfs diff=lfs merge=lfs -text
+evaluation/v3/mert-alignment.png filter=lfs diff=lfs merge=lfs -text
+evaluation/v3/training-history.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,409 @@
+---
+library_name: pytorch
+datasets:
+- bghira/minimax-music3-rvq-reverse-distillation
+tags:
+- audio
+- music
+- rvq
+- reverse-distillation
+- minimax-music-3
+- mup
+- mert
+---
+
+# Open RVQ Encoder for MiniMax Music 3, 155M, v3
+
+## Status
+
+- Training complete: 17,660 optimizer steps.
+- Recommended checkpoint: `final`.
+- Not an official MiniMax model.
+- Not the original MiniMax Music 3 RVQ encoder.
+- No original encoder weights or source code were used.
+- V3 was initialized from scratch. No v1 or v2 weights were loaded.
+- Real-audio generalization is not established.
+- MERT was used only as a frozen training teacher. MERT weights are not included.
+- A packaged `from_pretrained` loader is not present yet.
+
+## Result
+
+V3 adds MERT representation alignment to the v2 architecture.
+
+At the matched step 17,500, relative to v2:
+
+- loss: 5.264569 -> 5.260236;
+- semantic top-1: 42.86% -> 43.03%;
+- semantic top-5: 80.17% -> 80.48%;
+- acoustic top-1: 7.62% -> 7.65%;
+- acoustic top-5: 21.98% -> 22.02%.
+
+The MERT objective converged. Holdout MERT cosine similarity reached 0.762. RVQ accuracy changed only slightly. MERT alignment is not the main missing component for this architecture and corpus.
+
+## Objective
+
+Approximate the missing audio-to-RVQ path used by MiniMax Music 3.
+
+```text
+44.1 kHz waveform
+  -> frozen DAV / Flow-VAE encoder
+  -> 128-channel DAV latents
+  -> this encoder
+  -> 8 RVQ distributions per 25 Hz frame
+  -> 1 semantic code + 7 acoustic codes
+```
+
+| Head | Role | Vocabulary |
+|---:|---|---:|
+| 0 | semantic | 16,384 |
+| 1-7 | acoustic | 1,024 each |
+
+Argmax gives a discrete code stream. The downstream path replays those codes through the MiniMax Music 3 LM, condition encoder, diffusion transformer, and DAV decoder.
+
+## Architecture
+
+Exported encoder parameters: **154,736,064**.
+
+Training-only MERT projection parameters: **835,584**.
+
+Total trainable parameters during v3 training: **155,571,648**.
+
+| Component | Configuration | Parameters |
+|---|---|---:|
+| DAV latent input stem | Conv1d, 128 -> 1,088, kernel 7 | 975,936 |
+| Local residual stack | 3 blocks, dilations 1/3/9, GroupNorm, kernel-3 and kernel-1 convolutions | 14,217,984 |
+| Position embedding | learned, 128 x 1,088 | 139,264 |
+| Transformer | 8 pre-norm layers, width 1,088, 17 heads, FFN 4,352, GELU, dropout 0.1 | 113,752,576 |
+| Final normalization | LayerNorm(1,088) | 2,176 |
+| RVQ readouts | 8 independent `mup.MuReadout` heads | 25,648,128 |
+| MERT projection | training-only `mup.MuReadout`, 1,088 -> 768, no bias | 835,584 |
+
+Processing:
+
+1. Apply the convolutional stem and residual stack at DAV latent rate.
+2. Average-pool exact DAV spans into 25 Hz frames.
+3. Add learned positions.
+4. Apply eight bidirectional Transformer encoder layers.
+5. Apply final LayerNorm.
+6. Produce eight independent RVQ distributions.
+
+Context: 128 frames, or 5.12 seconds. There is no cross-window state.
+
+The per-sample pool matrix preserves stitched-chunk alignment. It is not a fixed-ratio resampler.
+
+## Architecture Selection
+
+| Version | Exported parameters | Change |
+|---|---:|---|
+| v1 | 40,978,944 | 512-wide baseline |
+| v2 | 154,736,064 | width increased to 1,088 |
+| v3 | 154,736,064 | v2 encoder plus training-only MERT alignment |
+
+V3 keeps the v2 encoder unchanged. This isolates the MERT auxiliary objective.
+
+The encoder still predicts the seven acoustic books independently. Head `k` does not receive selected codes from heads `< k`. The per-head results show a strong accuracy decline with codebook depth. V4 addresses that separately with a causal depth decoder.
+
+## Initialization and muTransfer
+
+Package: [`microsoft/mup`](https://github.com/microsoft/mup).
+
+Encoder shape family:
+
+| Model | Width | Heads | Head dimension |
+|---|---:|---:|---:|
+| base | 128 | 2 | 64 |
+| delta | 256 | 4 | 64 |
+| target | 1,088 | 17 | 64 |
+
+Initialization order:
+
+1. Construct target, base, and delta training wrappers.
+2. Attach wrapper-level base shapes with `mup.set_base_shapes`.
+3. Construct `mup.MuAdamW` after infshapes are attached.
+4. Save wrapper and exported-encoder base-shape files.
+
+The wrapper-level shape family includes the MERT projection. Encoder-only v1/v2 base-shape files are incompatible with the v3 training wrapper.
+
+RVQ readouts:
+
+- `mup.MuReadout`;
+- output multiplier 1.0;
+- zero initialized;
+- initial output distributions uniform within each vocabulary.
+
+MERT projection:
+
+- `mup.MuReadout`, 1,088 -> 768;
+- no bias;
+- nonzero initialization required for cosine loss;
+- removed from exported encoder checkpoints.
+
+Attention score scale is `8 / head_dim`. At head dimension 64 this equals standard `1/sqrt(64)` scaling.
+
+Seed: 42, device-specific under DDP.
+
+## Data
+
+Dataset: [`bghira/minimax-music3-rvq-reverse-distillation`](https://huggingface.co/datasets/bghira/minimax-music3-rvq-reverse-distillation).
+
+Run snapshot:
+
+- 2,972 one-track ZIP shards;
+- 2,837 training records;
+- 135 holdout records before exact-alignment filtering;
+- approximately 178 GB;
+- synthetic tracks generated by MiniMax Music 3;
+- not MiniMax's original training set.
+
+Trainer inputs:
+
+- waveform audio;
+- sampled RVQ codes;
+- teacher top-50 token IDs and logits;
+- exact chunk-stitching metadata.
+
+Waveforms are re-encoded with [`SimpleTuner/MiniMax-Music-3-Encoder`](https://huggingface.co/SimpleTuner/MiniMax-Music-3-Encoder). DAV latents are cached once. Window reads use `safetensors.safe_open(...).get_slice(...)`.
+
+Stored flow-VAE latents in the dataset are not consumed.
+
+## Timeline Alignment
+
+DAV hop: 512 samples at 44.1 kHz.
+
+Frame center:
+
+```text
+((latent_start + latent_end) / 2) * 512 / 44100 seconds
+```
+
+Stitched rollout rules:
+
+- rollout window: 200 semantic frames;
+- rollout hop: 100 semantic frames;
+- full stitched hop: 345 DAV latents;
+- later chunks begin ownership 25 semantic frames after nominal start;
+- code row 0 is warm-up;
+- semantic frame `i` uses code row `i + 1`;
+- the final partial chunk uses its own integer latent length;
+- records without exact `chunk_stitching` metadata are excluded.
+
+MERT features are linearly interpolated onto these exact DAV frame centers. A naive 3:1 reshape is not used.
+
+## MERT Alignment
+
+Teacher: [`m-a-p/MERT-v1-95M`](https://huggingface.co/m-a-p/MERT-v1-95M).
+
+Pinned revision: `12af15fef9d0ac838c3f475bfbbf26d2060dd4f5`.
+
+| Setting | Value |
+|---|---:|
+| Teacher layer | 9 |
+| Student capture layer | 4, zero-based |
+| Teacher hidden size | 768 |
+| Teacher sample rate | 24 kHz |
+| Teacher feature rate | 75 Hz |
+| Chunk length | 5 seconds |
+| Chunk overlap | 1 second |
+| Cache dtype | bfloat16 |
+| Initial alignment weight | 0.5 |
+| Constant phase | 0% through 70% of training |
+| Linear decay | 70% through 90% |
+| Disabled weight | final 10% |
+
+MERT sidecars are generated before training. Cache metadata records the model, revision, hidden layers, chunk geometry, dtype, emitted frame count, and alignment version `dav512-mert75-center-v1`.
+
+The projection forward remains active after its scheduled weight reaches zero. This preserves DDP parameter participation. The projection is not exported.
+
+MERT-v1-95M is published under CC-BY-NC-4.0. This repository does not redistribute MERT weights. Users remain responsible for applicable model, dataset, and teacher terms.
+
+## Loss
+
+```text
+reported_loss = mean(CE_head_0 ... CE_head_7)
+              + 0.25 * mean(KL_head_0 ... KL_head_7)
+
+optimization_loss = reported_loss
+                  + scheduled_MERT_weight * cosine_alignment_loss
+```
+
+Hard targets:
+
+- cross-entropy against sampled RVQ codes;
+- equal weight for all eight heads;
+- padding target `-100`.
+
+Soft targets:
+
+- teacher top-k 50;
+- temperature 1.0;
+- Hinton `T^2` scaling;
+- teacher renormalized over valid stored IDs;
+- student full-vocabulary log-softmax gathered at teacher IDs;
+- no student top-k renormalization;
+- invalid, EOS, and out-of-vocabulary IDs excluded;
+- frames with no valid teacher IDs skipped for KL.
+
+MERT target:
+
+- cosine distance between projected student layer 4 and frozen MERT layer 9;
+- mean over batch and frames.
+
+Reported loss excludes MERT. V1, v2, and v3 loss curves therefore remain directly comparable.
+
+## Training
+
+| Setting | Value |
+|---|---:|
+| Hardware | 4 x NVIDIA L40S |
+| Distribution | PyTorch DDP through Accelerate |
+| Precision | bfloat16 mixed precision |
+| Epochs | 20 |
+| Optimizer steps | 17,660 |
+| Batch per rank | 16 |
+| Global batch | 64 |
+| Gradient accumulation | 1 |
+| Optimizer | `mup.MuAdamW` |
+| Learning rate | 3e-4 |
+| Weight decay | 0.01 |
+| LR schedule | polynomial, power 1.0 |
+| Linear warmup | 500 steps |
+| Final learning rate | 1e-7 |
+| Gradient norm limit | 1.0 |
+| Train crop | random 128-frame window |
+| Validation crop | deterministic 128-frame windows |
+| Validation interval | 500 steps |
+| Checkpoint interval | 500 steps |
+
+The learning-rate multiplier warms linearly for 500 steps, then decays linearly to the final learning rate. It does not restart or reheat.
+
+Training metrics: [Weights & Biases](https://wandb.ai/bghira/simpletuner-rvq-encoder/runs/tap14a1y).
+
+## Checkpoint Format
+
+Each exported checkpoint contains:
+
+| File | Contents |
+|---|---|
+| `rvq_encoder.safetensors` | exported encoder state dictionary |
+| `rvq_encoder_config.json` | architecture and muP configuration |
+| `mup_base_shapes.bsh` | exported-encoder muP base shapes |
+
+The training-only MERT projection and MERT teacher are not included.
+
+Loading currently requires `RVQEncoderConfig` and `MiniMaxMusicRVQEncoder` from `scripts/train_minimax_music_rvq_encoder.py`.
+
+## Evaluation
+
+Protocol:
+
+- exact-alignment holdout;
+- 130 tracks;
+- 2,768 deterministic windows;
+- all 35 numbered checkpoints and `final`;
+- four-rank evaluation; no distributed-sampler padding;
+- exact-token top-1 and top-5.
+
+Recommended `final` result:
+
+| Metric | Value |
+|---|---:|
+| loss | 5.259917 |
+| hard CE | 4.629320 |
+| teacher KL before 0.25 weighting | 2.522392 |
+| semantic top-1 | 43.03% |
+| semantic top-5 | 80.49% |
+| acoustic top-1 | 7.66% |
+| acoustic top-5 | 22.03% |
+
+Top-k accuracy measures exact token inclusion. It does not measure perceptual code equivalence.
+
+### Matched Comparison at Step 17,500
+
+| Metric | v1, 41M | v2, 155M | v3, 155M + MERT | v3 vs v2 |
+|---|---:|---:|---:|---:|
+| loss | 5.337856 | 5.264569 | **5.260236** | -0.004334 |
+| semantic top-1 | 41.03% | 42.86% | **43.03%** | +0.17 pp |
+| semantic top-5 | 78.38% | 80.17% | **80.48%** | +0.31 pp |
+| acoustic top-1 | 7.17% | 7.62% | **7.65%** | +0.03 pp |
+| acoustic top-5 | 20.94% | 21.98% | **22.02%** | +0.04 pp |
+
+![V1, v2, and v3 comparison](evaluation/v3/v1-v2-v3-comparison.png)
+
+Interpretation:
+
+- v2's width increase produced the main gain over v1;
+- v3 learned the MERT alignment target;
+- v3 wins every listed aggregate metric over v2;
+- the v3-v2 differences are small;
+- MERT alignment did not remove the acoustic codebook-depth gradient;
+- causal conditioning across acoustic books is the next structural test.
+
+Machine-readable comparison: [`comparison-metrics.json`](evaluation/v3/comparison-metrics.json).
+
+## Limitations
+
+- 5.12-second context.
+- No cross-window state.
+- Synthetic model-output training domain.
+- Real audio remains out of distribution.
+- Teacher top-k uncertainty is from LM rollout, not an audio-conditioned posterior.
+- Exact-token accuracy understates perceptual equivalence.
+- Semantic CE can dominate early because its vocabulary is larger.
+- Acoustic heads are independent despite residual-codebook dependence.
+- End-to-end condition-embedding replay evaluation is pending.
+- No stable packaged loading API.
+- Use is subject to MiniMax Music 3, dataset, and MERT terms.
+
+## Discussion and Attribution
+
+Primary discussion: [MiniMaxAI/MiniMax-Music3 discussion #10](https://huggingface.co/MiniMaxAI/MiniMax-Music3/discussions/10).
+
+Attribution covers public discussion, measurements, datasets, and independent experiments. It does not imply shared authorship.
+
+- [`bghira`](https://huggingface.co/bghira): SimpleTuner experiments, trace extraction, teacher distributions, alignment records, corpus publication, and training runs.
+- [`marduk191`](https://huggingface.co/marduk191): WAV/code samples, early mel encoder proof, corpus-scale observations, and encoder experiments.
+- [`scragnog`](https://huggingface.co/scragnog): SimpleTuner calibration, relative-weight analysis, caption-cache and rollout-seam findings, and GGML interoperability tests.
+- [`Serveurperso`](https://huggingface.co/Serveurperso): independent encoder, corpus generator, replay evaluation, and stitched-timeline findings.
+- [`dernet`](https://huggingface.co/dernet): RVQ supervision, internal-alignment, and tokenizer reverse-engineering analysis.
+
+Additional artifacts:
+
+- [`SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1`](https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1)
+- [`SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2`](https://huggingface.co/SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2)
+- [`marduk191/Minmax_music3_experiments`](https://huggingface.co/marduk191/Minmax_music3_experiments/tree/main/corpus)
+- [`ServeurpersoCom/minimaxmusic.cpp` proof commit](https://github.com/ServeurpersoCom/minimaxmusic.cpp/commit/d19efe9f94e41ac4c900aa30d56fe90c8dac7ef1)
+
+<!-- simpletuner-rvq-evaluation-start -->
+## Offline Checkpoint Evaluation
+
+Exact-alignment holdout: 130 tracks, 2,768 windows.
+
+| Selection | Checkpoint | Step | Loss | Semantic top-1 | Semantic top-5 | Acoustic top-1 | Acoustic top-5 |
+|---|---|---:|---:|---:|---:|---:|---:|
+| best semantic top-1 | `checkpoint-17500` | 17,500 | 5.260236 | 0.4303 | 0.8048 | 0.0765 | 0.2202 |
+| lowest loss; best semantic top-5; best acoustic top-1; best acoustic top-5; final | `final` | 17,660 | 5.259917 | 0.4303 | 0.8049 | 0.0766 | 0.2203 |
+
+Top-k accuracy measures exact token inclusion. It does not measure perceptual code equivalence.
+
+### Checkpoint Loss
+
+![Checkpoint Loss](evaluation/v3/checkpoint-loss.png)
+
+### Checkpoint Accuracy
+
+![Checkpoint Accuracy](evaluation/v3/checkpoint-accuracy.png)
+
+### Codebook Top1
+
+![Codebook Top1](evaluation/v3/codebook-top1.png)
+
+### Training History
+
+![Training History](evaluation/v3/training-history.png)
+
+### MERT Alignment
+
+![MERT Alignment](evaluation/v3/mert-alignment.png)
+
+Full data: [`checkpoint-metrics.csv`](evaluation/v3/checkpoint-metrics.csv), [`evaluation-metrics.json`](evaluation/v3/evaluation-metrics.json), [`comparison-metrics.json`](evaluation/v3/comparison-metrics.json).
+<!-- simpletuner-rvq-evaluation-end -->

evaluation/v3/checkpoint-metrics.csv

@@ -0,0 +1,37 @@
+checkpoint,step,loss,ce_loss,teacher_kl_loss,semantic_top1,semantic_top5,acoustic_top1,acoustic_top5,head_0_top1,head_0_top5,head_1_top1,head_1_top5,head_2_top1,head_2_top5,head_3_top1,head_3_top5,head_4_top1,head_4_top5,head_5_top1,head_5_top5,head_6_top1,head_6_top5,head_7_top1,head_7_top5
+checkpoint-500,500,7.7227089621803975,6.719849326393821,4.011442704634233,0.015586159446022728,0.053095037286931816,0.009238899528206168,0.036695604200487016,0.015586159446022728,0.053095037286931816,0.01840764825994318,0.0687255859375,0.012598211115056818,0.046875,0.008930553089488636,0.03623546253551136,0.007501775568181818,0.03129161487926136,0.006394819779829545,0.027019153941761364,0.005809437144886364,0.024716463955965908,0.005029851740056818,0.022005948153409092
+checkpoint-1000,1000,6.984003240411932,6.109800858931108,3.49680571122603,0.09515935724431818,0.2500221946022727,0.021492301643668832,0.07571648932122566,0.09515935724431818,0.2500221946022727,0.042791193181818184,0.13984264026988635,0.035680597478693184,0.11644952947443182,0.021492697975852272,0.077972412109375,0.016404585404829544,0.06255548650568182,0.014210094105113636,0.05376364968039773,0.010320490056818182,0.04146229137073864,0.009546453302556818,0.037969415838068184
+checkpoint-1500,1500,6.443159623579546,5.658165324818004,3.13998066295277,0.187164306640625,0.4373418634588068,0.03389353566355519,0.11162458147321429,0.187164306640625,0.4373418634588068,0.06279407848011363,0.19470769708806818,0.056931929154829544,0.17598932439630682,0.037611527876420456,0.12383755770596591,0.029255260120738636,0.10182883522727272,0.021534312855113636,0.076873779296875,0.01553622159090909,0.05742575905539773,0.013591419566761364,0.05070911754261364
+checkpoint-2000,2000,6.119406266645952,5.383781433105469,2.942498640580611,0.2500277432528409,0.5477683327414773,0.04309121664468344,0.13684399096996752,0.2500277432528409,0.5477683327414773,0.07553932883522728,0.22703690962357956,0.07314786044034091,0.21626420454545456,0.051153009588068184,0.16249223188920456,0.038726806640625,0.12935846502130682,0.027579567649147728,0.09393865411931818,0.01888483220880682,0.06880881569602272,0.01660711115056818,0.06000865589488636
+checkpoint-2500,2500,5.922209306196733,5.215132973410866,2.8283004760742188,0.288177490234375,0.6095858487215909,0.04907464361810065,0.15349350966416397,0.288177490234375,0.6095858487215909,0.08429787375710228,0.24935080788352273,0.082489013671875,0.2406643954190341,0.05978116122159091,0.18552190607244318,0.04595669833096591,0.1512784090909091,0.03096979314630682,0.10400390625,0.021598122336647728,0.07710682262073863,0.018429842862215908,0.0665283203125
+checkpoint-3000,3000,5.79420956698331,5.10393662886186,2.7610919258811255,0.3153631036931818,0.6513006036931818,0.05358173320819805,0.16487498097605519,0.3153631036931818,0.6513006036931818,0.08938043767755682,0.2620322487571023,0.08984929865056818,0.2574213201349432,0.06611494584517046,0.20173228870738635,0.052515203302556816,0.16750266335227273,0.03349997780539773,0.11228526722301137,0.023739901455965908,0.082611083984375,0.019972367720170456,0.07053999467329546
+checkpoint-3500,3500,5.7021942138671875,5.024650226939809,2.710172479802912,0.3348305442116477,0.6776178533380682,0.05691250887784091,0.1734722186992695,0.3348305442116477,0.6776178533380682,0.09334494850852272,0.2713595303622159,0.09541736949573863,0.2713567560369318,0.07116421786221591,0.21482987837357956,0.05693470348011364,0.17842240767045456,0.03589699485085227,0.11775623668323863,0.024638782848011364,0.08634255149147728,0.02099054509943182,0.07423817027698863
+checkpoint-4000,4000,5.628925323486328,4.959530917080966,2.6775762384588067,0.3470098322088068,0.6964499733664773,0.059757777622767856,0.1807615602171266,0.3470098322088068,0.6964499733664773,0.09620527787642046,0.2793801047585227,0.10038341175426137,0.2814858176491477,0.07474587180397728,0.22456221147017044,0.061082319779829544,0.18967784534801135,0.03770862926136364,0.12312733043323863,0.026200727982954544,0.09014615145596591,0.02197820490056818,0.07695146040482954
+checkpoint-4500,4500,5.572572534734553,4.910030364990234,2.6501664248379795,0.36024336381392047,0.7150379527698864,0.06188251445819805,0.18595668247767858,0.36024336381392047,0.7150379527698864,0.09985628995028409,0.28624378551136365,0.10342129794034091,0.2893094149502841,0.07832197709517046,0.23265491832386365,0.06331842595880682,0.195159912109375,0.03867686878551136,0.1260986328125,0.02681940252130682,0.09317571466619318,0.022763338955965908,0.07905439897017046
+checkpoint-5000,5000,5.529521595348012,4.872190995649858,2.6293215318159624,0.3691323020241477,0.7278886274857954,0.06362518706879058,0.19021566812094157,0.3691323020241477,0.7278886274857954,0.10185935280539772,0.2919339266690341,0.10581554066051137,0.29484696821732953,0.08092429421164772,0.23899147727272727,0.06595680930397728,0.20199030095880682,0.03998912464488636,0.12915871360085227,0.027371493252840908,0.09384432705965909,0.023459694602272728,0.08074396306818182
+checkpoint-5500,5500,5.494660810990767,4.841509732333097,2.612605008212003,0.3771445534446023,0.7390358664772727,0.0650262213372565,0.19364949015827923,0.3771445534446023,0.7390358664772727,0.10403997247869318,0.2962119362571023,0.10716108842329546,0.29948564009232953,0.08321311257102272,0.24372447620738635,0.06810136274857954,0.2066983309659091,0.040674382990056816,0.13139204545454544,0.028012362393465908,0.09567538174715909,0.02398126775568182,0.08235862038352272
+checkpoint-6000,6000,5.464677290482954,4.81493481722745,2.5989679856733843,0.38371415571732953,0.7480829412286932,0.06640506100344967,0.1969537115716315,0.38371415571732953,0.7480829412286932,0.10564630681818182,0.30029296875,0.10914750532670454,0.30354725230823865,0.08547696200284091,0.24945900656960227,0.06962446732954546,0.2105435458096591,0.04170643199573864,0.1339749422940341,0.029069380326704544,0.09763128107244318,0.02416437322443182,0.08322698419744318
+checkpoint-6500,6500,5.438798731023615,4.791322187943892,2.589907559481534,0.3905029296875,0.7559537020596591,0.06748704786424513,0.19937847186992694,0.3905029296875,0.7559537020596591,0.10705011541193182,0.30353892933238635,0.111846923828125,0.3086631081321023,0.08660888671875,0.2523748224431818,0.07125299627130682,0.21385054154829544,0.041883988813920456,0.1345547762784091,0.029185901988636364,0.09835260564630682,0.024580522017045456,0.08431451970880682
+checkpoint-7000,7000,5.417517228560015,4.770946849476207,2.5862830768931997,0.3950112082741477,0.761627197265625,0.06844181209415584,0.2019443264255276,0.3950112082741477,0.761627197265625,0.10875632546164772,0.30757002397017047,0.11321466619318182,0.3115511807528409,0.08842329545454546,0.25660566850142047,0.07161920720880682,0.21552068536931818,0.04279674183238636,0.13665771484375,0.029560435901988636,0.10024192116477272,0.024722012606534092,0.08546309037642046
+checkpoint-7500,7500,5.398602572354403,4.75521503795277,2.573550484397195,0.39908946644176135,0.7672479802911932,0.0693311815137987,0.20379400872564934,0.39908946644176135,0.7672479802911932,0.10926680131392046,0.30851606889204547,0.11490423029119318,0.3149025656960227,0.08939153497869318,0.25968794389204547,0.07276777787642046,0.21742664683948865,0.043543035333806816,0.1387356844815341,0.030270663174715908,0.10109086470170454,0.025174227627840908,0.08619828657670454
+checkpoint-8000,8000,5.380239313299006,4.73772569136186,2.5700543143532495,0.40406383167613635,0.7727494673295454,0.07033429827008929,0.20586088106229708,0.40406383167613635,0.7727494673295454,0.11074274236505682,0.3129327947443182,0.11608054421164772,0.31727461381392047,0.09094515713778409,0.2620821866122159,0.07451005415482954,0.22078635475852273,0.04418113014914773,0.13911299272017044,0.030376087535511364,0.10216730291193182,0.025504372336647728,0.086669921875
+checkpoint-8500,8500,5.372574199329723,4.731698816472834,2.5635046525435015,0.4052789861505682,0.7765641645951704,0.07049798346185066,0.20650175020292208,0.4052789861505682,0.7765641645951704,0.11169711026278409,0.31425614790482953,0.11592240767045454,0.318634033203125,0.09131414240056818,0.2623069069602273,0.07414384321732954,0.22156316583806818,0.044203324751420456,0.1398953524502841,0.030550870028409092,0.10174283114346591,0.025654185901988636,0.08711381392045454
+checkpoint-9000,9000,5.357324773615057,4.716484416614879,2.5633629885586826,0.409759521484375,0.7805591930042614,0.07141985212053571,0.2084417962408685,0.409759521484375,0.7805591930042614,0.11305098100142046,0.31657825816761365,0.11829168146306818,0.3209783380681818,0.09186900745738637,0.26542247425426135,0.07552268288352272,0.22413773970170456,0.04474709250710227,0.14101063121448865,0.030917080965909092,0.10291082208806818,0.025540438565340908,0.08805431019176137
+checkpoint-9500,9500,5.344036449085582,4.704330097545277,2.558825059370561,0.4119234952059659,0.7830005992542614,0.07206151392552759,0.2099094143161526,0.4119234952059659,0.7830005992542614,0.11342274058948863,0.3182123357599432,0.11939863725142046,0.3241632634943182,0.09335604580965909,0.2679387872869318,0.076202392578125,0.22467041015625,0.04496071555397727,0.1418179598721591,0.031008633700284092,0.10361272638494318,0.026081431995738636,0.08895041725852272
+checkpoint-10000,10000,5.334482366388494,4.697110262784091,2.5494896281849253,0.4138544256036932,0.7849870161576704,0.07225968001724838,0.21085744089894481,0.4138544256036932,0.7849870161576704,0.11370294744318182,0.319427490234375,0.11955677379261363,0.3252341530539773,0.09388594193892046,0.26970048384232953,0.07647705078125,0.22669844193892044,0.04510220614346591,0.14306085759943182,0.030964244495738636,0.10337968306107954,0.02612859552556818,0.0885009765625
+checkpoint-10500,10500,5.326873779296875,4.6891538446599785,2.550879044966264,0.4155856045809659,0.7870844060724432,0.0728006734476461,0.21215186181006493,0.4155856045809659,0.7870844060724432,0.11428555575284091,0.3206703879616477,0.12022816051136363,0.32636885209517047,0.09464333274147728,0.2720725319602273,0.07749522816051137,0.2285128506747159,0.045496160333806816,0.14343816583806818,0.03131380948153409,0.10450883345170454,0.026142467151988636,0.08949141068892046
+checkpoint-11000,11000,5.317876295609907,4.681384693492543,2.5459686626087534,0.41792436079545453,0.790924072265625,0.07338090376420454,0.2130745231331169,0.41792436079545453,0.790924072265625,0.11578369140625,0.3231728293678977,0.12080799449573863,0.3278447931463068,0.09537298029119318,0.27268843217329547,0.07770052823153409,0.22972245649857956,0.04572365500710227,0.14333551580255682,0.031693892045454544,0.10498324307528409,0.026583584872159092,0.08977439186789772
+checkpoint-11500,11500,5.309654582630504,4.673743161288175,2.5436446449973364,0.41938365589488635,0.7926885431463068,0.0737257127637987,0.21399163580560066,0.41938365589488635,0.7926885431463068,0.11606667258522728,0.3235112970525568,0.12167635830965909,0.33009199662642047,0.09567538174715909,0.27430863813920453,0.07805286754261363,0.22990556196732956,0.046067671342329544,0.14473932439630682,0.031632856889204544,0.10498046875,0.026908180930397728,0.09040416370738637
+checkpoint-12000,12000,5.3017966530539775,4.666432467373935,2.5414588234641333,0.4220636541193182,0.7952104048295454,0.07417713512073863,0.21510453657670456,0.4220636541193182,0.7952104048295454,0.11665482954545454,0.32521195845170453,0.12220625443892046,0.3312155983664773,0.09684614701704546,0.27543501420454547,0.07895729758522728,0.23249123313210227,0.04594837535511364,0.14491410688920456,0.031538529829545456,0.10548262162642046,0.027088512073863636,0.09098122336647728
+checkpoint-12500,12500,5.29755332253196,4.66277451948686,2.5391148653897373,0.4229403409090909,0.7964366566051136,0.07441453809862013,0.2154005967177354,0.4229403409090909,0.7964366566051136,0.116546630859375,0.3252286044034091,0.12289151278409091,0.3319646661931818,0.09722622958096591,0.27640880237926135,0.07928189364346591,0.23256891424005682,0.04620083895596591,0.14500565962357956,0.03171053799715909,0.10569624467329546,0.02704412286931818,0.09093128551136363
+checkpoint-13000,13000,5.29179590398615,4.658017591996626,2.535112901167436,0.4236422452059659,0.7974798029119318,0.0748176079291802,0.2161968280742695,0.4236422452059659,0.7974798029119318,0.11768132990056818,0.32719005237926135,0.12344637784090909,0.3330327814275568,0.09734275124289772,0.27726884321732953,0.07952048561789772,0.232513427734375,0.046539306640625,0.14640669389204544,0.032049005681818184,0.10556585138494318,0.027143998579545456,0.091400146484375
+checkpoint-13500,13500,5.286717501553622,4.653559598055753,2.532631267200817,0.424591064453125,0.7983537153764204,0.0749464158887987,0.21663120814732142,0.424591064453125,0.7983537153764204,0.11791159889914772,0.3276727849786932,0.12347966974431818,0.3338650790127841,0.09767844460227272,0.27809559215198865,0.07981456409801137,0.2337979403409091,0.04675015536221591,0.14597112482244318,0.03198519620028409,0.10572398792613637,0.027005282315340908,0.09129194779829546
+checkpoint-14000,14000,5.28161655772816,4.648884166370738,2.530928698453036,0.4258062189275568,0.7999184348366477,0.07525634765625,0.21730695452008927,0.4258062189275568,0.7999184348366477,0.11814464222301137,0.3286077325994318,0.12398459694602272,0.33491099964488635,0.09790316495028409,0.27874755859375,0.08057195490056818,0.23488270152698865,0.046675248579545456,0.14664528586647727,0.03222378817471591,0.105987548828125,0.027291037819602272,0.09136685458096591
+checkpoint-14500,14500,5.2772085016424,4.64499907060103,2.528839284723455,0.42650257457386365,0.8008450594815341,0.07545213575487013,0.2177809678114854,0.42650257457386365,0.8008450594815341,0.11890758167613637,0.3288629705255682,0.12393465909090909,0.33546309037642047,0.098663330078125,0.28053699840198865,0.08038052645596591,0.2348355379971591,0.046758478338068184,0.14672296697443182,0.03206287730823864,0.10642866654829546,0.027457497336647728,0.09161654385653409
+checkpoint-15000,15000,5.273670890114524,4.641587690873579,2.5283348777077417,0.4278314763849432,0.8019270463423296,0.07563563755580358,0.21843293425324675,0.4278314763849432,0.8019270463423296,0.11882990056818182,0.3296425559303977,0.12473366477272728,0.3363869406960227,0.09859397194602272,0.2816689231178977,0.08034446022727272,0.23612559925426135,0.047185724431818184,0.1467618075284091,0.03230701793323864,0.10666725852272728,0.027454723011363636,0.09177745472301137
+checkpoint-15500,15500,5.270176280628551,4.638293526389382,2.527528936212713,0.428680419921875,0.8027316006747159,0.07592139306006493,0.21882213245738635,0.428680419921875,0.8027316006747159,0.11943470348011363,0.330535888671875,0.12488625266335228,0.33720536665482953,0.09928200461647728,0.2818270596590909,0.08101307262073863,0.23612559925426135,0.04690274325284091,0.14707253196022727,0.032395796342329544,0.10703901811079546,0.027535178444602272,0.091949462890625
+checkpoint-16000,16000,5.268038316206499,4.636166659268466,2.5274857607754795,0.4286471280184659,0.8030811656605114,0.07601889077719155,0.21922837294541397,0.4286471280184659,0.8030811656605114,0.11922662908380682,0.3308160955255682,0.12523304332386365,0.33769364790482953,0.09915993430397728,0.2824346368963068,0.08130715110085228,0.2362643155184659,0.047124689275568184,0.14772449840198865,0.03244295987215909,0.10739690607244318,0.027637828480113636,0.09226851029829546
+checkpoint-16500,16500,5.265233820134943,4.634354331276634,2.523518302223899,0.4292879971590909,0.8040050159801136,0.07617702731838474,0.21954226803469967,0.4292879971590909,0.8040050159801136,0.11981201171875,0.3312738591974432,0.12568803267045456,0.33790449662642047,0.099365234375,0.28257612748579547,0.08132102272727272,0.2370328036221591,0.04709972034801136,0.14814619584517044,0.032282049005681816,0.10744961825284091,0.027671120383522728,0.09241277521306818
+checkpoint-17000,17000,5.2625732421875,4.631568561900746,2.524019414728338,0.4292824485085227,0.8041298606178977,0.07635735846185066,0.21998576374797077,0.4292824485085227,0.8041298606178977,0.12002840909090909,0.3319785378196023,0.12601262872869318,0.33867298473011365,0.09950395063920454,0.28306718306107953,0.08132102272727272,0.23765425248579544,0.0472412109375,0.14818503639914773,0.032745361328125,0.10767433860085228,0.02764892578125,0.09266801313920454
+checkpoint-17500,17500,5.260235873135653,4.629444122314453,2.523169430819425,0.4303006258877841,0.8048234419389204,0.07652936662946429,0.22019304547991073,0.4303006258877841,0.8048234419389204,0.12037519975142046,0.3322837136008523,0.12618463689630682,0.3389920321377841,0.09975918856534091,0.28349720348011365,0.08173439719460228,0.23786510120738635,0.04738270152698864,0.14839311079545456,0.032556707208806816,0.10774924538352272,0.027712735262784092,0.09257091175426137
+final,17660,5.259916825727983,4.629319624467329,2.522391752763228,0.43027843128551135,0.8049205433238636,0.07655988420758929,0.22027746423498376,0.43027843128551135,0.8049205433238636,0.12036410245028409,0.33223100142045453,0.12618186257102273,0.33906139026988635,0.09992842240767046,0.28361372514204547,0.08188143643465909,0.23798439719460227,0.04734386097301136,0.14848466352982956,0.03249289772727273,0.107879638671875,0.027726606889204544,0.09268743341619318

evaluation/v3/comparison-metrics.json

@@ -0,0 +1,52 @@
+{
+  "format": "simpletuner-minimaxmusic-rvq-three-model-comparison-v1",
+  "models": [
+    {
+      "acoustic_top1": 0.07169886997767858,
+      "acoustic_top5": 0.20938229251217533,
+      "ce_loss": 4.696246407248757,
+      "checkpoint": "checkpoint-17500",
+      "label": "v1, 41M",
+      "loss": 5.337855945933949,
+      "model_id": "SimpleTuner/open-rvq-encoder-minimax-music3-41m-v1",
+      "semantic_top1": 0.41028941761363635,
+      "semantic_top5": 0.7838023792613636,
+      "step": 17500,
+      "teacher_kl_loss": 2.5664437033913354
+    },
+    {
+      "acoustic_top1": 0.0762291301094137,
+      "acoustic_top5": 0.21976204453963666,
+      "ce_loss": 4.6332796129877165,
+      "checkpoint": "checkpoint-17500",
+      "label": "v2, 155M",
+      "loss": 5.264569409320809,
+      "model_id": "SimpleTuner/open-rvq-encoder-minimax-music3-155m-v2",
+      "semantic_top1": 0.42864884393063585,
+      "semantic_top5": 0.8017380554552023,
+      "step": 17500,
+      "teacher_kl_loss": 2.525154951679913
+    },
+    {
+      "acoustic_top1": 0.07652936662946429,
+      "acoustic_top5": 0.22019304547991073,
+      "ce_loss": 4.629444122314453,
+      "checkpoint": "checkpoint-17500",
+      "label": "v3, 155M + MERT",
+      "loss": 5.260235873135653,
+      "model_id": "SimpleTuner/open-rvq-encoder-minimax-music3-155m-v3",
+      "semantic_top1": 0.4303006258877841,
+      "semantic_top5": 0.8048234419389204,
+      "step": 17500,
+      "teacher_kl_loss": 2.523169430819425
+    }
+  ],
+  "protocol": {
+    "dataset": "bghira/minimax-music3-rvq-reverse-distillation",
+    "matched_step": 17500,
+    "records": 130,
+    "require_exact_alignment": true,
+    "split": "holdout",
+    "windows": 2768
+  }
+}