Add Dataset 8 MergeKit checkpoint and validation scores

README.md

@@ -8,7 +8,7 @@ tags:
 
 # MML26 Singing Synthesis Checkpoints
 
-Six controlled Basic Pitch checkpoints trained with the same 30-epoch configuration.
+Eight Basic Pitch checkpoints, including a MergeKit model soup selected on the Klangio validation set.
 
 ## Best-checkpoint test metrics
 
@@ -21,14 +21,16 @@ Six controlled Basic Pitch checkpoints trained with the same 30-epoch configurat
 | Dataset 5 — `syntheticdataset_aceopencpop_soulx` | 0.2424 | 0.4400 | 0.5933 | 27 | [`checkpoints/syntheticdataset-aceopencpop-soulx-best-epoch27-COnPOff_f1-0.2424.ckpt`](./checkpoints/syntheticdataset-aceopencpop-soulx-best-epoch27-COnPOff_f1-0.2424.ckpt) | [smau9uor](https://wandb.ai/krishnakalyan/MMLHackathon/runs/smau9uor) | 1582 s |
 | Dataset 6 — Synthetic + ACE + SoulX | 0.2388 | 0.4420 | 0.6236 | 25 | [`checkpoints/syntheticdataset-all-three-best-epoch25-COnPOff_f1-0.2388.ckpt`](./checkpoints/syntheticdataset-all-three-best-epoch25-COnPOff_f1-0.2388.ckpt) | [9r5p4hoa](https://wandb.ai/krishnakalyan/MMLHackathon/runs/9r5p4hoa) | 2228 s |
 | Dataset 7 — audio augmentations (`room_reverb`, `phone_codec`, `noisy_codec`) | 0.3028 | 0.4246 | 0.6644 | 75 | [`checkpoints/best-epoch75-step003116.ckpt`](./checkpoints/best-epoch75-step003116.ckpt) | [yhns6pet](https://wandb.ai/krishnakalyan/MMLHackathon/runs/yhns6pet) | — |
+| Dataset 8 — MergeKit (99% Dataset 7 + 1% Dataset 2) | 0.2409 | 0.4199 | 0.6615 | MergeKit | [`checkpoints/mergekit-dataset8-aug75-soulx-99-01-COnPOff_f1-0.2409.ckpt`](./checkpoints/mergekit-dataset8-aug75-soulx-99-01-COnPOff_f1-0.2409.ckpt) | Not created (MergeKit) | — |
 
 ### Evaluation context
 
-- Datasets 1, 2, 4, 5, and 6 were evaluated on the same 35-song Klangio validation/test set.
+- Datasets 1, 2, 4, 5, 6, 7, and 8 were evaluated on the same 35-song Klangio validation set.
 - Dataset 3 used 28 teacher-pseudo-labeled Klangio songs for training and a fixed 7-song clean holdout for validation/test. Its metric is therefore not directly comparable to the 35-song metrics without accounting for the smaller evaluation set.
 - All six used 30 epochs, 8-second chunks, batch size 32, Adam at 1e-4, onset/frame weights 18/9, BF16, and `torch.compile(reduce-overhead)`.
 - Dataset 4 contains 25.52 hours, 330 tracks, 192,086 pitched notes, and 30 ACE-OpenCpop synthetic voices. The derived training data retains the source dataset's CC BY-NC 4.0 terms.
 - Dataset 5 is the 51.07-hour ACE-OpenCpop + SoulX union; Dataset 6 is the union of procedural, SoulX, and ACE-OpenCpop synthetic singing (1,130 tracks).
+- Dataset 8 is a normalized linear merge made with MergeKit 0.1.4 from 99% Dataset 7 and 1% Dataset 2. Its decoding thresholds were selected on Klangio (`onset=0.70`, `frame=0.01`). These are validation scores, not held-out test scores. The reproducible merge config is [`mergekit/dataset8-merge.yml`](./mergekit/dataset8-merge.yml).
 - `metadata.json` records SHA-256 hashes, exact checkpoint epochs, steps, and scores.
 
 An earlier, separate SoulX experiment is available at [W&B run `ltqkp16d`](https://wandb.ai/krishnakalyan/MMLHackathon/runs/ltqkp16d), but it produced a different checkpoint and is not the tracking run for Dataset 2 above.

checkpoints/mergekit-dataset8-aug75-soulx-99-01-COnPOff_f1-0.2409.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49afc7df568860c929f138afb070a5c27ea8f7df356c1154539fb15ea23a6de2
+size 162968

mergekit/dataset8-merge.yml

@@ -0,0 +1,8 @@
+merge_method: linear
+dtype: float32
+parameters: {normalize: true}
+models:
+  - model: best-epoch75-step003116.pt
+    parameters: {weight: 0.99}
+  - model: syntheticdataset-soulx-best-epoch22-COnPOff_f1-0.2498.pt
+    parameters: {weight: 0.01}

mergekit/dataset8-validation.json

@@ -0,0 +1,15 @@
+{
+  "dataset": "klangiodataset",
+  "songs": 35,
+  "precision": "bf16-mixed",
+  "selection_metric": "COnPOff_f1",
+  "onset_threshold": 0.7,
+  "frame_threshold": 0.01,
+  "metrics": {
+    "COnPOff_f1": 0.24085187911987305,
+    "COnP_f1": 0.419907808303833,
+    "COn_f1": 0.6614999771118164
+  },
+  "wandb_run": null,
+  "note": "Klangio is a validation set; these are validation scores, not held-out test scores."
+}

metadata.json

@@ -68,6 +68,43 @@
       "runtime_seconds": 2228,
       "sha256": "40534c0502419ff19bdc27e617ddbbf7cbdf78333b8922af486cff1a654e92e5",
       "bytes": 308632
+    },
+    {
+      "name": "audio_augmentations",
+      "dataset_number": 7,
+      "remote": "checkpoints/best-epoch75-step003116.ckpt",
+      "epoch": 75,
+      "global_step": 3116,
+      "COnPOff_f1": 0.3028,
+      "COnP_f1": 0.4246,
+      "COn_f1": 0.6644,
+      "wandb_run": "https://wandb.ai/krishnakalyan/MMLHackathon/runs/yhns6pet",
+      "sha256": "d2eb267922a1c4d90ca32b5d328b766e0f2f144888f18fbc6b521a0fc68080d3",
+      "bytes": 308632
+    },
+    {
+      "name": "mergekit_aug75_soulx_99_01",
+      "dataset_number": 8,
+      "remote": "checkpoints/mergekit-dataset8-aug75-soulx-99-01-COnPOff_f1-0.2409.ckpt",
+      "epoch": null,
+      "global_step": null,
+      "merge_method": "linear",
+      "mergekit_version": "0.1.4",
+      "parents": [
+        {"name": "audio_augmentations", "weight": 0.99},
+        {"name": "syntheticdataset_soulx", "weight": 0.01}
+      ],
+      "validation_dataset": "klangiodataset",
+      "validation_songs": 35,
+      "selection_metric": "COnPOff_f1",
+      "onset_threshold": 0.7,
+      "frame_threshold": 0.01,
+      "COnPOff_f1": 0.24085187911987305,
+      "COnP_f1": 0.419907808303833,
+      "COn_f1": 0.6614999771118164,
+      "wandb_run": null,
+      "sha256": "49afc7df568860c929f138afb070a5c27ea8f7df356c1154539fb15ea23a6de2",
+      "bytes": 162968
     }
   ]
 }