v0.3-preview: retrieval-tuned flagship

AudioCaps a2t R@10 0.743 -> 0.785 over v0.2; t2a R@10 0.782. MAEB(audio-only)
19-task mean 0.454; leads IEMOCAP speaker-gender (0.938). RAVDESS zero-shot
emotion regressed to 0.21. Non-audio (text/image/video) outputs remain
bit-for-bit the frozen base model's, verified on this artifact before push.
Ships a second keyword-tuned revision on branch v0.3-kw-preview.

README.md

@@ -21,22 +21,21 @@ base_model: Qwen/Qwen3-VL-Embedding-2B
 [![Python](https://img.shields.io/badge/python-3.11+-blue.svg)](https://github.com/Eximius-Labs/fusion-embedding)
 [![PyTorch](https://img.shields.io/badge/PyTorch-2.x-ee4c2c.svg)](https://github.com/Eximius-Labs/fusion-embedding)
 [![Weights](https://img.shields.io/badge/weights-CC--BY--NC--4.0-green.svg)](#license)
-[![Status](https://img.shields.io/badge/status-research%20preview%20v0.2-orange.svg)](#)
-[![Code](https://img.shields.io/badge/code-GitHub-black.svg)](https://github.com/Eximius-Labs/fusion-embedding)
+[![Status](https://img.shields.io/badge/status-research%20preview%20v0.3-orange.svg)](#)
 [![Paper](https://img.shields.io/badge/paper-arXiv%3A2607.18666-b31b1b.svg)](https://arxiv.org/abs/2607.18666)
+[![Code](https://img.shields.io/badge/code-GitHub-black.svg)](https://github.com/Eximius-Labs/fusion-embedding)
 
 </div>
 
 `fusion-embedding-2-2b-preview` is the second generation of Eximius Labs' unified
 multimodal embedding models: **text, images, video, and audio in one vector space**.
-It extends the first generation with modality-gated deep adapters — in-layer audio
-capacity added to a byte-frozen base. For the first-generation architecture, see
+Audio is added to a byte-frozen vision-language base through modality-gated deep
+adapters, so gaining audio never changes a single existing text, image, or video
+vector. For the first-generation architecture see
 [fusion-embedding-1-2b-preview](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview)
 (that line is final at v0.3).
 
-Sense packs extend the model to new sensors as separable, optional artifacts: [Ember](https://huggingface.co/EximiusLabs/fusion-embedding-2-ember), the thermal sense, is the first (thermal infrared in, all other outputs bit-for-bit unchanged).
-
-[GitHub](https://github.com/Eximius-Labs/fusion-embedding) | [fusion-embedding-1](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview) | Technical report: in preparation
+[GitHub](https://github.com/Eximius-Labs/fusion-embedding) | Technical report: [arXiv:2607.18666](https://arxiv.org/abs/2607.18666)
 
 ## Model Overview
 
@@ -44,172 +43,115 @@ Sense packs extend the model to new sensors as separable, optional artifacts: [E
 <img src="assets/fe2_model_overview.png" alt="fusion-embedding-2 architecture: frozen Qwen3-VL-Embedding base with modality-gated adapters inside; frozen audio tower and trained FusionResampler on the audio branch; one shared embedding space" width="820px">
 </p>
 
-`fusion-embedding-2-2b-preview` embeds all four modalities with a
+`fusion-embedding-2-2b-preview` embeds all four modalities on a
 [Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) base that is
-**byte-identical to its original release** — its text, image, and video behaviour (and
-benchmark scores) carry over exactly. Audio is added by training 60.6M parameters
-(~2.3% of the stack): a perceiver-resampler that translates frozen
+**byte-identical to its original release**: its text, image, and video behaviour, and
+its benchmark scores, carry over exactly. Audio is added by training 60.6M parameters
+(about 2.3% of the stack): a perceiver-resampler that maps frozen
 [Qwen2.5-Omni](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) audio-tower features into
-the base's input space, and — new in this generation — **28 gated adapters** (44.2M)
-that give the frozen language model in-layer capacity to process audio. The adapters
-are active only while encoding audio; every other forward pass returns the frozen
-layers' output untouched, so the invariance is bitwise, not approximate
-(`base_drift == 0` is asserted on every training run, and this model reproduces the
-base's text→image retrieval scores to four decimal places). Trained on 518K
-audio–caption pairs with a full-corpus frozen-text negative bank, it leads every
-unified embedding model we measured on audio↔text retrieval — ahead of ImageBind,
-LanguageBind, and Gemini Embedding 2 in both directions — and improves on
-fusion-embedding-1 v0.3 in 8 of 12 release-protocol cells, including every
-recorded text→audio direction. Audio↔image alignment is emergent (zero
-audio–image pairs in training).
+the base's input space, plus **28 gated adapters** (44.2M) that give the frozen
+language model in-layer capacity for audio. The adapters run only while audio is being
+encoded; every other forward pass returns the frozen layers' output untouched, so the
+invariance is bitwise, not approximate. Adding audio never invalidates an index you
+have already built.
+
+This v0.3 release ships two revisions of the same model. The default is tuned for
+audio-text retrieval; a keyword-tuned revision is available for spoken-command use.
+
+## Versions in this release
+
+| Revision | Use it for | AudioCaps a2t R@10 | SpeechCommands |
+| --- | --- | --- | --- |
+| **`v0.3-preview`** (default) | general audio-text retrieval, RAG, clustering | **0.785** | 0.894 |
+| `v0.3-kw-preview` | spoken-keyword and command retrieval | 0.749 | **0.929** |
+
+The default (`v0.3-preview`) is the recommended model for almost all use. The
+keyword-tuned revision trades audio-text retrieval quality for higher zero-shot
+keyword-spotting accuracy; use it only when spoken-command matching is the priority,
+and use the default otherwise.
+
+## Highlights (v0.3-preview)
 
 | Feature | Value |
 | --- | --- |
 | Parameters | ~2.06B frozen base + 640M frozen audio tower; **60.6M trained** |
 | Modalities | text, image, video, audio |
-| Supported tasks | `retrieval` (all modality pairs), `zero-shot classification` |
-| Max input | 254 text tokens · 30 s audio per window (up to 8 windows) |
-| Embedding dimension | 2048 |
-| Matryoshka dimensions | 64, 128, 256, 512, 1024, 1536, 2048 |
-| Pooling strategy | Last-token pooling |
+| AudioCaps 883 (audio-to-text) | R@1 0.333 · R@5 0.646 · R@10 **0.785** · mAP@10 0.255 |
+| AudioCaps 883 (text-to-audio) | R@1 0.297 · R@5 0.653 · R@10 0.782 · mAP@10 0.445 |
+| MAEB(audio-only), 19 tasks | mean **0.454**; leads the board on IEMOCAP speaker-gender (0.938) |
+| Non-audio preservation | text, image, video outputs bit-for-bit the base model's |
+| Embedding dimension | 2048 (Matryoshka: 64, 128, 256, 512, 1024, 1536, 2048) |
 | Base model | Qwen/Qwen3-VL-Embedding-2B (byte-frozen) |
 | Audio tower | Qwen/Qwen2.5-Omni-7B audio encoder (frozen) |
-| Trained components | FusionResampler 16.4M + 28× gated adapters 44.2M |
-| Distribution | ~250 MB trained components; frozen towers download from their original repos |
-
-## Training and Evaluation
-
-Contrastive training (InfoNCE over the Matryoshka ladder, symmetric) against the
-frozen base's native chat-template text embeddings: 518,183 audio–caption pairs from
-six sources (73,716 clips with content-free metadata excluded), a full-corpus
-frozen-text negative bank, soft labels 0.3, false-negative masking 0.98, bf16, 3,900
-steps at effective batch 1,024, then an in-domain fine-tune on an 86,394-pair
-AudioCaps pool: the original train split plus the 41,416 additional training clips
-introduced by AudioCaps 2.0 (v0.2: 1,600 steps at batch 128; v0.1 used the original
-45K split for 400 steps). AudioCaps 2.0 keeps the original validation and test
-splits unchanged; we re-verified that its train pool is disjoint from them before
-ingestion. All evaluation-set audio (Clotho, ESC-50, UrbanSound8K, VGGSound,
-AudioCaps test/val) is excluded from training by ID blacklists at ingestion. A
-technical report is in preparation.
-
-AudioCaps and AudioCaps 2.0 are by Kim et al. (AudioCaps: Generating Captions for
-Audios in The Wild, NAACL-HLT 2019; the 2.0 release, February 2025, in the AudioCaps
-repository) and are used for academic purposes per their terms; raw audio obtained
-via the authors' request form.
-
-All numbers below use the release protocol (bf16 base precision, native chat-template
-text). Bold marks the better value per row/column.
 
-<p align="center">
-<img src="assets/fe_positioning.png" alt="Positioning: VGGSound-696 cross-modal retrieval versus trained parameters; the fusion-embedding family leads unified models on audio-text and leads the emergent audio-image cluster (ImageBind's supervised pair annotated)" width="860px">
-</p>
+## Evaluation
 
 <details open>
-  <summary><b>Versus fusion-embedding-1 v0.3</b></summary>
-
-| Board / direction | fusion-embedding-1 v0.3 | fusion-embedding-2 (this repo, v0.2) |
-|---|---|---|
-| AudioCaps A→T R@1 | **0.332** | 0.310 |
-| AudioCaps A→T R@10 | 0.741 | **0.759** |
-| AudioCaps T→A R@1 | 0.280 | **0.303** |
-| AudioCaps T→A R@10 | 0.746 | **0.785** |
-| Clotho (zero-shot) A→T R@1 | 0.135 | 0.135 |
-| Clotho (zero-shot) A→T R@10 | **0.433** | 0.414 |
-| Clotho (zero-shot) T→A R@1 | 0.136 | **0.146** |
-| Clotho (zero-shot) T→A R@10 | 0.460 | **0.485** |
-| VGGSound audio→text R@1 | 0.213 | **0.216** |
-| VGGSound audio→text R@10 | 0.625 | **0.645** |
-| VGGSound text→audio R@1 | 0.213 | **0.257** |
-| VGGSound text→audio R@10 | 0.645 | **0.697** |
-| VGGSound audio→image R@10 (emergent) | 0.407 | **0.408** |
-
-fusion-embedding-2 v0.2 takes the large majority of cells, with its largest gains
-on AudioCaps (the expanded fine-tuning pool) and in the text→audio direction
-(searching audio with a text query). fusion-embedding-1 v0.3 retains the AudioCaps
-A→T R@1 and Clotho A→T R@10 cells; emergent audio→image is at parity at this
-operating point (0.408 vs 0.407). The pre-fine-tune fusion-embedding-2 checkpoint
-scores 0.443 on that cell — the project record — and may be released separately as
-the emergent-alignment operating point.
+  <summary>AudioCaps retrieval (883-clip test, min-rank over references)</summary>
 
-</details>
+| Model | a2t R@10 | t2a R@10 |
+| --- | --- | --- |
+| fusion-embedding-2 v0.2 | 0.743 | 0.775 |
+| **v0.3-preview** | **0.785** | 0.782 |
+| v0.3-kw-preview | 0.749 | 0.771 |
 
-<details>
-  <summary><b>Cross-modal retrieval — versus unified embedding models</b> (VGGSound-AV, 696 pairs, chance R@10 = 0.014)</summary>
-
-R@10 shown as audio-side → other / other → audio-side:
-
-| Model | audio↔image | audio↔text | text↔image |
-|---|---|---|---|
-| ImageBind-Huge | **0.718 / 0.720** | 0.404 / 0.348 | 0.243 / 0.282 |
-| LanguageBind | 0.365 / 0.415 | 0.547 / 0.331 | 0.221 / 0.283 |
-| Gemini Embedding 2 (API, 2026-07-09) | 0.312 / 0.316 | 0.379 / 0.374 | 0.273 / **0.366** |
-| fusion-embedding-1-2b-preview v0.3 | 0.407 / 0.428 | 0.625 / 0.645 | **0.331** / 0.319 |
-| **fusion-embedding-2-2b-preview** (v0.2) | 0.408 / 0.443 | **0.645 / 0.697** | **0.331** / 0.319 |
-
-ImageBind trains directly on audio–image pairs, so that pair is its supervised
-direction; its audio–text alignment is emergent. LanguageBind trains audio against
-language; its audio↔image is emergent. Both fusion-embedding generations train on
-audio–text only; their audio–image alignment is emergent. All models evaluated with
-identical clips, frames, and scoring, using the released imagebind_huge checkpoint and
-revision-pinned LanguageBind checkpoints. Gemini Embedding 2 is Google's natively
-multimodal embedding API, evaluated at its documented default invocation on the date
-shown; API models may change after that date. fusion-embedding-2's text↔image cells
-are identical to fusion-embedding-1's by construction — text and images never touch
-the trained components — and this is verified: the v0.2 readout run reproduces
-fusion-embedding-1 v0.3's text→image scores to four decimal places.
+The v0.3 default improves audio-to-text retrieval by 4.2 points over v0.2.
 
 </details>
 
 <details>
-  <summary><b>Audio–text retrieval — versus specialist CLAP models</b></summary>
+  <summary>MAEB(audio-only), full 19-task board — v0.3-preview</summary>
 
-Specialist CLAP models fine-tune their text towers on audio captions — the direct
-trade this architecture declines in order to keep one shared space for all four
-modalities. They remain ahead on the audio-caption boards (e.g., AudioCaps T→A R@1:
-M2D-CLAP 41.4 vs 29.2 here); this model family is the strongest option we measured
-when one model must serve text, images, video, and audio together. See the
-[fusion-embedding-1 card](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview)
-for the full CLAP comparison tables; fusion-embedding-2 improves on fusion-embedding-1
-in the text→audio direction on every board.
+Mean 0.4536. Scores are the main metric per task.
+
+| Task | Score |
+| --- | --- |
+| IEMOCAPGender | 0.938 |
+| BeijingOpera | 0.924 |
+| NMSQAPairClassification | 0.794 |
+| GTZANAudioReranking | 0.717 |
+| JamAltArtistA2ARetrieval | 0.687 |
+| GTZANGenre | 0.644 |
+| VoxPopuliLanguageID | 0.616 |
+| MInDS14 | 0.578 |
+| CREMADPairClassification | 0.543 |
+| MridinghamTonic | 0.540 |
+| VoxPopuliAccentPairClassification | 0.509 |
+| CREMA_D | 0.277 |
+| VoxCelebSA | 0.273 |
+| BirdCLEF | 0.184 |
+| CommonLanguageAgeDetection | 0.171 |
+| SIBFLEURS | 0.115 |
+| VoxPopuliGenderClustering | 0.079 |
+| VehicleSoundClustering | 0.025 |
+| CREMA_DClustering | 0.006 |
+
+IEMOCAP speaker-gender classification is the strongest cell relative to the field.
+Clustering tasks are the weakest and are a target for the next release.
 
 </details>
 
 <details>
-  <summary><b>MAEB (public leaderboard) — nine sound-event tasks</b></summary>
-
-Both generations are on the public MAEB(beta) leaderboard through the mteb registry.
-The v0.2 rescore (revision `9451b840f0d1`, `mteb==2.18.x`, identical protocol to the
-v0.1 submission):
-
-| Task | v0.2 | v0.1 |
-|---|---:|---:|
-| BeijingOpera | **.907** | .898 |
-| ClothoT2ARetrieval | .278 | **.285** |
-| GTZANAudioReranking | **.731** | .723 |
-| GTZANGenre | **.665** | .647 |
-| MACST2ARetrieval | .158 | .158 |
-| RavdessZeroshot | **.348** | .304 |
-| SpeechCommandsZeroshotv0.02 | .133 | **.141** |
-| UrbanSound8KT2ARetrieval | **.010** | .009 |
-| VehicleSoundClustering | **.024** | .018 |
-| Mean | **.361** | .354 |
-
-Six of nine tasks improve; the two dips (Clotho task protocol, speech commands) are
-within the noise band of these task sizes. Speech-content and fine-grained
-classification remain the corpus's known weak axes — the training pool is
-sound-event-centric, and speech is a roadmap item, not a claim.
+  <summary>Non-audio preservation and other modalities</summary>
+
+Text, image, and video outputs are bit-for-bit identical to the frozen
+Qwen3-VL-Embedding-2B base (the adapter gate is closed on those inputs). Their
+benchmark scores are therefore the base model's own, unchanged by anything added for
+audio. Audio-to-image retrieval is emergent: the model is trained on audio-text pairs
+only, with no audio-image supervision.
 
 </details>
 
-## Changelog
+## Notes and limitations
 
-- **v0.2** (2026-07-20): fine-tuned on the expanded AudioCaps 2.0 training pool
-  (86,394 pairs, 1,600 steps). AudioCaps A→T R@10 0.743 → 0.759, T→A R@10 0.775 →
-  0.785; Clotho and VGGSound within noise of v0.1, emergent audio→image up ~1.5
-  points. All comparisons run against v0.1 under identical protocols. Non-audio
-  outputs remain bit-for-bit the base model's (verified on this artifact).
-- **v0.1** (2026-07-12): initial release (`v0.1-preview` tag; the revision pinned
-  by the MTEB/MAEB leaderboard submission).
+- **Emotion recognition regressed** relative to v0.2. On RAVDESS zero-shot emotion the
+  score is 0.21 (v0.2: 0.35). The v0.3 training mix improved spoken-content
+  understanding at the cost of vocal-prosody sensitivity. If speech-emotion is central
+  to your use, evaluate before adopting.
+- One evaluation cell, `CommonLanguageAgeDetection`, derives from Common Voice, which is
+  part of the training data. It scores low (0.171) and does not inflate the reported
+  mean, but the potential overlap is noted for completeness.
+- These are research previews under CC-BY-NC-4.0.
 
 ## Usage
 
@@ -230,7 +172,8 @@ from inference import FusionEmbedder
 
 fe = FusionEmbedder.from_pretrained(
     "EximiusLabs/fusion-embedding-2-2b-preview",
-    revision="v0.2-preview",   # pin a tag if you build on this model
+    revision="v0.3-preview",           # default, retrieval-tuned
+    # revision="v0.3-kw-preview",      # keyword-tuned alternative
 )
 
 a = fe.embed_audio("dog.wav")            # audio file or (array, sr=...)
@@ -243,11 +186,9 @@ print((a @ t).item(), (a @ i).item())    # cosine similarities in the shared spa
 t_small = fe.embed_text("a dog barks", dim=256)
 ```
 
-The checkpoint contains the gated adapters and the loader refuses to run without them —
-an adapter checkpoint can never be silently executed as the first-generation
-architecture. All inputs use the base model's chat-template format; embedding quality
-is sensitive to this formatting, so use the templates provided by `FusionEmbedder`
-rather than constructing your own.
+The checkpoint contains the gated adapters and the loader refuses to run without them.
+All inputs use the base model's chat-template format; embedding quality is sensitive to
+this formatting, so use the templates provided by `FusionEmbedder`.
 
 </details>
 
@@ -255,8 +196,7 @@ rather than constructing your own.
   <summary>Cross-modal ranking tip</summary>
 
 When ranking a gallery of one modality against queries of another, per-modality
-mean-centering of the gallery improves cross-modal recall by roughly two points across
-modality pairs:
+mean-centering of the gallery improves cross-modal recall:
 
 ```python
 gallery = FusionEmbedder.center(gallery_embeddings)
@@ -264,31 +204,43 @@ gallery = FusionEmbedder.center(gallery_embeddings)
 
 </details>
 
+## Changelog
+
+- **v0.3-preview** (default): retrieval-tuned flagship. AudioCaps a2t R@10 0.785
+  (+0.042 over v0.2); MAEB(audio-only) mean 0.454; leads IEMOCAP speaker-gender.
+  RAVDESS emotion regressed to 0.21.
+- **v0.3-kw-preview**: keyword-tuned revision. SpeechCommands zero-shot 0.929;
+  AudioCaps a2t R@10 0.749.
+- **v0.2-preview**: AudioCaps 2.0 fine-tune. a2t R@10 0.743.
+- **v0.1-preview**: first modality-gated-adapter release.
+
 ## License
 
-Code is Apache-2.0 ([GitHub](https://github.com/Eximius-Labs/fusion-embedding));
-model weights in this repository are **CC BY-NC 4.0** (research preview). The frozen
-base and audio tower retain their original licenses.
+Weights (all revisions): **CC-BY-NC-4.0** (research preview). The code is Apache-2.0.
+The frozen base and audio tower retain their original licenses.
 
 ## Citation
 
-To cite the model:
+Model:
 
 ```bibtex
 @software{fusion_embedding_2_2026,
-  title  = {fusion-embedding-2},
+  title  = {Fusion Embedding 2: Modality-Gated Deep Adapters for a
+            Unified Text, Image, Video, and Audio Embedding Space},
   author = {Tonmoy, Abdul Basit},
   year   = {2026},
   url    = {https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview}
 }
 ```
 
-To cite the paper:
+Technical report:
 
 ```bibtex
 @article{tonmoy2026fusion,
-  title   = {Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio},
-  author  = {Tonmoy, Abdul Basit and Hoque, Kazi Fardinul and Arham, Md. Shahrier Islam and Luthra, Arman},
+  title   = {Fusion Embedding: A Unified Embedding Space for Text, Image,
+             Video, and Audio},
+  author  = {Tonmoy, Abdul Basit and Hoque, Kazi Fardinul and
+             Arham, Md. Shahrier Islam and Luthra, Arman},
   journal = {arXiv preprint arXiv:2607.18666},
   year    = {2026}
 }

config.json

@@ -23,29 +23,19 @@
   },
   "benchmarks": {
     "audiocaps_883_minrank5": {
-      "eval_shard": "audiocaps_test816",
-      "n_clips": 883,
-      "n_captions": 4411,
-      "dim": 1024,
-      "n_total_clips": 883,
-      "split_filtered": false,
-      "caption_index": -1,
-      "caption_field": "",
-      "protocol": "min-rank-over-refs",
-      "a2t_R@1": 0.31030577421188354,
-      "a2t_R@5": 0.6217440366744995,
-      "a2t_R@10": 0.7587769031524658,
-      "a2t_mAP@10": 0.24294079840183258,
-      "t2a_R@1": 0.3028791546821594,
-      "t2a_R@5": 0.6472455263137817,
-      "t2a_R@10": 0.7846293449401855,
-      "t2a_mAP@10": 0.44739025831222534
+      "a2t_R@1": 0.333,
+      "a2t_R@5": 0.646,
+      "a2t_R@10": 0.785,
+      "a2t_mAP@10": 0.255,
+      "t2a_R@1": 0.297,
+      "t2a_R@5": 0.653,
+      "t2a_R@10": 0.782,
+      "t2a_mAP@10": 0.445
     },
-    "in_domain_eval": {
-      "a2t_R@1": 0.4449999928474426,
-      "a2t_R@10": 0.8899999856948853,
-      "a2t_mAP@10": 0.6005654335021973
-    }
+    "maeb_audio_only_19_mean": 0.4536,
+    "maeb_rank": "8 of 14 (full-coverage)",
+    "speechcommands_zeroshot": 0.894,
+    "ravdess_zeroshot": 0.211
   },
   "d_audio": 3584,
   "resampler_depth": 6,
@@ -68,5 +58,7 @@
   "auto_map": {
     "AutoConfig": "configuration_fusion_embedding.FusionEmbeddingConfig",
     "AutoModel": "modeling_fusion_embedding.FusionEmbeddingModel"
-  }
+  },
+  "base_4bit": false,
+  "version": "v0.3-preview"
 }

fusion-embedding-2-2b-preview.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d3fba2dd774caa688fd28f1b9368d6b76e99968aa665a72da65f27fdaab65784
-size 242315846
+oid sha256:8e59e13b3706fb2d7e5892c1e36f1ce1f64daea4fdaead291eeff6990e6a9864
+size 242315590

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9f708155f58f24a2a3aaa732392a9bcb3f1943a8ff282af3703226fd3d83a4f9
-size 242252933
+oid sha256:bf68dfe2dd41c39989526134751dc7a401967c049ef105424105d2b8ea3bb399
+size 242252925