Card: restore banner, add RunPod deploy

README.md

@@ -14,6 +14,10 @@ tags:
 base_model: Qwen/Qwen3-VL-Embedding-2B
 ---
 
+<p align="center">
+  <img src="assets/fusion-embedding-2-banner.png" alt="Fusion Embedding 2 (2B-Preview) — Eximius Labs" width="100%">
+</p>
+
 # fusion-embedding-2-2b-preview
 
 <div align="center">
@@ -21,7 +25,8 @@ base_model: Qwen/Qwen3-VL-Embedding-2B
 [![Python](https://img.shields.io/badge/python-3.11+-blue.svg)](https://github.com/Eximius-Labs/fusion-embedding)
 [![PyTorch](https://img.shields.io/badge/PyTorch-2.x-ee4c2c.svg)](https://github.com/Eximius-Labs/fusion-embedding)
 [![Weights](https://img.shields.io/badge/weights-CC--BY--NC--4.0-green.svg)](#license)
-[![Status](https://img.shields.io/badge/status-research%20preview%20v0.1-orange.svg)](#)
+[![Status](https://img.shields.io/badge/status-research%20preview%20v0.3-orange.svg)](#)
+[![Paper](https://img.shields.io/badge/paper-arXiv%3A2607.18666-b31b1b.svg)](https://arxiv.org/abs/2607.18666)
 [![Code](https://img.shields.io/badge/code-GitHub-black.svg)](https://github.com/Eximius-Labs/fusion-embedding)
 [![Deploy on RunPod](https://api.runpod.io/badge/Eximius-Labs/fusion-embedding)](https://www.runpod.io/console/hub/Eximius-Labs/fusion-embedding)
 
@@ -29,12 +34,13 @@ base_model: Qwen/Qwen3-VL-Embedding-2B
 
 `fusion-embedding-2-2b-preview` is the second generation of Eximius Labs' unified
 multimodal embedding models: **text, images, video, and audio in one vector space**.
-It extends the first generation with modality-gated deep adapters — in-layer audio
-capacity added to a byte-frozen base. For the first-generation architecture, see
+Audio is added to a byte-frozen vision-language base through modality-gated deep
+adapters, so gaining audio never changes a single existing text, image, or video
+vector. For the first-generation architecture see
 [fusion-embedding-1-2b-preview](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview)
 (that line is final at v0.3).
 
-[GitHub](https://github.com/Eximius-Labs/fusion-embedding) | [fusion-embedding-1](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview) | Technical report: in preparation
+[GitHub](https://github.com/Eximius-Labs/fusion-embedding) | Technical report: [arXiv:2607.18666](https://arxiv.org/abs/2607.18666)
 
 ## Model Overview
 
@@ -42,127 +48,116 @@ capacity added to a byte-frozen base. For the first-generation architecture, see
 <img src="assets/fe2_model_overview.png" alt="fusion-embedding-2 architecture: frozen Qwen3-VL-Embedding base with modality-gated adapters inside; frozen audio tower and trained FusionResampler on the audio branch; one shared embedding space" width="820px">
 </p>
 
-`fusion-embedding-2-2b-preview` embeds all four modalities with a
+`fusion-embedding-2-2b-preview` embeds all four modalities on a
 [Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) base that is
-**byte-identical to its original release** — its text, image, and video behaviour (and
-benchmark scores) carry over exactly. Audio is added by training 60.6M parameters
-(~2.3% of the stack): a perceiver-resampler that translates frozen
+**byte-identical to its original release**: its text, image, and video behaviour, and
+its benchmark scores, carry over exactly. Audio is added by training 60.6M parameters
+(about 2.3% of the stack): a perceiver-resampler that maps frozen
 [Qwen2.5-Omni](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) audio-tower features into
-the base's input space, and — new in this generation — **28 gated adapters** (44.2M)
-that give the frozen language model in-layer capacity to process audio. The adapters
-are active only while encoding audio; every other forward pass returns the frozen
-layers' output untouched, so the invariance is bitwise, not approximate
-(`base_drift == 0` is asserted on every training run, and this model reproduces the
-base's text→image retrieval scores to four decimal places). Trained on 518K
-audio–caption pairs with a full-corpus frozen-text negative bank, it leads every
-unified embedding model we measured on audio↔text retrieval — ahead of ImageBind,
-LanguageBind, and Gemini Embedding 2 in both directions — and improves on
-fusion-embedding-1 v0.3 in 8 of 12 release-protocol cells, including every
-recorded text→audio direction. Audio↔image alignment is emergent (zero
-audio–image pairs in training).
+the base's input space, plus **28 gated adapters** (44.2M) that give the frozen
+language model in-layer capacity for audio. The adapters run only while audio is being
+encoded; every other forward pass returns the frozen layers' output untouched, so the
+invariance is bitwise, not approximate. Adding audio never invalidates an index you
+have already built.
+
+This v0.3 release ships two revisions of the same model. The default is tuned for
+audio-text retrieval; a keyword-tuned revision is available for spoken-command use.
+
+## Versions in this release
+
+| Revision | Use it for | AudioCaps a2t R@10 | SpeechCommands |
+| --- | --- | --- | --- |
+| **`v0.3-preview`** (default) | general audio-text retrieval, RAG, clustering | **0.785** | 0.894 |
+| `v0.3-kw-preview` | spoken-keyword and command retrieval | 0.749 | **0.929** |
+
+The default (`v0.3-preview`) is the recommended model for almost all use. The
+keyword-tuned revision trades audio-text retrieval quality for higher zero-shot
+keyword-spotting accuracy; use it only when spoken-command matching is the priority,
+and use the default otherwise.
+
+## Highlights (v0.3-preview)
 
 | Feature | Value |
 | --- | --- |
 | Parameters | ~2.06B frozen base + 640M frozen audio tower; **60.6M trained** |
 | Modalities | text, image, video, audio |
-| Supported tasks | `retrieval` (all modality pairs), `zero-shot classification` |
-| Max input | 254 text tokens · 30 s audio per window (up to 8 windows) |
-| Embedding dimension | 2048 |
-| Matryoshka dimensions | 64, 128, 256, 512, 1024, 1536, 2048 |
-| Pooling strategy | Last-token pooling |
+| AudioCaps 883 (audio-to-text) | R@1 0.333 · R@5 0.646 · R@10 **0.785** · mAP@10 0.255 |
+| AudioCaps 883 (text-to-audio) | R@1 0.297 · R@5 0.653 · R@10 0.782 · mAP@10 0.445 |
+| MAEB(audio-only), 19 tasks | mean **0.454**; leads the board on IEMOCAP speaker-gender (0.938) |
+| Non-audio preservation | text, image, video outputs bit-for-bit the base model's |
+| Embedding dimension | 2048 (Matryoshka: 64, 128, 256, 512, 1024, 1536, 2048) |
 | Base model | Qwen/Qwen3-VL-Embedding-2B (byte-frozen) |
 | Audio tower | Qwen/Qwen2.5-Omni-7B audio encoder (frozen) |
-| Trained components | FusionResampler 16.4M + 28× gated adapters 44.2M |
-| Distribution | ~250 MB trained components; frozen towers download from their original repos |
 
-## Training and Evaluation
+## Evaluation
 
-Contrastive training (InfoNCE over the Matryoshka ladder, symmetric) against the
-frozen base's native chat-template text embeddings: 518,183 audio–caption pairs from
-six sources (73,716 clips with content-free metadata excluded), a full-corpus
-frozen-text negative bank, soft labels 0.3, false-negative masking 0.98, bf16, 3,900
-steps at effective batch 1,024, then a 400-step in-domain fine-tune on the AudioCaps
-train split. All evaluation-set audio (Clotho, ESC-50, UrbanSound8K, VGGSound,
-AudioCaps test/val) is excluded from training by ID blacklists at ingestion. A
-technical report is in preparation.
+<details open>
+  <summary>AudioCaps retrieval (883-clip test, min-rank over references)</summary>
 
-All numbers below use the release protocol (bf16 base precision, native chat-template
-text). Bold marks the better value per row/column.
+| Model | a2t R@10 | t2a R@10 |
+| --- | --- | --- |
+| fusion-embedding-2 v0.2 | 0.743 | 0.775 |
+| **v0.3-preview** | **0.785** | 0.782 |
+| v0.3-kw-preview | 0.749 | 0.771 |
 
-<p align="center">
-<img src="assets/fe_positioning.png" alt="Positioning: VGGSound-696 cross-modal retrieval versus trained parameters; the fusion-embedding family leads unified models on audio-text and leads the emergent audio-image cluster (ImageBind's supervised pair annotated)" width="860px">
-</p>
-
-<details open>
-  <summary><b>Versus fusion-embedding-1 v0.3</b></summary>
-
-| Board / direction | fusion-embedding-1 v0.3 | fusion-embedding-2 (this repo) |
-|---|---|---|
-| AudioCaps A→T R@1 | **0.332** | 0.302 |
-| AudioCaps A→T R@10 | 0.741 | **0.743** |
-| AudioCaps T→A R@1 | 0.280 | **0.292** |
-| AudioCaps T→A R@10 | 0.746 | **0.775** |
-| Clotho (zero-shot) A→T R@1 | **0.135** | 0.127 |
-| Clotho (zero-shot) A→T R@10 | **0.433** | 0.421 |
-| Clotho (zero-shot) T→A R@1 | 0.136 | **0.151** |
-| Clotho (zero-shot) T→A R@10 | 0.460 | **0.482** |
-| VGGSound audio→text R@1 | **0.213** | 0.211 |
-| VGGSound audio→text R@10 | 0.625 | **0.665** |
-| VGGSound text→audio R@1 | 0.213 | **0.266** |
-| VGGSound text→audio R@10 | 0.645 | **0.681** |
-| VGGSound audio→image R@10 (emergent) | **0.407** | 0.392 |
-
-fusion-embedding-2 takes the majority of cells, with its largest gains in the
-text→audio direction (searching audio with a text query) and on the cross-modal
-audio↔text pair. fusion-embedding-1 v0.3 retains the AudioCaps and Clotho A→T R@1
-cells and a ~1.5-point edge on emergent audio→image at this fine-tuned operating
-point; the pre-fine-tune fusion-embedding-2 checkpoint scores 0.443 on that cell — the
-project record — and may be released separately as the emergent-alignment operating
-point.
+The v0.3 default improves audio-to-text retrieval by 4.2 points over v0.2.
 
 </details>
 
 <details>
-  <summary><b>Cross-modal retrieval — versus unified embedding models</b> (VGGSound-AV, 696 pairs, chance R@10 = 0.014)</summary>
-
-R@10 shown as audio-side → other / other → audio-side:
-
-| Model | audio↔image | audio↔text | text↔image |
-|---|---|---|---|
-| ImageBind-Huge | **0.718 / 0.720** | 0.404 / 0.348 | 0.243 / 0.282 |
-| LanguageBind | 0.365 / 0.415 | 0.547 / 0.331 | 0.221 / 0.283 |
-| Gemini Embedding 2 (API, 2026-07-09) | 0.312 / 0.316 | 0.379 / 0.374 | 0.273 / **0.366** |
-| fusion-embedding-1-2b-preview v0.3 | 0.407 / 0.428 | 0.625 / 0.645 | **0.331** / 0.319 |
-| **fusion-embedding-2-2b-preview** | 0.392 / 0.430 | **0.665 / 0.681** | **0.331** / 0.319 |
-
-ImageBind trains directly on audio–image pairs, so that pair is its supervised
-direction; its audio–text alignment is emergent. LanguageBind trains audio against
-language; its audio↔image is emergent. Both fusion-embedding generations train on
-audio–text only; their audio–image alignment is emergent. All models evaluated with
-identical clips, frames, and scoring, using the released imagebind_huge checkpoint and
-revision-pinned LanguageBind checkpoints. Gemini Embedding 2 is Google's natively
-multimodal embedding API, evaluated at its documented default invocation on the date
-shown; API models may change after that date. fusion-embedding-2's text↔image cells
-are identical to fusion-embedding-1's by construction — text and images never touch
-the trained components — and this is verified: its own readout run reproduces
-fusion-embedding-1 v0.3's text→image scores to four decimal places.
+  <summary>MAEB(audio-only), full 19-task board — v0.3-preview</summary>
+
+Mean 0.4536. Scores are the main metric per task.
+
+| Task | Score |
+| --- | --- |
+| IEMOCAPGender | 0.938 |
+| BeijingOpera | 0.924 |
+| NMSQAPairClassification | 0.794 |
+| GTZANAudioReranking | 0.717 |
+| JamAltArtistA2ARetrieval | 0.687 |
+| GTZANGenre | 0.644 |
+| VoxPopuliLanguageID | 0.616 |
+| MInDS14 | 0.578 |
+| CREMADPairClassification | 0.543 |
+| MridinghamTonic | 0.540 |
+| VoxPopuliAccentPairClassification | 0.509 |
+| CREMA_D | 0.277 |
+| VoxCelebSA | 0.273 |
+| BirdCLEF | 0.184 |
+| CommonLanguageAgeDetection | 0.171 |
+| SIBFLEURS | 0.115 |
+| VoxPopuliGenderClustering | 0.079 |
+| VehicleSoundClustering | 0.025 |
+| CREMA_DClustering | 0.006 |
+
+IEMOCAP speaker-gender classification is the strongest cell relative to the field.
+Clustering tasks are the weakest and are a target for the next release.
 
 </details>
 
 <details>
-  <summary><b>Audio–text retrieval — versus specialist CLAP models</b></summary>
+  <summary>Non-audio preservation and other modalities</summary>
 
-Specialist CLAP models fine-tune their text towers on audio captions — the direct
-trade this architecture declines in order to keep one shared space for all four
-modalities. They remain ahead on the audio-caption boards (e.g., AudioCaps T→A R@1:
-M2D-CLAP 41.4 vs 29.2 here); this model family is the strongest option we measured
-when one model must serve text, images, video, and audio together. See the
-[fusion-embedding-1 card](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview)
-for the full CLAP comparison tables; fusion-embedding-2 improves on fusion-embedding-1
-in the text→audio direction on every board.
+Text, image, and video outputs are bit-for-bit identical to the frozen
+Qwen3-VL-Embedding-2B base (the adapter gate is closed on those inputs). Their
+benchmark scores are therefore the base model's own, unchanged by anything added for
+audio. Audio-to-image retrieval is emergent: the model is trained on audio-text pairs
+only, with no audio-image supervision.
 
 </details>
 
+## Notes and limitations
+
+- **Emotion recognition regressed** relative to v0.2. On RAVDESS zero-shot emotion the
+  score is 0.21 (v0.2: 0.35). The v0.3 training mix improved spoken-content
+  understanding at the cost of vocal-prosody sensitivity. If speech-emotion is central
+  to your use, evaluate before adopting.
+- One evaluation cell, `CommonLanguageAgeDetection`, derives from Common Voice, which is
+  part of the training data. It scores low (0.171) and does not inflate the reported
+  mean, but the potential overlap is noted for completeness.
+- These are research previews under CC-BY-NC-4.0.
+
 ## Usage
 
 <details>
@@ -182,7 +177,8 @@ from inference import FusionEmbedder
 
 fe = FusionEmbedder.from_pretrained(
     "EximiusLabs/fusion-embedding-2-2b-preview",
-    revision="v0.1-preview",   # pin a tag if you build on this model
+    revision="v0.3-preview",           # default, retrieval-tuned
+    # revision="v0.3-kw-preview",      # keyword-tuned alternative
 )
 
 a = fe.embed_audio("dog.wav")            # audio file or (array, sr=...)
@@ -195,11 +191,9 @@ print((a @ t).item(), (a @ i).item())    # cosine similarities in the shared spa
 t_small = fe.embed_text("a dog barks", dim=256)
 ```
 
-The checkpoint contains the gated adapters and the loader refuses to run without them —
-an adapter checkpoint can never be silently executed as the first-generation
-architecture. All inputs use the base model's chat-template format; embedding quality
-is sensitive to this formatting, so use the templates provided by `FusionEmbedder`
-rather than constructing your own.
+The checkpoint contains the gated adapters and the loader refuses to run without them.
+All inputs use the base model's chat-template format; embedding quality is sensitive to
+this formatting, so use the templates provided by `FusionEmbedder`.
 
 </details>
 
@@ -207,8 +201,7 @@ rather than constructing your own.
   <summary>Cross-modal ranking tip</summary>
 
 When ranking a gallery of one modality against queries of another, per-modality
-mean-centering of the gallery improves cross-modal recall by roughly two points across
-modality pairs:
+mean-centering of the gallery improves cross-modal recall:
 
 ```python
 gallery = FusionEmbedder.center(gallery_embeddings)
@@ -216,32 +209,42 @@ gallery = FusionEmbedder.center(gallery_embeddings)
 
 </details>
 
-### Deploy as an API on RunPod
+## Changelog
+
+- **v0.3-preview** (default): retrieval-tuned flagship. AudioCaps a2t R@10 0.785
+  (+0.042 over v0.2); MAEB(audio-only) mean 0.454; leads IEMOCAP speaker-gender.
+  RAVDESS emotion regressed to 0.21.
+- **v0.3-kw-preview**: keyword-tuned revision. SpeechCommands zero-shot 0.929;
+  AudioCaps a2t R@10 0.749.
+- **v0.2-preview**: AudioCaps 2.0 fine-tune. a2t R@10 0.743.
+- **v0.1-preview**: first modality-gated-adapter release.
+
+## Deploy on RunPod
+
+[![Deploy on RunPod](https://api.runpod.io/badge/Eximius-Labs/fusion-embedding)](https://www.runpod.io/console/hub/Eximius-Labs/fusion-embedding)
 
 One-click deploy the endpoint from the
-[RunPod Hub](https://www.runpod.io/console/hub/Eximius-Labs/fusion-embedding)
-(serverless, scales to zero when idle). Once it is running, call it:
+[RunPod Hub](https://www.runpod.io/console/hub/Eximius-Labs/fusion-embedding) (serverless,
+scales to zero when idle). Once it is running, call it:
 
 ```bash
-curl -s https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync \
-  -H "Authorization: Bearer $RUNPOD_API_KEY" \
-  -H "Content-Type: application/json" \
-  -d '{"input": {"text": "a dog on a beach"}}'
+curl -s https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync   -H "Authorization: Bearer $RUNPOD_API_KEY"   -H "Content-Type: application/json"   -d '{"input": {"text": "a dog on a beach"}}'
 ```
 
-The endpoint embeds all four modalities. Swap `text` for `image`, `video`, or
-`audio` (the value is an https URL, a data URI, or base64). Audio needs the audio
-tower, so set `FE2_ENABLE_AUDIO=1` on the endpoint and use a 24 GB or larger GPU.
-Returns 1024-d embeddings; add `"dim": 512` to truncate.
+The endpoint embeds all four modalities. Swap `text` for `image`, `video`, or `audio`
+(the value is an https URL, a data URI, or base64). Audio needs the audio tower, so set
+`FE2_ENABLE_AUDIO=1` on the endpoint and use a 24 GB or larger GPU. Returns 1024-d
+embeddings; add `"dim": 512` to truncate.
 
 ## License
 
-Code is Apache-2.0 ([GitHub](https://github.com/Eximius-Labs/fusion-embedding));
-model weights in this repository are **CC BY-NC 4.0** (research preview). The frozen
-base and audio tower retain their original licenses.
+Weights (all revisions): **CC-BY-NC-4.0** (research preview). The code is Apache-2.0.
+The frozen base and audio tower retain their original licenses.
 
 ## Citation
 
+Model:
+
 ```bibtex
 @software{fusion_embedding_2_2026,
   title  = {Fusion Embedding 2: Modality-Gated Deep Adapters for a
@@ -251,3 +254,16 @@ base and audio tower retain their original licenses.
   url    = {https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview}
 }
 ```
+
+Technical report:
+
+```bibtex
+@article{tonmoy2026fusion,
+  title   = {Fusion Embedding: A Unified Embedding Space for Text, Image,
+             Video, and Audio},
+  author  = {Tonmoy, Abdul Basit and Hoque, Kazi Fardinul and
+             Arham, Md. Shahrier Islam and Luthra, Arman},
+  journal = {arXiv preprint arXiv:2607.18666},
+  year    = {2026}
+}
+```