Card: add RunPod Hub deploy badge + endpoint call instructions

README.md

@@ -14,10 +14,6 @@ tags:
 base_model: Qwen/Qwen3-VL-Embedding-2B
 ---
 
-<p align="center">
-  <img src="assets/fusion-embedding-2-banner.png" alt="Fusion Embedding 2 (2B-Preview) — Eximius Labs" width="100%">
-</p>
-
 # fusion-embedding-2-2b-preview
 
 <div align="center">
@@ -25,21 +21,20 @@ base_model: Qwen/Qwen3-VL-Embedding-2B
 [![Python](https://img.shields.io/badge/python-3.11+-blue.svg)](https://github.com/Eximius-Labs/fusion-embedding)
 [![PyTorch](https://img.shields.io/badge/PyTorch-2.x-ee4c2c.svg)](https://github.com/Eximius-Labs/fusion-embedding)
 [![Weights](https://img.shields.io/badge/weights-CC--BY--NC--4.0-green.svg)](#license)
-[![Status](https://img.shields.io/badge/status-research%20preview%20v0.3-orange.svg)](#)
-[![Paper](https://img.shields.io/badge/paper-arXiv%3A2607.18666-b31b1b.svg)](https://arxiv.org/abs/2607.18666)
+[![Status](https://img.shields.io/badge/status-research%20preview%20v0.1-orange.svg)](#)
 [![Code](https://img.shields.io/badge/code-GitHub-black.svg)](https://github.com/Eximius-Labs/fusion-embedding)
+[![Deploy on RunPod](https://api.runpod.io/badge/Eximius-Labs/fusion-embedding)](https://www.runpod.io/console/hub/Eximius-Labs/fusion-embedding)
 
 </div>
 
 `fusion-embedding-2-2b-preview` is the second generation of Eximius Labs' unified
 multimodal embedding models: **text, images, video, and audio in one vector space**.
-Audio is added to a byte-frozen vision-language base through modality-gated deep
-adapters, so gaining audio never changes a single existing text, image, or video
-vector. For the first-generation architecture see
+It extends the first generation with modality-gated deep adapters — in-layer audio
+capacity added to a byte-frozen base. For the first-generation architecture, see
 [fusion-embedding-1-2b-preview](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview)
 (that line is final at v0.3).
 
-[GitHub](https://github.com/Eximius-Labs/fusion-embedding) | Technical report: [arXiv:2607.18666](https://arxiv.org/abs/2607.18666)
+[GitHub](https://github.com/Eximius-Labs/fusion-embedding) | [fusion-embedding-1](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview) | Technical report: in preparation
 
 ## Model Overview
 
@@ -47,116 +42,127 @@ vector. For the first-generation architecture see
 <img src="assets/fe2_model_overview.png" alt="fusion-embedding-2 architecture: frozen Qwen3-VL-Embedding base with modality-gated adapters inside; frozen audio tower and trained FusionResampler on the audio branch; one shared embedding space" width="820px">
 </p>
 
-`fusion-embedding-2-2b-preview` embeds all four modalities on a
+`fusion-embedding-2-2b-preview` embeds all four modalities with a
 [Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) base that is
-**byte-identical to its original release**: its text, image, and video behaviour, and
-its benchmark scores, carry over exactly. Audio is added by training 60.6M parameters
-(about 2.3% of the stack): a perceiver-resampler that maps frozen
+**byte-identical to its original release** — its text, image, and video behaviour (and
+benchmark scores) carry over exactly. Audio is added by training 60.6M parameters
+(~2.3% of the stack): a perceiver-resampler that translates frozen
 [Qwen2.5-Omni](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) audio-tower features into
-the base's input space, plus **28 gated adapters** (44.2M) that give the frozen
-language model in-layer capacity for audio. The adapters run only while audio is being
-encoded; every other forward pass returns the frozen layers' output untouched, so the
-invariance is bitwise, not approximate. Adding audio never invalidates an index you
-have already built.
-
-This v0.3 release ships two revisions of the same model. The default is tuned for
-audio-text retrieval; a keyword-tuned revision is available for spoken-command use.
-
-## Versions in this release
-
-| Revision | Use it for | AudioCaps a2t R@10 | SpeechCommands |
-| --- | --- | --- | --- |
-| **`v0.3-preview`** (default) | general audio-text retrieval, RAG, clustering | **0.785** | 0.894 |
-| `v0.3-kw-preview` | spoken-keyword and command retrieval | 0.749 | **0.929** |
-
-The default (`v0.3-preview`) is the recommended model for almost all use. The
-keyword-tuned revision trades audio-text retrieval quality for higher zero-shot
-keyword-spotting accuracy; use it only when spoken-command matching is the priority,
-and use the default otherwise.
-
-## Highlights (v0.3-preview)
+the base's input space, and — new in this generation — **28 gated adapters** (44.2M)
+that give the frozen language model in-layer capacity to process audio. The adapters
+are active only while encoding audio; every other forward pass returns the frozen
+layers' output untouched, so the invariance is bitwise, not approximate
+(`base_drift == 0` is asserted on every training run, and this model reproduces the
+base's text→image retrieval scores to four decimal places). Trained on 518K
+audio–caption pairs with a full-corpus frozen-text negative bank, it leads every
+unified embedding model we measured on audio↔text retrieval — ahead of ImageBind,
+LanguageBind, and Gemini Embedding 2 in both directions — and improves on
+fusion-embedding-1 v0.3 in 8 of 12 release-protocol cells, including every
+recorded text→audio direction. Audio↔image alignment is emergent (zero
+audio–image pairs in training).
 
 | Feature | Value |
 | --- | --- |
 | Parameters | ~2.06B frozen base + 640M frozen audio tower; **60.6M trained** |
 | Modalities | text, image, video, audio |
-| AudioCaps 883 (audio-to-text) | R@1 0.333 · R@5 0.646 · R@10 **0.785** · mAP@10 0.255 |
-| AudioCaps 883 (text-to-audio) | R@1 0.297 · R@5 0.653 · R@10 0.782 · mAP@10 0.445 |
-| MAEB(audio-only), 19 tasks | mean **0.454**; leads the board on IEMOCAP speaker-gender (0.938) |
-| Non-audio preservation | text, image, video outputs bit-for-bit the base model's |
-| Embedding dimension | 2048 (Matryoshka: 64, 128, 256, 512, 1024, 1536, 2048) |
+| Supported tasks | `retrieval` (all modality pairs), `zero-shot classification` |
+| Max input | 254 text tokens · 30 s audio per window (up to 8 windows) |
+| Embedding dimension | 2048 |
+| Matryoshka dimensions | 64, 128, 256, 512, 1024, 1536, 2048 |
+| Pooling strategy | Last-token pooling |
 | Base model | Qwen/Qwen3-VL-Embedding-2B (byte-frozen) |
 | Audio tower | Qwen/Qwen2.5-Omni-7B audio encoder (frozen) |
+| Trained components | FusionResampler 16.4M + 28× gated adapters 44.2M |
+| Distribution | ~250 MB trained components; frozen towers download from their original repos |
 
-## Evaluation
+## Training and Evaluation
 
-<details open>
-  <summary>AudioCaps retrieval (883-clip test, min-rank over references)</summary>
+Contrastive training (InfoNCE over the Matryoshka ladder, symmetric) against the
+frozen base's native chat-template text embeddings: 518,183 audio–caption pairs from
+six sources (73,716 clips with content-free metadata excluded), a full-corpus
+frozen-text negative bank, soft labels 0.3, false-negative masking 0.98, bf16, 3,900
+steps at effective batch 1,024, then a 400-step in-domain fine-tune on the AudioCaps
+train split. All evaluation-set audio (Clotho, ESC-50, UrbanSound8K, VGGSound,
+AudioCaps test/val) is excluded from training by ID blacklists at ingestion. A
+technical report is in preparation.
 
-| Model | a2t R@10 | t2a R@10 |
-| --- | --- | --- |
-| fusion-embedding-2 v0.2 | 0.743 | 0.775 |
-| **v0.3-preview** | **0.785** | 0.782 |
-| v0.3-kw-preview | 0.749 | 0.771 |
+All numbers below use the release protocol (bf16 base precision, native chat-template
+text). Bold marks the better value per row/column.
 
-The v0.3 default improves audio-to-text retrieval by 4.2 points over v0.2.
+<p align="center">
+<img src="assets/fe_positioning.png" alt="Positioning: VGGSound-696 cross-modal retrieval versus trained parameters; the fusion-embedding family leads unified models on audio-text and leads the emergent audio-image cluster (ImageBind's supervised pair annotated)" width="860px">
+</p>
+
+<details open>
+  <summary><b>Versus fusion-embedding-1 v0.3</b></summary>
+
+| Board / direction | fusion-embedding-1 v0.3 | fusion-embedding-2 (this repo) |
+|---|---|---|
+| AudioCaps A→T R@1 | **0.332** | 0.302 |
+| AudioCaps A→T R@10 | 0.741 | **0.743** |
+| AudioCaps T→A R@1 | 0.280 | **0.292** |
+| AudioCaps T→A R@10 | 0.746 | **0.775** |
+| Clotho (zero-shot) A→T R@1 | **0.135** | 0.127 |
+| Clotho (zero-shot) A→T R@10 | **0.433** | 0.421 |
+| Clotho (zero-shot) T→A R@1 | 0.136 | **0.151** |
+| Clotho (zero-shot) T→A R@10 | 0.460 | **0.482** |
+| VGGSound audio→text R@1 | **0.213** | 0.211 |
+| VGGSound audio→text R@10 | 0.625 | **0.665** |
+| VGGSound text→audio R@1 | 0.213 | **0.266** |
+| VGGSound text→audio R@10 | 0.645 | **0.681** |
+| VGGSound audio→image R@10 (emergent) | **0.407** | 0.392 |
+
+fusion-embedding-2 takes the majority of cells, with its largest gains in the
+text→audio direction (searching audio with a text query) and on the cross-modal
+audio↔text pair. fusion-embedding-1 v0.3 retains the AudioCaps and Clotho A→T R@1
+cells and a ~1.5-point edge on emergent audio→image at this fine-tuned operating
+point; the pre-fine-tune fusion-embedding-2 checkpoint scores 0.443 on that cell — the
+project record — and may be released separately as the emergent-alignment operating
+point.
 
 </details>
 
 <details>
-  <summary>MAEB(audio-only), full 19-task board — v0.3-preview</summary>
-
-Mean 0.4536. Scores are the main metric per task.
-
-| Task | Score |
-| --- | --- |
-| IEMOCAPGender | 0.938 |
-| BeijingOpera | 0.924 |
-| NMSQAPairClassification | 0.794 |
-| GTZANAudioReranking | 0.717 |
-| JamAltArtistA2ARetrieval | 0.687 |
-| GTZANGenre | 0.644 |
-| VoxPopuliLanguageID | 0.616 |
-| MInDS14 | 0.578 |
-| CREMADPairClassification | 0.543 |
-| MridinghamTonic | 0.540 |
-| VoxPopuliAccentPairClassification | 0.509 |
-| CREMA_D | 0.277 |
-| VoxCelebSA | 0.273 |
-| BirdCLEF | 0.184 |
-| CommonLanguageAgeDetection | 0.171 |
-| SIBFLEURS | 0.115 |
-| VoxPopuliGenderClustering | 0.079 |
-| VehicleSoundClustering | 0.025 |
-| CREMA_DClustering | 0.006 |
-
-IEMOCAP speaker-gender classification is the strongest cell relative to the field.
-Clustering tasks are the weakest and are a target for the next release.
+  <summary><b>Cross-modal retrieval — versus unified embedding models</b> (VGGSound-AV, 696 pairs, chance R@10 = 0.014)</summary>
+
+R@10 shown as audio-side → other / other → audio-side:
+
+| Model | audio↔image | audio↔text | text↔image |
+|---|---|---|---|
+| ImageBind-Huge | **0.718 / 0.720** | 0.404 / 0.348 | 0.243 / 0.282 |
+| LanguageBind | 0.365 / 0.415 | 0.547 / 0.331 | 0.221 / 0.283 |
+| Gemini Embedding 2 (API, 2026-07-09) | 0.312 / 0.316 | 0.379 / 0.374 | 0.273 / **0.366** |
+| fusion-embedding-1-2b-preview v0.3 | 0.407 / 0.428 | 0.625 / 0.645 | **0.331** / 0.319 |
+| **fusion-embedding-2-2b-preview** | 0.392 / 0.430 | **0.665 / 0.681** | **0.331** / 0.319 |
+
+ImageBind trains directly on audio–image pairs, so that pair is its supervised
+direction; its audio–text alignment is emergent. LanguageBind trains audio against
+language; its audio↔image is emergent. Both fusion-embedding generations train on
+audio–text only; their audio–image alignment is emergent. All models evaluated with
+identical clips, frames, and scoring, using the released imagebind_huge checkpoint and
+revision-pinned LanguageBind checkpoints. Gemini Embedding 2 is Google's natively
+multimodal embedding API, evaluated at its documented default invocation on the date
+shown; API models may change after that date. fusion-embedding-2's text↔image cells
+are identical to fusion-embedding-1's by construction — text and images never touch
+the trained components — and this is verified: its own readout run reproduces
+fusion-embedding-1 v0.3's text→image scores to four decimal places.
 
 </details>
 
 <details>
-  <summary>Non-audio preservation and other modalities</summary>
+  <summary><b>Audio–text retrieval — versus specialist CLAP models</b></summary>
 
-Text, image, and video outputs are bit-for-bit identical to the frozen
-Qwen3-VL-Embedding-2B base (the adapter gate is closed on those inputs). Their
-benchmark scores are therefore the base model's own, unchanged by anything added for
-audio. Audio-to-image retrieval is emergent: the model is trained on audio-text pairs
-only, with no audio-image supervision.
+Specialist CLAP models fine-tune their text towers on audio captions — the direct
+trade this architecture declines in order to keep one shared space for all four
+modalities. They remain ahead on the audio-caption boards (e.g., AudioCaps T→A R@1:
+M2D-CLAP 41.4 vs 29.2 here); this model family is the strongest option we measured
+when one model must serve text, images, video, and audio together. See the
+[fusion-embedding-1 card](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview)
+for the full CLAP comparison tables; fusion-embedding-2 improves on fusion-embedding-1
+in the text→audio direction on every board.
 
 </details>
 
-## Notes and limitations
-
-- **Emotion recognition regressed** relative to v0.2. On RAVDESS zero-shot emotion the
-  score is 0.21 (v0.2: 0.35). The v0.3 training mix improved spoken-content
-  understanding at the cost of vocal-prosody sensitivity. If speech-emotion is central
-  to your use, evaluate before adopting.
-- One evaluation cell, `CommonLanguageAgeDetection`, derives from Common Voice, which is
-  part of the training data. It scores low (0.171) and does not inflate the reported
-  mean, but the potential overlap is noted for completeness.
-- These are research previews under CC-BY-NC-4.0.
-
 ## Usage
 
 <details>
@@ -176,8 +182,7 @@ from inference import FusionEmbedder
 
 fe = FusionEmbedder.from_pretrained(
     "EximiusLabs/fusion-embedding-2-2b-preview",
-    revision="v0.3-preview",           # default, retrieval-tuned
-    # revision="v0.3-kw-preview",      # keyword-tuned alternative
+    revision="v0.1-preview",   # pin a tag if you build on this model
 )
 
 a = fe.embed_audio("dog.wav")            # audio file or (array, sr=...)
@@ -190,9 +195,11 @@ print((a @ t).item(), (a @ i).item())    # cosine similarities in the shared spa
 t_small = fe.embed_text("a dog barks", dim=256)
 ```
 
-The checkpoint contains the gated adapters and the loader refuses to run without them.
-All inputs use the base model's chat-template format; embedding quality is sensitive to
-this formatting, so use the templates provided by `FusionEmbedder`.
+The checkpoint contains the gated adapters and the loader refuses to run without them —
+an adapter checkpoint can never be silently executed as the first-generation
+architecture. All inputs use the base model's chat-template format; embedding quality
+is sensitive to this formatting, so use the templates provided by `FusionEmbedder`
+rather than constructing your own.
 
 </details>
 
@@ -200,7 +207,8 @@ this formatting, so use the templates provided by `FusionEmbedder`.
   <summary>Cross-modal ranking tip</summary>
 
 When ranking a gallery of one modality against queries of another, per-modality
-mean-centering of the gallery improves cross-modal recall:
+mean-centering of the gallery improves cross-modal recall by roughly two points across
+modality pairs:
 
 ```python
 gallery = FusionEmbedder.center(gallery_embeddings)
@@ -208,25 +216,30 @@ gallery = FusionEmbedder.center(gallery_embeddings)
 
 </details>
 
-## Changelog
+### Deploy as an API on RunPod
+
+One-click deploy the text endpoint from the
+[RunPod Hub](https://www.runpod.io/console/hub/Eximius-Labs/fusion-embedding)
+(serverless, scales to zero when idle). Once it is running, call it:
+
+```bash
+curl -s https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync \
+  -H "Authorization: Bearer $RUNPOD_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"input": {"input": "a dog barks in the distance"}}'
+```
 
-- **v0.3-preview** (default): retrieval-tuned flagship. AudioCaps a2t R@10 0.785
-  (+0.042 over v0.2); MAEB(audio-only) mean 0.454; leads IEMOCAP speaker-gender.
-  RAVDESS emotion regressed to 0.21.
-- **v0.3-kw-preview**: keyword-tuned revision. SpeechCommands zero-shot 0.929;
-  AudioCaps a2t R@10 0.749.
-- **v0.2-preview**: AudioCaps 2.0 fine-tune. a2t R@10 0.743.
-- **v0.1-preview**: first modality-gated-adapter release.
+Returns 1024-d embeddings. Batch with `"input": ["a", "b"]`; truncate dimensions
+with `"dim": 512`.
 
 ## License
 
-Weights (all revisions): **CC-BY-NC-4.0** (research preview). The code is Apache-2.0.
-The frozen base and audio tower retain their original licenses.
+Code is Apache-2.0 ([GitHub](https://github.com/Eximius-Labs/fusion-embedding));
+model weights in this repository are **CC BY-NC 4.0** (research preview). The frozen
+base and audio tower retain their original licenses.
 
 ## Citation
 
-Model:
-
 ```bibtex
 @software{fusion_embedding_2_2026,
   title  = {Fusion Embedding 2: Modality-Gated Deep Adapters for a
@@ -236,16 +249,3 @@ Model:
   url    = {https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview}
 }
 ```
-
-Technical report:
-
-```bibtex
-@article{tonmoy2026fusion,
-  title   = {Fusion Embedding: A Unified Embedding Space for Text, Image,
-             Video, and Audio},
-  author  = {Tonmoy, Abdul Basit and Hoque, Kazi Fardinul and
-             Arham, Md. Shahrier Islam and Luthra, Arman},
-  journal = {arXiv preprint arXiv:2607.18666},
-  year    = {2026}
-}
-```