Card: clean rebuild - single-row badges + RunPod on next row, fixed frontmatter

README.md

@@ -1,51 +1,25 @@
 ---
-
 license: cc-by-nc-4.0
-
 language:
-
   - en
-
 pipeline_tag: feature-extraction
-
 tags:
-
   - embeddings
-
   - multimodal
-
   - audio
-
   - retrieval
-
   - matryoshka
-
   - qwen3-vl
-
   - adapters
-
 base_model: Qwen/Qwen3-VL-Embedding-2B
-
 ---
 
-
-
 <p align="center">
-
   <img src="assets/fusion-embedding-2-banner.png" alt="Fusion Embedding 2 (2B-Preview) — Eximius Labs" width="100%">
-
 </p>
 
-
-
 # fusion-embedding-2-2b-preview
 
-
-
-<div align="center">
-
-
-
 <div align="center">
 
 [![Python](https://img.shields.io/badge/python-3.11+-blue.svg)](https://github.com/Eximius-Labs/fusion-embedding) [![PyTorch](https://img.shields.io/badge/PyTorch-2.x-ee4c2c.svg)](https://github.com/Eximius-Labs/fusion-embedding) [![Weights](https://img.shields.io/badge/weights-CC--BY--NC--4.0-green.svg)](#license) [![Status](https://img.shields.io/badge/status-research%20preview%20v0.3-orange.svg)](#) [![Paper](https://img.shields.io/badge/paper-arXiv%3A2607.18666-b31b1b.svg)](https://arxiv.org/abs/2607.18666) [![Code](https://img.shields.io/badge/code-GitHub-black.svg)](https://github.com/Eximius-Labs/fusion-embedding)
@@ -54,479 +28,231 @@ base_model: Qwen/Qwen3-VL-Embedding-2B
 
 </div>
 
-
-
-</div>
-
-
-
 `fusion-embedding-2-2b-preview` is the second generation of Eximius Labs' unified
-
 multimodal embedding models: **text, images, video, and audio in one vector space**.
-
 Audio is added to a byte-frozen vision-language base through modality-gated deep
-
 adapters, so gaining audio never changes a single existing text, image, or video
-
 vector. For the first-generation architecture see
-
 [fusion-embedding-1-2b-preview](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview)
-
 (that line is final at v0.3).
 
-
-
 [GitHub](https://github.com/Eximius-Labs/fusion-embedding) | Technical report: [arXiv:2607.18666](https://arxiv.org/abs/2607.18666)
 
-
-
 ## Model Overview
 
-
-
 <p align="center">
-
 <img src="assets/fe2_model_overview.png" alt="fusion-embedding-2 architecture: frozen Qwen3-VL-Embedding base with modality-gated adapters inside; frozen audio tower and trained FusionResampler on the audio branch; one shared embedding space" width="820px">
-
 </p>
 
-
-
 `fusion-embedding-2-2b-preview` embeds all four modalities on a
-
 [Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) base that is
-
 **byte-identical to its original release**: its text, image, and video behaviour, and
-
 its benchmark scores, carry over exactly. Audio is added by training 60.6M parameters
-
 (about 2.3% of the stack): a perceiver-resampler that maps frozen
-
 [Qwen2.5-Omni](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) audio-tower features into
-
 the base's input space, plus **28 gated adapters** (44.2M) that give the frozen
-
 language model in-layer capacity for audio. The adapters run only while audio is being
-
 encoded; every other forward pass returns the frozen layers' output untouched, so the
-
 invariance is bitwise, not approximate. Adding audio never invalidates an index you
-
 have already built.
 
-
-
 This v0.3 release ships two revisions of the same model. The default is tuned for
-
 audio-text retrieval; a keyword-tuned revision is available for spoken-command use.
 
-
-
 ## Versions in this release
 
-
-
 | Revision | Use it for | AudioCaps a2t R@10 | SpeechCommands |
-
 | --- | --- | --- | --- |
-
 | **`v0.3-preview`** (default) | general audio-text retrieval, RAG, clustering | **0.785** | 0.894 |
-
 | `v0.3-kw-preview` | spoken-keyword and command retrieval | 0.749 | **0.929** |
 
-
-
 The default (`v0.3-preview`) is the recommended model for almost all use. The
-
 keyword-tuned revision trades audio-text retrieval quality for higher zero-shot
-
 keyword-spotting accuracy; use it only when spoken-command matching is the priority,
-
 and use the default otherwise.
 
-
-
 ## Highlights (v0.3-preview)
 
-
-
 | Feature | Value |
-
 | --- | --- |
-
 | Parameters | ~2.06B frozen base + 640M frozen audio tower; **60.6M trained** |
-
 | Modalities | text, image, video, audio |
-
 | AudioCaps 883 (audio-to-text) | R@1 0.333 · R@5 0.646 · R@10 **0.785** · mAP@10 0.255 |
-
 | AudioCaps 883 (text-to-audio) | R@1 0.297 · R@5 0.653 · R@10 0.782 · mAP@10 0.445 |
-
 | MAEB(audio-only), 19 tasks | mean **0.454**; leads the board on IEMOCAP speaker-gender (0.938) |
-
 | Non-audio preservation | text, image, video outputs bit-for-bit the base model's |
-
 | Embedding dimension | 2048 (Matryoshka: 64, 128, 256, 512, 1024, 1536, 2048) |
-
 | Base model | Qwen/Qwen3-VL-Embedding-2B (byte-frozen) |
-
 | Audio tower | Qwen/Qwen2.5-Omni-7B audio encoder (frozen) |
 
-
-
 ## Evaluation
 
-
-
 <details open>
-
   <summary>AudioCaps retrieval (883-clip test, min-rank over references)</summary>
 
-
-
 | Model | a2t R@10 | t2a R@10 |
-
 | --- | --- | --- |
-
 | fusion-embedding-2 v0.2 | 0.743 | 0.775 |
-
 | **v0.3-preview** | **0.785** | 0.782 |
-
 | v0.3-kw-preview | 0.749 | 0.771 |
 
-
-
 The v0.3 default improves audio-to-text retrieval by 4.2 points over v0.2.
 
-
-
 </details>
 
-
-
 <details>
-
   <summary>MAEB(audio-only), full 19-task board — v0.3-preview</summary>
 
-
-
 Mean 0.4536. Scores are the main metric per task.
 
-
-
 | Task | Score |
-
 | --- | --- |
-
 | IEMOCAPGender | 0.938 |
-
 | BeijingOpera | 0.924 |
-
 | NMSQAPairClassification | 0.794 |
-
 | GTZANAudioReranking | 0.717 |
-
 | JamAltArtistA2ARetrieval | 0.687 |
-
 | GTZANGenre | 0.644 |
-
 | VoxPopuliLanguageID | 0.616 |
-
 | MInDS14 | 0.578 |
-
 | CREMADPairClassification | 0.543 |
-
 | MridinghamTonic | 0.540 |
-
 | VoxPopuliAccentPairClassification | 0.509 |
-
 | CREMA_D | 0.277 |
-
 | VoxCelebSA | 0.273 |
-
 | BirdCLEF | 0.184 |
-
 | CommonLanguageAgeDetection | 0.171 |
-
 | SIBFLEURS | 0.115 |
-
 | VoxPopuliGenderClustering | 0.079 |
-
 | VehicleSoundClustering | 0.025 |
-
 | CREMA_DClustering | 0.006 |
 
-
-
 IEMOCAP speaker-gender classification is the strongest cell relative to the field.
-
 Clustering tasks are the weakest and are a target for the next release.
 
-
-
 </details>
 
-
-
 <details>
-
   <summary>Non-audio preservation and other modalities</summary>
 
-
-
 Text, image, and video outputs are bit-for-bit identical to the frozen
-
 Qwen3-VL-Embedding-2B base (the adapter gate is closed on those inputs). Their
-
 benchmark scores are therefore the base model's own, unchanged by anything added for
-
 audio. Audio-to-image retrieval is emergent: the model is trained on audio-text pairs
-
 only, with no audio-image supervision.
 
-
-
 </details>
 
-
-
 ## Notes and limitations
 
-
-
 - **Emotion recognition regressed** relative to v0.2. On RAVDESS zero-shot emotion the
-
   score is 0.21 (v0.2: 0.35). The v0.3 training mix improved spoken-content
-
   understanding at the cost of vocal-prosody sensitivity. If speech-emotion is central
-
   to your use, evaluate before adopting.
-
 - One evaluation cell, `CommonLanguageAgeDetection`, derives from Common Voice, which is
-
   part of the training data. It scores low (0.171) and does not inflate the reported
-
   mean, but the potential overlap is noted for completeness.
-
 - These are research previews under CC-BY-NC-4.0.
 
-
-
 ## Usage
 
-
-
 <details>
-
   <summary>Requirements</summary>
 
-
-
 - `fusion_embedding` package: `pip install git+https://github.com/Eximius-Labs/fusion-embedding`
-
 - `transformers>=4.46`, `torch` (CUDA), `torchvision`, `pillow`, `soundfile`, `librosa`
-
 - ~14 GB GPU memory at bf16
 
-
-
 </details>
 
-
-
 <details open>
-
   <summary>via <code>inference.py</code> (this repository)</summary>
 
-
-
 ```python
-
 from inference import FusionEmbedder
 
-
-
 fe = FusionEmbedder.from_pretrained(
-
     "EximiusLabs/fusion-embedding-2-2b-preview",
-
     revision="v0.3-preview",           # default, retrieval-tuned
-
     # revision="v0.3-kw-preview",      # keyword-tuned alternative
-
 )
 
-
-
 a = fe.embed_audio("dog.wav")            # audio file or (array, sr=...)
-
 t = fe.embed_text("a dog barks")         # uses the base's native chat template
-
 i = fe.embed_image("dog.jpg")            # PIL image or path
 
-
-
 print((a @ t).item(), (a @ i).item())    # cosine similarities in the shared space
 
-
-
 # Matryoshka: pass dim= for smaller embeddings (64..2048)
-
 t_small = fe.embed_text("a dog barks", dim=256)
-
 ```
 
-
-
 The checkpoint contains the gated adapters and the loader refuses to run without them.
-
 All inputs use the base model's chat-template format; embedding quality is sensitive to
-
 this formatting, so use the templates provided by `FusionEmbedder`.
 
-
-
 </details>
 
-
-
 <details>
-
   <summary>Cross-modal ranking tip</summary>
 
-
-
 When ranking a gallery of one modality against queries of another, per-modality
-
 mean-centering of the gallery improves cross-modal recall:
 
-
-
 ```python
-
 gallery = FusionEmbedder.center(gallery_embeddings)
-
 ```
 
-
-
 </details>
 
-
-
 ## Changelog
 
-
-
 - **v0.3-preview** (default): retrieval-tuned flagship. AudioCaps a2t R@10 0.785
-
   (+0.042 over v0.2); MAEB(audio-only) mean 0.454; leads IEMOCAP speaker-gender.
-
   RAVDESS emotion regressed to 0.21.
-
 - **v0.3-kw-preview**: keyword-tuned revision. SpeechCommands zero-shot 0.929;
-
   AudioCaps a2t R@10 0.749.
-
 - **v0.2-preview**: AudioCaps 2.0 fine-tune. a2t R@10 0.743.
-
 - **v0.1-preview**: first modality-gated-adapter release.
 
-
-
 ## Deploy on RunPod
 
-
-
-[![Deploy on RunPod](https://api.runpod.io/badge/Eximius-Labs/fusion-embedding)](https://www.runpod.io/console/hub/Eximius-Labs/fusion-embedding)
-
-
-
-One-click deploy the endpoint from the
-
-[RunPod Hub](https://www.runpod.io/console/hub/Eximius-Labs/fusion-embedding) (serverless,
-
-scales to zero when idle). Once it is running, call it:
-
-
+One-click from the [RunPod Hub](https://www.runpod.io/console/hub/Eximius-Labs/fusion-embedding).
 
 ```bash
-
-curl -s https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync   -H "Authorization: Bearer $RUNPOD_API_KEY"   -H "Content-Type: application/json"   -d '{"input": {"text": "a dog on a beach"}}'
-
+curl -s https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync \n  -H "Authorization: Bearer $RUNPOD_API_KEY" \n  -H "Content-Type: application/json" \n  -d '{"input": {"text": "a dog on a beach"}}'
 ```
 
-
-
-The endpoint embeds all four modalities. Swap `text` for `image`, `video`, or `audio`
-
-(the value is an https URL, a data URI, or base64). Audio needs the audio tower, so set
-
-`FE2_ENABLE_AUDIO=1` on the endpoint and use a 24 GB or larger GPU. Returns 1024-d
-
-embeddings; add `"dim": 512` to truncate.
-
-
+Swap `text` for `image`, `video`, or `audio` (value = an https URL, data URI, or base64). Audio needs the audio tower: set `FE2_ENABLE_AUDIO=1` on the endpoint (a 24 GB+ GPU). Returns 1024-d embeddings; add `"dim": 512` to truncate.
 
 ## License
 
-
-
 Weights (all revisions): **CC-BY-NC-4.0** (research preview). The code is Apache-2.0.
-
 The frozen base and audio tower retain their original licenses.
 
-
-
 ## Citation
 
-
-
 Model:
 
-
-
 ```bibtex
-
 @software{fusion_embedding_2_2026,
-
   title  = {Fusion Embedding 2: Modality-Gated Deep Adapters for a
-
             Unified Text, Image, Video, and Audio Embedding Space},
-
   author = {Tonmoy, Abdul Basit},
-
   year   = {2026},
-
   url    = {https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview}
-
 }
-
 ```
 
-
-
 Technical report:
 
-
-
 ```bibtex
-
 @article{tonmoy2026fusion,
-
   title   = {Fusion Embedding: A Unified Embedding Space for Text, Image,
-
              Video, and Audio},
-
   author  = {Tonmoy, Abdul Basit and Hoque, Kazi Fardinul and
-
              Arham, Md. Shahrier Islam and Luthra, Arman},
-
   journal = {arXiv preprint arXiv:2607.18666},
-
   year    = {2026}
-
 }
-
 ```
-