README.md

---
license: cc-by-nc-4.0
language:
  - en
pipeline_tag: feature-extraction
tags:
  - embeddings
  - multimodal
  - audio
  - retrieval
  - matryoshka
  - qwen3-vl
  - adapters
base_model: Qwen/Qwen3-VL-Embedding-2B
---

<p align="center">
  <img src="assets/fusion-embedding-2-banner.png" alt="Fusion Embedding 2 (2B-Preview) — Eximius Labs" width="100%">
</p>

# fusion-embedding-2-2b-preview

<div align="center">

[![Python](https://img.shields.io/badge/python-3.11+-blue.svg)](https://github.com/Eximius-Labs/fusion-embedding) [![PyTorch](https://img.shields.io/badge/PyTorch-2.x-ee4c2c.svg)](https://github.com/Eximius-Labs/fusion-embedding) [![Weights](https://img.shields.io/badge/weights-CC--BY--NC--4.0-green.svg)](#license) [![Status](https://img.shields.io/badge/status-research%20preview%20v0.3-orange.svg)](#) [![Paper](https://img.shields.io/badge/paper-arXiv%3A2607.18666-b31b1b.svg)](https://arxiv.org/abs/2607.18666) [![Code](https://img.shields.io/badge/code-GitHub-black.svg)](https://github.com/Eximius-Labs/fusion-embedding)

[![Deploy on RunPod](https://api.runpod.io/badge/Eximius-Labs/fusion-embedding)](https://www.runpod.io/console/hub/Eximius-Labs/fusion-embedding)

</div>

`fusion-embedding-2-2b-preview` is the second generation of Eximius Labs' unified
multimodal embedding models: **text, images, video, and audio in one vector space**.
Audio is added to a byte-frozen vision-language base through modality-gated deep
adapters, so gaining audio never changes a single existing text, image, or video
vector. For the first-generation architecture see
[fusion-embedding-1-2b-preview](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview)
(that line is final at v0.3).

[GitHub](https://github.com/Eximius-Labs/fusion-embedding) | Technical report: [arXiv:2607.18666](https://arxiv.org/abs/2607.18666)

## Model Overview

<p align="center">
<img src="assets/fe2_model_overview.png" alt="fusion-embedding-2 architecture: frozen Qwen3-VL-Embedding base with modality-gated adapters inside; frozen audio tower and trained FusionResampler on the audio branch; one shared embedding space" width="820px">
</p>

`fusion-embedding-2-2b-preview` embeds all four modalities on a
[Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) base that is
**byte-identical to its original release**: its text, image, and video behaviour, and
its benchmark scores, carry over exactly. Audio is added by training 60.6M parameters
(about 2.3% of the stack): a perceiver-resampler that maps frozen
[Qwen2.5-Omni](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) audio-tower features into
the base's input space, plus **28 gated adapters** (44.2M) that give the frozen
language model in-layer capacity for audio. The adapters run only while audio is being
encoded; every other forward pass returns the frozen layers' output untouched, so the
invariance is bitwise, not approximate. Adding audio never invalidates an index you
have already built.

This v0.3 release ships two revisions of the same model. The default is tuned for
audio-text retrieval; a keyword-tuned revision is available for spoken-command use.

## Versions in this release

| Revision | Use it for | AudioCaps a2t R@10 | SpeechCommands |
| --- | --- | --- | --- |
| **`v0.3-preview`** (default) | general audio-text retrieval, RAG, clustering | **0.785** | 0.894 |
| `v0.3-kw-preview` | spoken-keyword and command retrieval | 0.749 | **0.929** |

The default (`v0.3-preview`) is the recommended model for almost all use. The
keyword-tuned revision trades audio-text retrieval quality for higher zero-shot
keyword-spotting accuracy; use it only when spoken-command matching is the priority,
and use the default otherwise.

## Highlights (v0.3-preview)

| Feature | Value |
| --- | --- |
| Parameters | ~2.06B frozen base + 640M frozen audio tower; **60.6M trained** |
| Modalities | text, image, video, audio |
| AudioCaps 883 (audio-to-text) | R@1 0.333 · R@5 0.646 · R@10 **0.785** · mAP@10 0.255 |
| AudioCaps 883 (text-to-audio) | R@1 0.297 · R@5 0.653 · R@10 0.782 · mAP@10 0.445 |
| MAEB(audio-only), 19 tasks | mean **0.454**; leads the board on IEMOCAP speaker-gender (0.938) |
| Non-audio preservation | text, image, video outputs bit-for-bit the base model's |
| Embedding dimension | 2048 (Matryoshka: 64, 128, 256, 512, 1024, 1536, 2048) |
| Base model | Qwen/Qwen3-VL-Embedding-2B (byte-frozen) |
| Audio tower | Qwen/Qwen2.5-Omni-7B audio encoder (frozen) |

## Evaluation

<details open>
  <summary>AudioCaps retrieval (883-clip test, min-rank over references)</summary>

| Model | a2t R@10 | t2a R@10 |
| --- | --- | --- |
| fusion-embedding-2 v0.2 | 0.743 | 0.775 |
| **v0.3-preview** | **0.785** | 0.782 |
| v0.3-kw-preview | 0.749 | 0.771 |

The v0.3 default improves audio-to-text retrieval by 4.2 points over v0.2.

</details>

<details>
  <summary>MAEB(audio-only), full 19-task board — v0.3-preview</summary>

Mean 0.4536. Scores are the main metric per task.

| Task | Score |
| --- | --- |
| IEMOCAPGender | 0.938 |
| BeijingOpera | 0.924 |
| NMSQAPairClassification | 0.794 |
| GTZANAudioReranking | 0.717 |
| JamAltArtistA2ARetrieval | 0.687 |
| GTZANGenre | 0.644 |
| VoxPopuliLanguageID | 0.616 |
| MInDS14 | 0.578 |
| CREMADPairClassification | 0.543 |
| MridinghamTonic | 0.540 |
| VoxPopuliAccentPairClassification | 0.509 |
| CREMA_D | 0.277 |
| VoxCelebSA | 0.273 |
| BirdCLEF | 0.184 |
| CommonLanguageAgeDetection | 0.171 |
| SIBFLEURS | 0.115 |
| VoxPopuliGenderClustering | 0.079 |
| VehicleSoundClustering | 0.025 |
| CREMA_DClustering | 0.006 |

IEMOCAP speaker-gender classification is the strongest cell relative to the field.
Clustering tasks are the weakest and are a target for the next release.

</details>

<details>
  <summary>Non-audio preservation and other modalities</summary>

Text, image, and video outputs are bit-for-bit identical to the frozen
Qwen3-VL-Embedding-2B base (the adapter gate is closed on those inputs). Their
benchmark scores are therefore the base model's own, unchanged by anything added for
audio. Audio-to-image retrieval is emergent: the model is trained on audio-text pairs
only, with no audio-image supervision.

</details>

## Notes and limitations

- **Emotion recognition regressed** relative to v0.2. On RAVDESS zero-shot emotion the
  score is 0.21 (v0.2: 0.35). The v0.3 training mix improved spoken-content
  understanding at the cost of vocal-prosody sensitivity. If speech-emotion is central
  to your use, evaluate before adopting.
- One evaluation cell, `CommonLanguageAgeDetection`, derives from Common Voice, which is
  part of the training data. It scores low (0.171) and does not inflate the reported
  mean, but the potential overlap is noted for completeness.
- These are research previews under CC-BY-NC-4.0.

## Usage

<details>
  <summary>Requirements</summary>

- `fusion_embedding` package: `pip install git+https://github.com/Eximius-Labs/fusion-embedding`
- `transformers>=4.46`, `torch` (CUDA), `torchvision`, `pillow`, `soundfile`, `librosa`
- ~14 GB GPU memory at bf16

</details>

<details open>
  <summary>via <code>inference.py</code> (this repository)</summary>

```python
from inference import FusionEmbedder

fe = FusionEmbedder.from_pretrained(
    "EximiusLabs/fusion-embedding-2-2b-preview",
    revision="v0.3-preview",           # default, retrieval-tuned
    # revision="v0.3-kw-preview",      # keyword-tuned alternative
)

a = fe.embed_audio("dog.wav")            # audio file or (array, sr=...)
t = fe.embed_text("a dog barks")         # uses the base's native chat template
i = fe.embed_image("dog.jpg")            # PIL image or path

print((a @ t).item(), (a @ i).item())    # cosine similarities in the shared space

# Matryoshka: pass dim= for smaller embeddings (64..2048)
t_small = fe.embed_text("a dog barks", dim=256)
```

The checkpoint contains the gated adapters and the loader refuses to run without them.
All inputs use the base model's chat-template format; embedding quality is sensitive to
this formatting, so use the templates provided by `FusionEmbedder`.

</details>

<details>
  <summary>Cross-modal ranking tip</summary>

When ranking a gallery of one modality against queries of another, per-modality
mean-centering of the gallery improves cross-modal recall:

```python
gallery = FusionEmbedder.center(gallery_embeddings)
```

</details>

## Changelog

- **v0.3-preview** (default): retrieval-tuned flagship. AudioCaps a2t R@10 0.785
  (+0.042 over v0.2); MAEB(audio-only) mean 0.454; leads IEMOCAP speaker-gender.
  RAVDESS emotion regressed to 0.21.
- **v0.3-kw-preview**: keyword-tuned revision. SpeechCommands zero-shot 0.929;
  AudioCaps a2t R@10 0.749.
- **v0.2-preview**: AudioCaps 2.0 fine-tune. a2t R@10 0.743.
- **v0.1-preview**: first modality-gated-adapter release.

## Deploy on RunPod

One-click from the [RunPod Hub](https://www.runpod.io/console/hub/Eximius-Labs/fusion-embedding).

```bash
curl -s https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync \n  -H "Authorization: Bearer $RUNPOD_API_KEY" \n  -H "Content-Type: application/json" \n  -d '{"input": {"text": "a dog on a beach"}}'
```

Swap `text` for `image`, `video`, or `audio` (value = an https URL, data URI, or base64). Audio needs the audio tower: set `FE2_ENABLE_AUDIO=1` on the endpoint (a 24 GB+ GPU). Returns 1024-d embeddings; add `"dim": 512` to truncate.

## License

Weights (all revisions): **CC-BY-NC-4.0** (research preview). The code is Apache-2.0.
The frozen base and audio tower retain their original licenses.

## Citation

Model:

```bibtex
@software{fusion_embedding_2_2026,
  title  = {Fusion Embedding 2: Modality-Gated Deep Adapters for a
            Unified Text, Image, Video, and Audio Embedding Space},
  author = {Tonmoy, Abdul Basit},
  year   = {2026},
  url    = {https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview}
}
```

Technical report:

```bibtex
@article{tonmoy2026fusion,
  title   = {Fusion Embedding: A Unified Embedding Space for Text, Image,
             Video, and Audio},
  author  = {Tonmoy, Abdul Basit and Hoque, Kazi Fardinul and
             Arham, Md. Shahrier Islam and Luthra, Arman},
  journal = {arXiv preprint arXiv:2607.18666},
  year    = {2026}
}
```