abtonmoy commited on
Commit
bf1eb1a
Β·
verified Β·
1 Parent(s): abe5432

Model card: architecture diagram; fix bank size to 484K (v0.2)

Browse files
Files changed (1) hide show
  1. README.md +4 -14
README.md CHANGED
@@ -122,24 +122,14 @@ are unaffected by this extension.
122
 
123
  ## Architecture
124
 
125
- ```
126
- β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ Qwen3-VL-Embedding-2B (frozen) ────────┐
127
- text / image / video ───▢ β”‚ native encoding paths, unmodified β”‚
128
- β”‚ β”‚
129
- audio ─▢ Qwen2.5-Omni β”‚ β”‚
130
- audio tower β”‚ β”‚
131
- (frozen) β”‚ β”‚
132
- └─▢ FusionResampler ───┼─▢ audio tokens in the input stream ─▢ LLM ─▢ β”‚
133
- (trained, ~16M) β”‚ last-token pooled embeddingβ”‚
134
- β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
135
- ```
136
 
137
  A perceiver-resampler (width 384, 64 latent queries) translates frozen audio-tower frames
138
  into the base model's input embedding space; its outputs occupy placeholder positions in
139
  the input stream, mirroring the base model's image-token mechanism. Training is
140
- contrastive (InfoNCE over the Matryoshka ladder, symmetric, with a 131K-caption
141
- full-corpus negative bank) against the base model's text embeddings in its native
142
- input format.
143
 
144
  **Input formatting.** All inputs use the base model's chat-template format (instruction in
145
  the system turn, content in the user turn, last-token pooling). Embedding quality is
 
122
 
123
  ## Architecture
124
 
125
+ ![Fusion Embedding architecture: frozen Qwen3-VL-Embedding-2B base and frozen Qwen2.5-Omni audio tower; only the FusionResampler is trained](assets/architecture.png)
 
 
 
 
 
 
 
 
 
 
126
 
127
  A perceiver-resampler (width 384, 64 latent queries) translates frozen audio-tower frames
128
  into the base model's input embedding space; its outputs occupy placeholder positions in
129
  the input stream, mirroring the base model's image-token mechanism. Training is
130
+ contrastive (InfoNCE over the Matryoshka ladder, symmetric, with a full-corpus
131
+ frozen-text negative bank β€” 484K captions at v0.2) against the base model's text
132
+ embeddings in its native input format.
133
 
134
  **Input formatting.** All inputs use the base model's chat-template format (instruction in
135
  the system turn, content in the user turn, last-token pooling). Embedding quality is