Model card: architecture diagram; fix bank size to 484K (v0.2)
Browse files
README.md
CHANGED
|
@@ -122,24 +122,14 @@ are unaffected by this extension.
|
|
| 122 |
|
| 123 |
## Architecture
|
| 124 |
|
| 125 |
-
|
| 126 |
-
βββββββββββ Qwen3-VL-Embedding-2B (frozen) βββββββββ
|
| 127 |
-
text / image / video ββββΆ β native encoding paths, unmodified β
|
| 128 |
-
β β
|
| 129 |
-
audio ββΆ Qwen2.5-Omni β β
|
| 130 |
-
audio tower β β
|
| 131 |
-
(frozen) β β
|
| 132 |
-
βββΆ FusionResampler ββββΌββΆ audio tokens in the input stream ββΆ LLM ββΆ β
|
| 133 |
-
(trained, ~16M) β last-token pooled embeddingβ
|
| 134 |
-
ββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 135 |
-
```
|
| 136 |
|
| 137 |
A perceiver-resampler (width 384, 64 latent queries) translates frozen audio-tower frames
|
| 138 |
into the base model's input embedding space; its outputs occupy placeholder positions in
|
| 139 |
the input stream, mirroring the base model's image-token mechanism. Training is
|
| 140 |
-
contrastive (InfoNCE over the Matryoshka ladder, symmetric, with a
|
| 141 |
-
|
| 142 |
-
input format.
|
| 143 |
|
| 144 |
**Input formatting.** All inputs use the base model's chat-template format (instruction in
|
| 145 |
the system turn, content in the user turn, last-token pooling). Embedding quality is
|
|
|
|
| 122 |
|
| 123 |
## Architecture
|
| 124 |
|
| 125 |
+

|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 126 |
|
| 127 |
A perceiver-resampler (width 384, 64 latent queries) translates frozen audio-tower frames
|
| 128 |
into the base model's input embedding space; its outputs occupy placeholder positions in
|
| 129 |
the input stream, mirroring the base model's image-token mechanism. Training is
|
| 130 |
+
contrastive (InfoNCE over the Matryoshka ladder, symmetric, with a full-corpus
|
| 131 |
+
frozen-text negative bank β 484K captions at v0.2) against the base model's text
|
| 132 |
+
embeddings in its native input format.
|
| 133 |
|
| 134 |
**Input formatting.** All inputs use the base model's chat-template format (instruction in
|
| 135 |
the system turn, content in the user turn, last-token pooling). Embedding quality is
|