--- tags: - sentence-transformers - multi-vector - colbert - late-interaction - generated_from_trainer - dataset_size:10000 - loss:CachedMultiVectorMultipleNegativesRankingLoss base_model: Qwen/Qwen3-VL-Embedding-2B widget: - text: What are the traditional practices of indigenous peoples in managing their lands and territories? - text: How does the tracking and commission distribution work in affiliate marketing when a user makes a purchase? - text: How many climate adaptation activities by Indigenous peoples are identified? - text: What is the founding novel of the cyberpunk subgenre? - text: Where can observable equity security prices be found, as per ASC 321-10? pipeline_tag: feature-extraction library_name: sentence-transformers metrics: - maxsim_accuracy@1 - maxsim_accuracy@3 - maxsim_accuracy@5 - maxsim_accuracy@10 - maxsim_precision@1 - maxsim_precision@3 - maxsim_precision@5 - maxsim_precision@10 - maxsim_recall@1 - maxsim_recall@3 - maxsim_recall@5 - maxsim_recall@10 - maxsim_ndcg@10 - maxsim_mrr@10 - maxsim_map@100 model-index: - name: Multi-Vector Encoder results: - task: type: multi-vector-information-retrieval name: Multi Vector Information Retrieval dataset: name: vdr eval hard type: vdr-eval-hard metrics: - type: maxsim_accuracy@1 value: 0.8466666666666667 name: Maxsim Accuracy@1 - type: maxsim_accuracy@3 value: 0.9566666666666667 name: Maxsim Accuracy@3 - type: maxsim_accuracy@5 value: 0.98 name: Maxsim Accuracy@5 - type: maxsim_accuracy@10 value: 0.9966666666666667 name: Maxsim Accuracy@10 - type: maxsim_precision@1 value: 0.8466666666666667 name: Maxsim Precision@1 - type: maxsim_precision@3 value: 0.31888888888888883 name: Maxsim Precision@3 - type: maxsim_precision@5 value: 0.19599999999999998 name: Maxsim Precision@5 - type: maxsim_precision@10 value: 0.09966666666666665 name: Maxsim Precision@10 - type: maxsim_recall@1 value: 0.8466666666666667 name: Maxsim Recall@1 - type: maxsim_recall@3 value: 0.9566666666666667 name: Maxsim Recall@3 - type: maxsim_recall@5 value: 0.98 name: Maxsim Recall@5 - type: maxsim_recall@10 value: 0.9966666666666667 name: Maxsim Recall@10 - type: maxsim_ndcg@10 value: 0.9277675716215076 name: Maxsim Ndcg@10 - type: maxsim_mrr@10 value: 0.9049907407407407 name: Maxsim Mrr@10 - type: maxsim_map@100 value: 0.905293771043771 name: Maxsim Map@100 --- # Multi-Vector Encoder This is a [Multi-Vector Encoder](https://www.sbert.net/docs/multi_vector_encoder/usage/usage.html) model finetuned from [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) on the llamaindex-vdr-en-train-preprocessed dataset using the [sentence-transformers](https://www.SBERT.net) library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction. ## Model Details ### Model Description - **Model Type:** Multi-Vector Encoder - **Base model:** [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) - **Maximum Sequence Length:** 262144 tokens - **Output Dimensionality:** 128 dimensions - **Similarity Function:** maxsim - **Supported Modalities:** Text, Image, Video, Message - **Training Dataset:** - llamaindex-vdr-en-train-preprocessed ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Documentation:** [Multi-Vector Encoder Documentation](https://www.sbert.net/docs/multi_vector_encoder/usage/usage.html) - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) - **Hugging Face:** [Multi-Vector Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=multi-vector) ### Full Model Architecture ``` MultiVectorEncoder( (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}, 'image': {'method': 'forward', 'method_output_name': 'last_hidden_state'}, 'video': {'method': 'forward', 'method_output_name': 'last_hidden_state'}, 'message': {'method': 'forward', 'method_output_name': 'last_hidden_state', 'format': 'structured'}}, 'module_output_name': 'token_embeddings', 'processing_kwargs': {'chat_template': {'add_generation_prompt': True}}, 'unpad_inputs': False, 'architecture': 'Qwen3VLModel'}) (1): Dense({'in_features': 2048, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'}) (2): MultiVectorMask({'skiplist_words': [], 'keep_only_token_ids': None}) (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import MultiVectorEncoder # Download from the 🤗 Hub model = MultiVectorEncoder("tomaarsen/ColQwen3-VL-Embedding-2B-vdr") # Run inference: each input becomes a sequence of per-token vectors (variable length). queries = [ 'What are the new anthropological perspectives on development as discussed by Quarles Van Ufford and Giri in 2003?', ] documents = [ 'https://huggingface.co/tomaarsen/ColQwen3-VL-Embedding-2B-vdr/resolve/main/assets/image_0.jpg', 'https://huggingface.co/tomaarsen/ColQwen3-VL-Embedding-2B-vdr/resolve/main/assets/image_1.jpg', 'https://huggingface.co/tomaarsen/ColQwen3-VL-Embedding-2B-vdr/resolve/main/assets/image_2.jpg', ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) print(query_embeddings[0].shape, document_embeddings[0].shape) # (48, 128) (352, 128) # Get the MaxSim similarity scores similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) # tensor([[37.0712, 33.5348, 23.2705]]) ``` ## Evaluation ### Metrics #### Multi Vector Information Retrieval * Dataset: `vdr-eval-hard` * Evaluated with [MultiVectorInformationRetrievalEvaluator](https://sbert.net/docs/package_reference/multi_vector_encoder/evaluation.html#sentence_transformers.multi_vector_encoder.evaluation.MultiVectorInformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | maxsim_accuracy@1 | 0.8467 | | maxsim_accuracy@3 | 0.9567 | | maxsim_accuracy@5 | 0.98 | | maxsim_accuracy@10 | 0.9967 | | maxsim_precision@1 | 0.8467 | | maxsim_precision@3 | 0.3189 | | maxsim_precision@5 | 0.196 | | maxsim_precision@10 | 0.0997 | | maxsim_recall@1 | 0.8467 | | maxsim_recall@3 | 0.9567 | | maxsim_recall@5 | 0.98 | | maxsim_recall@10 | 0.9967 | | **maxsim_ndcg@10** | **0.9278** | | maxsim_mrr@10 | 0.905 | | maxsim_map@100 | 0.9053 | ## Training Details ### Training Dataset #### llamaindex-vdr-en-train-preprocessed * Dataset: llamaindex-vdr-en-train-preprocessed * Size: 10,000 training samples * Columns: query, image, and negative_0 * Approximate statistics based on the first 100 samples: | | query | image | negative_0 | |:---------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------| | type | string | image | image | | modality | text | image | image | | details | | | | * Samples: | query | image | negative_0 | |:-----------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------|:---------------------------------------------------| | What are the new anthropological perspectives on development as discussed by Quarles Van Ufford and Giri in 2003? | | | | What are the three main positions anthropologists have taken in relation to development, as discussed by David Lewis? | | | | Who are the three sisters known as the Fates in Greek mythology? | | | * Loss: [CachedMultiVectorMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/multi_vector_encoder/losses.html#cachedmultivectormultiplenegativesrankingloss) with these parameters: ```json { "score_metric": "colbert_scores", "mini_batch_size": 1, "mini_batch_num_tokens": null, "score_mini_batch_size": 1, "scale": 1.0, "size_average": true, "gather_across_devices": false } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `per_device_train_batch_size`: 64 - `num_train_epochs`: 1 - `learning_rate`: 2e-05 - `warmup_steps`: 0.1 - `max_grad_norm`: 30.0 - `bf16`: True - `learning_rate_mapping`: {'^(model\\.)?1\\.linear': 0.0002} #### All Hyperparameters
Click to expand - `per_device_train_batch_size`: 64 - `num_train_epochs`: 1 - `max_steps`: -1 - `learning_rate`: 2e-05 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: None - `warmup_steps`: 0.1 - `optim`: adamw_torch_fused - `optim_args`: None - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `optim_target_modules`: None - `gradient_accumulation_steps`: 1 - `average_tokens_across_devices`: True - `max_grad_norm`: 30.0 - `label_smoothing_factor`: 0.0 - `bf16`: True - `fp16`: False - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `use_liger_kernel`: False - `liger_kernel_config`: None - `use_cache`: False - `neftune_noise_alpha`: None - `torch_empty_cache_steps`: None - `auto_find_batch_size`: False - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `include_num_input_tokens_seen`: no - `log_level`: passive - `log_level_replica`: warning - `disable_tqdm`: False - `project`: huggingface - `trackio_space_id`: None - `trackio_bucket_id`: None - `trackio_static_space_id`: None - `per_device_eval_batch_size`: 8 - `prediction_loss_only`: True - `eval_on_start`: False - `eval_do_concat_batches`: True - `eval_use_gather_object`: False - `eval_accumulation_steps`: None - `include_for_metrics`: [] - `batch_eval_metrics`: False - `save_only_model`: False - `save_on_each_node`: False - `enable_jit_checkpoint`: False - `push_to_hub`: False - `hub_private_repo`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_always_push`: False - `hub_revision`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `restore_callback_states_from_checkpoint`: False - `full_determinism`: False - `seed`: 42 - `data_seed`: None - `use_cpu`: False - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `parallelism_config`: None - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `dataloader_prefetch_factor`: None - `remove_unused_columns`: True - `label_names`: None - `train_sampling_strategy`: random - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `ddp_static_graph`: None - `ddp_backend`: None - `ddp_timeout`: 1800 - `fsdp`: None - `fsdp_config`: None - `deepspeed`: None - `debug`: [] - `skip_memory_metrics`: True - `do_predict`: False - `resume_from_checkpoint`: None - `warmup_ratio`: None - `local_rank`: -1 - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {'^(model\\.)?1\\.linear': 0.0002} - `max_length`: None
### Training Logs | Epoch | Step | Training Loss | vdr-eval-hard_maxsim_ndcg@10 | |:------:|:----:|:-------------:|:----------------------------:| | -1 | -1 | - | 0.8244 | | 0.0510 | 8 | 2.2737 | - | | 0.1019 | 16 | 0.6985 | 0.9132 | | 0.1529 | 24 | 0.3765 | - | | 0.2038 | 32 | 0.3179 | 0.9133 | | 0.2548 | 40 | 0.2909 | - | | 0.3057 | 48 | 0.2984 | 0.9184 | | 0.3567 | 56 | 0.2649 | - | | 0.4076 | 64 | 0.2771 | 0.9232 | | 0.4586 | 72 | 0.2404 | - | | 0.5096 | 80 | 0.2456 | 0.9271 | | 0.5605 | 88 | 0.2933 | - | | 0.6115 | 96 | 0.2549 | 0.9224 | | 0.6624 | 104 | 0.2636 | - | | 0.7134 | 112 | 0.2815 | 0.9202 | | 0.7643 | 120 | 0.2829 | - | | 0.8153 | 128 | 0.2592 | 0.9206 | | 0.8662 | 136 | 0.2180 | - | | 0.9172 | 144 | 0.2647 | 0.9206 | | 0.9682 | 152 | 0.2411 | - | | 1.0 | 157 | - | 0.9278 | ### Training Time - **Training**: 2.4 hours - **Evaluation**: 26.8 minutes - **Total**: 2.8 hours ### Framework Versions - Python: 3.12.12 - Sentence Transformers: 5.7.0.dev0 - Transformers: 5.14.1 - PyTorch: 2.13.0+cu130 - Accelerate: 1.14.0 - Datasets: 5.0.0 - Tokenizers: 0.22.2 ## Additional Resources - [Sentence Transformers Documentation](https://www.sbert.net): the full documentation site, including training, evaluation, and pre-trained model catalogs. - [PyLate](https://github.com/lightonai/pylate): the upstream library whose features were absorbed into Sentence Transformers for multi-vector / late-interaction models. ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### CachedMultiVectorMultipleNegativesRankingLoss ```bibtex @misc{gao2021scaling, title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup}, author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan}, year={2021}, eprint={2101.06983}, archivePrefix={arXiv}, primaryClass={cs.LG} } ```