tomaarsen/multivector-colqwen2-v1.0-hf-docqa-energy

colqwen2-v1.0-hf finetuned on energy document pages

This is a Multi-Vector Encoder model finetuned from vidore/colqwen2-v1.0-hf on the synthetic_doc_qa_energy_train dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

• Model Type: Multi-Vector Encoder
• Base model: vidore/colqwen2-v1.0-hf
• Maximum Sequence Length: 32768 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: maxsim
• Supported Modalities: Text, Image
• Training Dataset:
  • synthetic_doc_qa_energy_train
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Multi-Vector Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'retrieval', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'embeddings'}, 'image': {'method': 'forward', 'method_output_name': 'embeddings'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ColQwen2ForRetrieval'})
  (1): MultiVectorMask({'skiplist_words': [], 'keep_only_token_ids': None})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/multivector-colqwen2-v1.0-hf-docqa-energy")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'What topics are covered in this index?',
]
documents = [
    'https://huggingface.co/tomaarsen/multivector-colqwen2-v1.0-hf-docqa-energy/resolve/main/assets/image_0.jpg',
    'https://huggingface.co/tomaarsen/multivector-colqwen2-v1.0-hf-docqa-energy/resolve/main/assets/image_1.jpg',
    'https://huggingface.co/tomaarsen/multivector-colqwen2-v1.0-hf-docqa-energy/resolve/main/assets/image_2.jpg',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (20, 128) (759, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[15.7523,  8.2611, 11.6049]])

Evaluation

Metrics

Multi Vector Information Retrieval

• Dataset: energy-dev
• Evaluated with MultiVectorInformationRetrievalEvaluator

Metric	Value
maxsim_accuracy@1	0.935
maxsim_accuracy@3	0.9675
maxsim_accuracy@5	0.9725
maxsim_accuracy@10	0.9825
maxsim_precision@1	0.935
maxsim_precision@3	0.3225
maxsim_precision@5	0.1945
maxsim_precision@10	0.0982
maxsim_recall@1	0.935
maxsim_recall@3	0.9675
maxsim_recall@5	0.9725
maxsim_recall@10	0.9825
maxsim_ndcg@10	0.9592
maxsim_mrr@10	0.9518
maxsim_map@100	0.9522

Training Details

Training Dataset

synthetic_doc_qa_energy_train

• Dataset: synthetic_doc_qa_energy_train at 438dd85
• Size: 3,475 training samples
• Columns: query and image
• Approximate statistics based on the first 100 samples:

  	query	image
  type	string	image
  modality	text	image
  details	min: 18 tokens mean: 28.12 tokens max: 48 tokens	min: 932x312 px mean: 1717x2057 px max: 3200x2339 px

• Samples:

  query	image
  What is the objective of the research task related to reactor pressure vessel steels?
  What recommendations does this study make regarding energy policy options?
  What are the typical materials used for the cathode, electrolyte, and anode in conventional solid-state batteries?

• Loss: MultiVectorMultipleNegativesRankingLoss with these parameters:

  {
      "score_metric": "colbert_scores",
      "scale": 1.0,
      "score_mini_batch_size": null,
      "size_average": true,
      "gather_across_devices": false
  }
  

Evaluation Dataset

synthetic_doc_qa_energy_train

• Dataset: synthetic_doc_qa_energy_train at 438dd85
• Size: 400 evaluation samples
• Columns: query and image
• Approximate statistics based on the first 100 samples:

  	query	image
  type	string	image
  modality	text	image
  details	min: 18 tokens mean: 27.27 tokens max: 44 tokens	min: 827x1125 px mean: 1728x2103 px max: 3400x3042 px

• Samples:

  query	image
  What topics are covered in this index?
  What are the different funding sources for projects listed in the table?
  What are the main sections covered in this report?

• Loss: MultiVectorMultipleNegativesRankingLoss with these parameters:

  {
      "score_metric": "colbert_scores",
      "scale": 1.0,
      "score_mini_batch_size": null,
      "size_average": true,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs: 1
• learning_rate: 2e-05
• warmup_steps: 0.05
• bf16: True
• save_only_model: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• per_device_train_batch_size: 8
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.05
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: True
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• max_length: None

Training Logs

Epoch	Step	Training Loss	Validation Loss	energy-dev_maxsim_ndcg@10
-1	-1	-	-	0.9571
0.0115	5	0.0964	-	-
0.0230	10	0.0489	-	-
0.0345	15	0.1147	-	-
0.0460	20	0.0682	-	-
0.0575	25	0.0311	-	-
0.0690	30	0.0563	-	-
0.0805	35	0.0086	-	-
0.0920	40	0.0599	-	-
0.1011	44	-	0.0606	0.9554
0.1034	45	0.0014	-	-
0.1149	50	0.0163	-	-
0.1264	55	0.0684	-	-
0.1379	60	0.0364	-	-
0.1494	65	0.0973	-	-
0.1609	70	0.0744	-	-
0.1724	75	0.0444	-	-
0.1839	80	0.0047	-	-
0.1954	85	0.1064	-	-
0.2023	88	-	0.0516	0.9548
0.2069	90	0.1071	-	-
0.2184	95	0.0783	-	-
0.2299	100	0.0627	-	-
0.2414	105	0.0181	-	-
0.2529	110	0.0073	-	-
0.2644	115	0.0430	-	-
0.2759	120	0.0013	-	-
0.2874	125	0.0500	-	-
0.2989	130	0.0044	-	-
0.3034	132	-	0.0442	0.9548
0.3103	135	0.0891	-	-
0.3218	140	0.0260	-	-
0.3333	145	0.0302	-	-
0.3448	150	0.0229	-	-
0.3563	155	0.1208	-	-
0.3678	160	0.0367	-	-
0.3793	165	0.0361	-	-
0.3908	170	0.0409	-	-
0.4023	175	0.0103	-	-
0.4046	176	-	0.0427	0.9559
0.4138	180	0.0072	-	-
0.4253	185	0.0649	-	-
0.4368	190	0.0405	-	-
0.4483	195	0.0026	-	-
0.4598	200	0.0352	-	-
0.4713	205	0.0342	-	-
0.4828	210	0.0098	-	-
0.4943	215	0.0057	-	-
0.5057	220	0.0235	0.0416	0.9567
0.5172	225	0.0026	-	-
0.5287	230	0.0119	-	-
0.5402	235	0.0013	-	-
0.5517	240	0.0417	-	-
0.5632	245	0.0118	-	-
0.5747	250	0.0060	-	-
0.5862	255	0.0069	-	-
0.5977	260	0.0620	-	-
0.6069	264	-	0.0410	0.9583
0.6092	265	0.0700	-	-
0.6207	270	0.0287	-	-
0.6322	275	0.1266	-	-
0.6437	280	0.0015	-	-
0.6552	285	0.0147	-	-
0.6667	290	0.0145	-	-
0.6782	295	0.0976	-	-
0.6897	300	0.0027	-	-
0.7011	305	0.0341	-	-
0.7080	308	-	0.0404	0.9583
0.7126	310	0.0570	-	-
0.7241	315	0.0302	-	-
0.7356	320	0.0047	-	-
0.7471	325	0.0238	-	-
0.7586	330	0.0514	-	-
0.7701	335	0.0022	-	-
0.7816	340	0.0579	-	-
0.7931	345	0.0030	-	-
0.8046	350	0.0407	-	-
0.8092	352	-	0.0404	0.9577
0.8161	355	0.0363	-	-
0.8276	360	0.0570	-	-
0.8391	365	0.0031	-	-
0.8506	370	0.0603	-	-
0.8621	375	0.0067	-	-
0.8736	380	0.0022	-	-
0.8851	385	0.0129	-	-
0.8966	390	0.0072	-	-
0.9080	395	0.0052	-	-
0.9103	396	-	0.0405	0.9574
0.9195	400	0.0165	-	-
0.9310	405	0.0060	-	-
0.9425	410	0.0020	-	-
0.9540	415	0.0144	-	-
0.9655	420	0.0572	-	-
0.9770	425	0.1479	-	-
0.9885	430	0.0381	-	-
1.0	435	0.0337	0.0405	0.9592
-1	-1	-	-	0.9592

• The bold row denotes the saved checkpoint.

Training Time

• Training: 14.2 minutes
• Evaluation: 24.0 minutes
• Total: 38.2 minutes

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.7.0.dev0
• Transformers: 5.14.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultiVectorMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}