tomaarsen/multivector-ModernBERT-base-msmarco-peft-no-query-expansion

ColBERT ModernBERT-base adapter trained on MS MARCO triplets

This is a Multi-Vector Encoder model finetuned from answerdotai/ModernBERT-base on the msmarco-bm25 dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

• Model Type: Multi-Vector Encoder
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: maxsim
• Supported Modality: Text
• Training Dataset:
  • msmarco-bm25
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Multi-Vector Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (2): MultiVectorMask({'skiplist_words': [], 'keep_only_token_ids': None})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/multivector-ModernBERT-base-msmarco-peft")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'what does marijuana cost per joint',
]
documents = [
    'With a 50-50 ratio of marijuana to tobacco, the cost of producing a pack of 20 pre-rolled joints could be brought down to just a little more than $20â\x80\x94so a $40 pack at the store. It isnâ\x80\x99t as easy as it seems, though. The government has a vested interest in producing income from the selling of marijuana.nother solution: mix the marijuana with tobacco. If marijuana cigarettes were to be mixed with tobacco, at a 50-50 ratio, it would bring the cost down significantly. Many tobacco farmers will wholesale a pound of their product for less than $2.',
    'What does a dime,dub,eigth,quarter,and a zip of marijuana look like and cost?',
    'In January of 1980, residents decided to incorporate by an overwhelming margin. The Town of Farragut was incorporated on January 16, 1980, with the first board of Mayor and Alderman elected on April 1, 1980.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (8, 128) (129, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[7.3787, 4.1125, 1.2479]])

Evaluation

Metrics

Multi Vector Information Retrieval

• Datasets: NanoMSMARCO, NanoNQ, NanoFiQA2018, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with MultiVectorInformationRetrievalEvaluator

Metric	NanoMSMARCO	NanoNQ	NanoFiQA2018	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoHotpotQA	NanoNFCorpus	NanoQuoraRetrieval	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
maxsim_accuracy@1	0.36	0.24	0.28	0.24	0.62	0.66	0.6	0.34	0.8	0.24	0.08	0.46	0.5102
maxsim_accuracy@3	0.48	0.48	0.36	0.36	0.78	0.86	0.78	0.44	0.88	0.52	0.34	0.6	0.6939
maxsim_accuracy@5	0.54	0.52	0.4	0.42	0.84	0.94	0.84	0.5	0.92	0.62	0.6	0.66	0.7959
maxsim_accuracy@10	0.66	0.68	0.54	0.58	0.84	0.96	0.88	0.56	0.96	0.72	0.7	0.68	0.9592
maxsim_precision@1	0.36	0.24	0.28	0.24	0.62	0.66	0.6	0.34	0.8	0.24	0.08	0.46	0.5102
maxsim_precision@3	0.16	0.16	0.1533	0.1267	0.5067	0.2933	0.3067	0.3067	0.3667	0.2467	0.1133	0.22	0.4354
maxsim_precision@5	0.108	0.108	0.112	0.096	0.46	0.192	0.212	0.26	0.24	0.204	0.12	0.144	0.3918
maxsim_precision@10	0.066	0.07	0.076	0.07	0.372	0.098	0.11	0.202	0.13	0.138	0.07	0.078	0.349
maxsim_recall@1	0.36	0.22	0.1675	0.115	0.076	0.6167	0.3	0.0458	0.6907	0.0507	0.08	0.435	0.0385
maxsim_recall@3	0.48	0.44	0.2549	0.1717	0.1345	0.8267	0.46	0.0753	0.8387	0.1527	0.34	0.59	0.1041
maxsim_recall@5	0.54	0.49	0.2767	0.21	0.1809	0.8867	0.53	0.0862	0.8953	0.2087	0.6	0.65	0.158
maxsim_recall@10	0.66	0.63	0.3509	0.2957	0.2442	0.9067	0.55	0.1095	0.9493	0.2827	0.7	0.68	0.2604
maxsim_ndcg@10	0.4944	0.4234	0.2959	0.2404	0.4753	0.7786	0.5334	0.265	0.8671	0.2627	0.3684	0.5754	0.4015
maxsim_mrr@10	0.4429	0.3764	0.3422	0.3295	0.7097	0.7647	0.7102	0.4045	0.8552	0.4024	0.2636	0.542	0.6441
maxsim_map@100	0.4596	0.3641	0.2562	0.1912	0.3332	0.7262	0.4606	0.1239	0.8334	0.2051	0.2759	0.5523	0.3436

Multi Vector Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with MultiVectorNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nq",
          "fiqa2018"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Metric	Value
maxsim_accuracy@1	0.2667
maxsim_accuracy@3	0.38
maxsim_accuracy@5	0.44
maxsim_accuracy@10	0.5733
maxsim_precision@1	0.2667
maxsim_precision@3	0.1356
maxsim_precision@5	0.0973
maxsim_precision@10	0.066
maxsim_recall@1	0.2236
maxsim_recall@3	0.3314
maxsim_recall@5	0.3871
maxsim_recall@10	0.5059
maxsim_ndcg@10	0.3643
maxsim_mrr@10	0.3467
maxsim_map@100	0.322

Multi Vector Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with MultiVectorNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "climatefever",
          "dbpedia",
          "fever",
          "fiqa2018",
          "hotpotqa",
          "msmarco",
          "nfcorpus",
          "nq",
          "quoraretrieval",
          "scidocs",
          "arguana",
          "scifact",
          "touche2020"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Metric	Value
maxsim_accuracy@1	0.4177
maxsim_accuracy@3	0.5826
maxsim_accuracy@5	0.6612
maxsim_accuracy@10	0.7476
maxsim_precision@1	0.4177
maxsim_precision@3	0.2612
maxsim_precision@5	0.2037
maxsim_precision@10	0.1407
maxsim_recall@1	0.2458
maxsim_recall@3	0.3745
maxsim_recall@5	0.4394
maxsim_recall@10	0.5092
maxsim_ndcg@10	0.4601
maxsim_mrr@10	0.5221
maxsim_map@100	0.3943

Training Details

Training Dataset

msmarco-bm25

• Dataset: msmarco-bm25 at ce8a493
• Size: 50,000 training samples
• Columns: query, positive, and negative
• Approximate statistics based on the first 100 samples:

  	query	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 5 tokens mean: 9.2 tokens max: 23 tokens	min: 24 tokens mean: 85.21 tokens max: 234 tokens	min: 24 tokens mean: 78.79 tokens max: 189 tokens

• Samples:

  query	positive	negative
  how many days to renew philippine passport to usa	The United States requires non-citizens to keep a foreign passport that is valid for six months beyond their date of departure. If you are in the United States legally, then you can renew your Philippine passport at the consulate general's office in Los Angeles.	How much does it cost to renew a Philippines passport? Philippine Passport Fees for Renewal is P 950 for 15 working days and 1,200 for 7 working days. This is according to the Department of Freign Affairs website.
  which sexually transmitted diseases can lead to infections inside joint spaces?	Gonorrhea is a sexually transmitted disease (STD) that can infect both men and women. It can cause infections in the genitals, rectum, and throat.It is a very common infection, especially among young people ages 15-24 years.omen with gonorrhea are at risk of developing serious complications from the infection, even if they don’t have any symptoms. Symptoms in women can include: 1 Painful or burning sensation when urinating; 2 Increased vaginal discharge; 3 Vaginal bleeding between periods.	STDs and Infertility. Sexually transmitted diseases, STDs, also called sexually transmitted infections or STIs, can cause immediate, annoying symptoms with long-lasting, serious repercussions. Few people realize that these sexually transmitted diseases can cause damage that may eventually lead to infertility.
  when was eviva amore constructed	Nasher Sculpture Center Press Images Back of the garden, Nasher Sculpture Center; photo by Tim Hursley. Mark di Suvero, Eviva Amore, 2001 in gardens of Nasher Sculpture Center; photo by Tim Hursley. Jaume Plensa, The Long Night (From Ausias March to Vincent Andres Andrés) , estelles, estellés 2007 At Nasher; sculpture center Photo By. tim hursley	Richard Serra, My Curves Are Not Mad, 1987 and Augustus Rodin, Eve, 1881 (cast before 1932) at Nasher Sculpture Center; photo by Tim Hursley. Mark di Suvero, Eviva Amore, 2001 at dusk in gardens of Nasher Sculpture Center; photo by Tim Hursley. Jeremy Strick, Director of the Nasher Sculpture.

• Loss: MultiVectorMultipleNegativesRankingLoss with these parameters:

  {
      "score_metric": "colbert_scores",
      "scale": 1.0,
      "score_mini_batch_size": null,
      "size_average": true,
      "gather_across_devices": false
  }
  

Evaluation Dataset

msmarco-bm25

• Dataset: msmarco-bm25 at ce8a493
• Size: 1,000 evaluation samples
• Columns: query, positive, and negative
• Approximate statistics based on the first 100 samples:

  	query	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 4 tokens mean: 9.15 tokens max: 19 tokens	min: 39 tokens mean: 82.03 tokens max: 199 tokens	min: 27 tokens mean: 81.62 tokens max: 157 tokens

• Samples:

  query	positive	negative
  what is chor means	• CHORE (noun) The noun CHORE has 1 sense: 1. a specific piece of work required to be done as a duty or for a specific fee. Familiarity information: CHORE used as a noun is very rare.	Any two different languages and not just English and other language. Example 1. Chore (pronounced as cHor) means 'a routine task' in English language. Whereas Chor {चोर} (also pronounced as CHor) means a thief or a burglar in both Hindi and Marathi language.
  how is gravity measured	The gravity of Earth, which is denoted by g, refers to the acceleration that the Earth imparts to objects on or near its surface due to gravity. In SI units this acceleration is measured in metres per second squared (in symbols, m/s2 or m·s−2) or equivalently in newtons per kilogram (N/kg or N·kg−1).	When the wort is first added to the yeast, the specific gravity of the mixture is measured. Later, the specific gravity may be measured again to determine how much alcohol is in the beer, and to know when to stop the fermentation.hen the wort is first added to the yeast, the specific gravity of the mixture is measured. Later, the specific gravity may be measured again to determine how much alcohol is in the beer, and to know when to stop the fermentation.
  salary of doctor during fellowship	Average fellowship salary and wage. The median expected salary for a Fellowship physician in the United States averages to about $150,353 per annum and an average hourly wage is around $20 per hour. fellowship physician in USA receives an average yearly salary ranging from between $34,225 – $59,542. In addition, a yearly bonus of around $4,888 will be included as part of the annual salary package.	Medical Fellowship Salary. Medical Fellowship average salary is $55,008, median salary is $- with a salary range from $- to $-.Medical Fellowship salaries are collected from government agencies and companies. Each salary is associated with a real job position.Medical Fellowship salary statistics is not exclusive and is for reference only.They are presented as is and updated regularly.edical Fellowship salaries are collected from government agencies and companies. Each salary is associated with a real job position. Medical Fellowship salary statistics is not exclusive and is for reference only. They are presented as is and updated regularly.

• Loss: MultiVectorMultipleNegativesRankingLoss with these parameters:

  {
      "score_metric": "colbert_scores",
      "scale": 1.0,
      "score_mini_batch_size": null,
      "size_average": true,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 1
• learning_rate: 3e-05
• warmup_steps: 0.05
• bf16: True
• per_device_eval_batch_size: 16
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 16
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 3e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.05
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 16
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_maxsim_ndcg@10	NanoNQ_maxsim_ndcg@10	NanoFiQA2018_maxsim_ndcg@10	NanoBEIR_mean_maxsim_ndcg@10	NanoClimateFEVER_maxsim_ndcg@10	NanoDBPedia_maxsim_ndcg@10	NanoFEVER_maxsim_ndcg@10	NanoHotpotQA_maxsim_ndcg@10	NanoNFCorpus_maxsim_ndcg@10	NanoQuoraRetrieval_maxsim_ndcg@10	NanoSCIDOCS_maxsim_ndcg@10	NanoArguAna_maxsim_ndcg@10	NanoSciFact_maxsim_ndcg@10	NanoTouche2020_maxsim_ndcg@10
-1	-1	-	-	0.1622	0.1676	0.1806	0.1701	-	-	-	-	-	-	-	-	-	-
0.0102	32	3.1388	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0205	64	3.0199	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0307	96	2.6161	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0410	128	1.9225	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0512	160	1.4234	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0614	192	1.3101	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0717	224	1.2589	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0819	256	1.1922	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0922	288	1.1277	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1002	313	-	1.0385	0.4334	0.3820	0.3084	0.3746	-	-	-	-	-	-	-	-	-	-
0.1024	320	1.0804	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1126	352	1.0462	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1229	384	0.9987	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1331	416	0.9750	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1434	448	0.9402	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1536	480	0.9562	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1638	512	0.9837	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1741	544	0.8821	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1843	576	0.8749	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1946	608	0.8840	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2003	626	-	0.8499	0.3980	0.3747	0.3384	0.3703	-	-	-	-	-	-	-	-	-	-
0.2048	640	0.8558	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2150	672	0.8557	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2253	704	0.8472	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2355	736	0.8419	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2458	768	0.7444	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.256	800	0.7030	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2662	832	0.6964	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2765	864	0.7182	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2867	896	0.7058	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2970	928	0.6766	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3005	939	-	0.6826	0.3959	0.3808	0.2961	0.3576	-	-	-	-	-	-	-	-	-	-
0.3072	960	0.7021	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3174	992	0.6570	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3277	1024	0.6808	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3379	1056	0.6907	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3482	1088	0.6609	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3584	1120	0.6528	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3686	1152	0.6631	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3789	1184	0.6654	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3891	1216	0.6284	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3994	1248	0.6762	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4006	1252	-	0.6361	0.3756	0.3687	0.3029	0.3491	-	-	-	-	-	-	-	-	-	-
0.4096	1280	0.6682	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4198	1312	0.6325	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4301	1344	0.6488	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4403	1376	0.6196	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4506	1408	0.6216	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4608	1440	0.6220	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4710	1472	0.6139	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4813	1504	0.6488	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4915	1536	0.6492	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5008	1565	-	0.6224	0.4053	0.3853	0.2963	0.3623	-	-	-	-	-	-	-	-	-	-
0.5018	1568	0.6268	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.512	1600	0.6198	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5222	1632	0.6044	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5325	1664	0.6817	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5427	1696	0.6618	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5530	1728	0.6119	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5632	1760	0.6654	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5734	1792	0.5773	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5837	1824	0.5980	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5939	1856	0.6382	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6010	1878	-	0.6125	0.3983	0.3761	0.3120	0.3621	-	-	-	-	-	-	-	-	-	-
0.6042	1888	0.5787	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6144	1920	0.5919	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6246	1952	0.5871	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6349	1984	0.6255	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6451	2016	0.5961	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6554	2048	0.5441	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6656	2080	0.6188	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6758	2112	0.5860	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6861	2144	0.5706	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6963	2176	0.5720	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7011	2191	-	0.5954	0.4944	0.4234	0.2959	0.4046	-	-	-	-	-	-	-	-	-	-
0.7066	2208	0.6503	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7168	2240	0.5401	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7270	2272	0.6104	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7373	2304	0.5852	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7475	2336	0.5631	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7578	2368	0.5845	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.768	2400	0.5561	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7782	2432	0.6066	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7885	2464	0.6072	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7987	2496	0.5782	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8013	2504	-	0.5808	0.3923	0.4085	0.3019	0.3676	-	-	-	-	-	-	-	-	-	-
0.8090	2528	0.6032	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8192	2560	0.5813	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8294	2592	0.5351	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8397	2624	0.5818	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8499	2656	0.5505	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8602	2688	0.5446	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8704	2720	0.5842	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8806	2752	0.5545	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8909	2784	0.5430	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9011	2816	0.5418	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9014	2817	-	0.5728	0.3911	0.3852	0.2770	0.3511	-	-	-	-	-	-	-	-	-	-
0.9114	2848	0.5551	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9216	2880	0.5842	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9318	2912	0.5949	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9421	2944	0.5717	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9523	2976	0.5669	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9626	3008	0.5380	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9728	3040	0.5454	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9830	3072	0.5609	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9933	3104	0.5555	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.0	3125	-	0.5623	0.4073	0.4027	0.2829	0.3643	-	-	-	-	-	-	-	-	-	-
-1	-1	-	-	0.4944	0.4234	0.2959	0.4601	0.2404	0.4753	0.7786	0.5334	0.2650	0.8671	0.2627	0.3684	0.5754	0.4015

• The bold row denotes the saved checkpoint.

Training Time

• Training: 46.0 minutes
• Evaluation: 17.2 minutes
• Total: 1.1 hours

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 5.7.0.dev0
• Transformers: 5.13.1
• PyTorch: 2.10.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.8.4
• Tokenizers: 0.22.2

Additional Resources

• Sentence Transformers Documentation: the full documentation site, including training, evaluation, and pre-trained model catalogs.
• PyLate: the upstream library whose features were absorbed into Sentence Transformers for multi-vector / late-interaction models.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultiVectorMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}