Add new MultiVectorEncoder model

6ae85e7 verified 3 days ago

110 kB

language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - multi-vector
  - colbert
  - late-interaction
  - generated_from_trainer
  - dataset_size:99000
  - loss:CachedMultiVectorMultipleNegativesRankingLoss
base_model: answerdotai/ModernBERT-base
widget:
  - text: >-
      In the United States, under current patent law, the term of patent,
      provided that maintenance fees are paid on time, are: 1  For applications
      filed on or after June 8, 1995, the patent term is 20 years from the
      filing date of the earliest U.S. or international (PCT) application to
      which priority is claimed (excluding provisional applications).
  - text: >-
      CBC with Differential Blood Test. CBC with Differential Blood Test. A
      Complete Blood Count (CBC) with Differential is a broad screening test
      which can aid in the diagnosis of a variety of conditions and diseases
      such as Anemia, Leukemia, bleeding disorders, and infections.
  - text: what are some vod platforms?
  - text: "Types of antibiotics. There are hundreds of different types of antibiotics, but most of them can be broadly classified into six groups. These are outlined below. Penicillins (such as penicillin and amoxicillin) â\x80\x93 widely used to treat a variety of infections, including skin infections, chest infections and urinary tract infections."
  - text: >-
      territorial sovereignty. Exclusive right of a state to exercise its powers
      within the boundaries of its territory.
datasets:
  - sentence-transformers/msmarco-bm25
pipeline_tag: feature-extraction
library_name: sentence-transformers
metrics:
  - maxsim_accuracy@1
  - maxsim_accuracy@3
  - maxsim_accuracy@5
  - maxsim_accuracy@10
  - maxsim_precision@1
  - maxsim_precision@3
  - maxsim_precision@5
  - maxsim_precision@10
  - maxsim_recall@1
  - maxsim_recall@3
  - maxsim_recall@5
  - maxsim_recall@10
  - maxsim_ndcg@10
  - maxsim_mrr@10
  - maxsim_map@100
model-index:
  - name: ColBERT ModernBERT-base trained on MS MARCO triplets with GradCache
    results:
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoMSMARCO
          type: NanoMSMARCO
        metrics:
          - type: maxsim_accuracy@1
            value: 0.32
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.5
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.58
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.7
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.32
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.16666666666666669
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.11599999999999999
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.32
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.5
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.58
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.7
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.5035525583268671
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.4419126984126984
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.4558177677456216
            name: Maxsim Map@100
          - type: maxsim_accuracy@1
            value: 0.34
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.5
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.6
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.72
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.34
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.16666666666666663
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.12000000000000002
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07200000000000001
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.34
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.5
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.6
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.72
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.5196566650374043
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.4571904761904762
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.4702762122466456
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoNQ
          type: NanoNQ
        metrics:
          - type: maxsim_accuracy@1
            value: 0.32
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.5
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.58
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.66
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.32
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.1733333333333333
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.12
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.068
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.3
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.49
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.56
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.62
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.4702309838483119
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.43272222222222223
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.42919136794674306
            name: Maxsim Map@100
          - type: maxsim_accuracy@1
            value: 0.38
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.5
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.6
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.7
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.38
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.1733333333333333
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.124
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07200000000000001
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.36
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.49
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.58
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.65
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.4984814791818691
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.4622222222222223
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.4559346570489271
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoFiQA2018
          type: NanoFiQA2018
        metrics:
          - type: maxsim_accuracy@1
            value: 0.32
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.38
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.44
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.52
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.32
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.16666666666666663
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.11599999999999999
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.076
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.21335714285714286
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.28488095238095235
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.31288095238095237
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.3684047619047619
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.32790029976167445
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.3730555555555556
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.2941689356621179
            name: Maxsim Map@100
          - type: maxsim_accuracy@1
            value: 0.3
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.4
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.44
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.52
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.3
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.16666666666666663
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.124
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07800000000000001
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.19335714285714287
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.28488095238095235
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.3257380952380952
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.37240476190476185
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.32253820883537515
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.36407936507936506
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.28501846515826573
            name: Maxsim Map@100
      - task:
          type: multi-vector-nano-beir
          name: Multi Vector Nano BEIR
        dataset:
          name: NanoBEIR mean
          type: NanoBEIR_mean
        metrics:
          - type: maxsim_accuracy@1
            value: 0.32
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.45999999999999996
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.5333333333333333
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.6266666666666666
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.32
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.16888888888888887
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.11733333333333333
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07133333333333335
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.2777857142857143
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.4249603174603174
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.48429365079365083
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.5628015873015872
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.4338946139789512
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.41589682539682543
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.39305935711816087
            name: Maxsim Map@100
          - type: maxsim_accuracy@1
            value: 0.4345368916797488
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.6119309262166405
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.6766718995290424
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.758430141287284
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.4345368916797488
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.2709576138147567
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.2108131868131868
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.14876609105180533
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.2568503828650686
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.39219839167176673
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.4528025716195824
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.526628596165496
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.47744189955965444
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.5363376442151953
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.40664976433960326
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoClimateFEVER
          type: NanoClimateFEVER
        metrics:
          - type: maxsim_accuracy@1
            value: 0.2
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.38
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.46
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.54
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.2
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.12666666666666665
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.09600000000000002
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.064
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.09166666666666666
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.17066666666666666
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.20633333333333334
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.27033333333333337
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.2154158108818191
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.3006904761904762
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.165789222440451
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoDBPedia
          type: NanoDBPedia
        metrics:
          - type: maxsim_accuracy@1
            value: 0.56
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.78
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.82
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.86
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.56
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.5
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.44800000000000006
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.3739999999999999
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.06463297360998158
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.1224040434115415
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.17209776544895988
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.24843171098171568
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.4665833494159522
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.6738571428571427
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.3212853411213698
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoFEVER
          type: NanoFEVER
        metrics:
          - type: maxsim_accuracy@1
            value: 0.58
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.9
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.94
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.98
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.58
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.30666666666666664
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.19199999999999995
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.102
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.5466666666666667
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.8466666666666667
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.8866666666666667
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.9333333333333332
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.7634632906900782
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.7285238095238096
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.6972606027433613
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoHotpotQA
          type: NanoHotpotQA
        metrics:
          - type: maxsim_accuracy@1
            value: 0.68
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.76
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.78
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.84
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.68
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.28
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.19199999999999995
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.10799999999999997
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.34
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.42
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.48
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.54
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.5293806892763252
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.7263888888888889
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.4467598142471342
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoNFCorpus
          type: NanoNFCorpus
        metrics:
          - type: maxsim_accuracy@1
            value: 0.42
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.5
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.54
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.56
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.42
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.33333333333333337
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.29600000000000004
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.23399999999999999
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.027922117605102565
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.09673465072936029
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.11351894804424907
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.1360204373302154
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.30660127324494973
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.4661904761904762
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.14095019453849242
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoQuoraRetrieval
          type: NanoQuoraRetrieval
        metrics:
          - type: maxsim_accuracy@1
            value: 0.78
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.88
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.9
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.94
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.78
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.3666666666666666
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.23199999999999996
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.12399999999999999
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.6806666666666666
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.8353333333333333
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.872
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.9226666666666666
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.8471248754431309
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.8372222222222221
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.8197089910089909
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoSCIDOCS
          type: NanoSCIDOCS
        metrics:
          - type: maxsim_accuracy@1
            value: 0.34
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.54
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.62
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.76
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.34
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.26666666666666666
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.21599999999999997
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.152
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.07166666666666667
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.16466666666666666
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.22166666666666668
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.31066666666666665
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.2973905321869307
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.4649920634920634
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.22650133778044473
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoArguAna
          type: NanoArguAna
        metrics:
          - type: maxsim_accuracy@1
            value: 0.14
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.46
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.62
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.76
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.14
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.15333333333333332
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.12400000000000003
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07600000000000001
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.14
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.46
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.62
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.76
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.422866421857334
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.3170714285714286
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.3249369303386584
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoSciFact
          type: NanoSciFact
        metrics:
          - type: maxsim_accuracy@1
            value: 0.48
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.6
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.64
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.7
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.48
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.22666666666666668
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.14800000000000002
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.08
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.445
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.595
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.64
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.7
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.5831859060929877
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.5505238095238095
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.5527097251437221
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoTouche2020
          type: NanoTouche2020
        metrics:
          - type: maxsim_accuracy@1
            value: 0.4489795918367347
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.7551020408163265
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.8367346938775511
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.9795918367346939
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.4489795918367347
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.45578231292517
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.42857142857142855
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.3979591836734694
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.0374760765069978
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.11222611187778031
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.16841195565659994
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.28231483993475376
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.43405619213135205
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.623436993845157
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.3793154425983796
            name: Maxsim Map@100

ColBERT ModernBERT-base trained on MS MARCO triplets with GradCache

This is a Multi-Vector Encoder model finetuned from answerdotai/ModernBERT-base on the msmarco-bm25 dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

Model Type: Multi-Vector Encoder
Base model: answerdotai/ModernBERT-base
Maximum Sequence Length: 8192 tokens
Output Dimensionality: 128 dimensions
Similarity Function: maxsim
Supported Modality: Text
Training Dataset:
- msmarco-bm25
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Documentation: Multi-Vector Encoder Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (2): MultiVectorMask({'skiplist_words': [], 'keep_only_token_ids': None})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/multivector-ModernBERT-base-msmarco-cached-contrastive")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'what is territorial sovereignty',
]
documents = [
    'territorial sovereignty. Exclusive right of a state to exercise its powers within the boundaries of its territory.',
    'Territorial preservation, as Agnew explains is merely one aspect of a states territorial integrity (2005). The lack of territorial sovereignty is often a key characteristic of so-called failed states where effective monopoly over the internal means of violence is lost.',
    '1 Active Transportï\x82§ Active Transport requires the cell to use energy, usually in the form of ATP.ï\x82§ Active Transport creates a charge gradient in the cell membrane. 2  For example in the mitochondrion, hydrogen ion pumps pump hydrogen ions into the intermembrane space of the organelle as part of making ATP. 3  15.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (6, 128) (24, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[5.6691, 5.5729, 1.3079]])

Evaluation

Metrics

Multi Vector Information Retrieval

Datasets: NanoMSMARCO, NanoNQ, NanoFiQA2018, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
Evaluated with MultiVectorInformationRetrievalEvaluator

Metric	NanoMSMARCO	NanoNQ	NanoFiQA2018	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoHotpotQA	NanoNFCorpus	NanoQuoraRetrieval	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
maxsim_accuracy@1	0.34	0.38	0.3	0.2	0.56	0.58	0.68	0.42	0.78	0.34	0.14	0.48	0.449
maxsim_accuracy@3	0.5	0.5	0.4	0.38	0.78	0.9	0.76	0.5	0.88	0.54	0.46	0.6	0.7551
maxsim_accuracy@5	0.6	0.6	0.44	0.46	0.82	0.94	0.78	0.54	0.9	0.62	0.62	0.64	0.8367
maxsim_accuracy@10	0.72	0.7	0.52	0.54	0.86	0.98	0.84	0.56	0.94	0.76	0.76	0.7	0.9796
maxsim_precision@1	0.34	0.38	0.3	0.2	0.56	0.58	0.68	0.42	0.78	0.34	0.14	0.48	0.449
maxsim_precision@3	0.1667	0.1733	0.1667	0.1267	0.5	0.3067	0.28	0.3333	0.3667	0.2667	0.1533	0.2267	0.4558
maxsim_precision@5	0.12	0.124	0.124	0.096	0.448	0.192	0.192	0.296	0.232	0.216	0.124	0.148	0.4286
maxsim_precision@10	0.072	0.072	0.078	0.064	0.374	0.102	0.108	0.234	0.124	0.152	0.076	0.08	0.398
maxsim_recall@1	0.34	0.36	0.1934	0.0917	0.0646	0.5467	0.34	0.0279	0.6807	0.0717	0.14	0.445	0.0375
maxsim_recall@3	0.5	0.49	0.2849	0.1707	0.1224	0.8467	0.42	0.0967	0.8353	0.1647	0.46	0.595	0.1122
maxsim_recall@5	0.6	0.58	0.3257	0.2063	0.1721	0.8867	0.48	0.1135	0.872	0.2217	0.62	0.64	0.1684
maxsim_recall@10	0.72	0.65	0.3724	0.2703	0.2484	0.9333	0.54	0.136	0.9227	0.3107	0.76	0.7	0.2823
maxsim_ndcg@10	0.5197	0.4985	0.3225	0.2154	0.4666	0.7635	0.5294	0.3066	0.8471	0.2974	0.4229	0.5832	0.4341
maxsim_mrr@10	0.4572	0.4622	0.3641	0.3007	0.6739	0.7285	0.7264	0.4662	0.8372	0.465	0.3171	0.5505	0.6234
maxsim_map@100	0.4703	0.4559	0.285	0.1658	0.3213	0.6973	0.4468	0.141	0.8197	0.2265	0.3249	0.5527	0.3793

Multi Vector Nano BEIR

Dataset: NanoBEIR_mean

Evaluated with MultiVectorNanoBEIREvaluator with these parameters:

{
    "dataset_names": [
        "msmarco",
        "nq",
        "fiqa2018"
    ],
    "dataset_id": "sentence-transformers/NanoBEIR-en"
}

Metric	Value
maxsim_accuracy@1	0.32
maxsim_accuracy@3	0.46
maxsim_accuracy@5	0.5333
maxsim_accuracy@10	0.6267
maxsim_precision@1	0.32
maxsim_precision@3	0.1689
maxsim_precision@5	0.1173
maxsim_precision@10	0.0713
maxsim_recall@1	0.2778
maxsim_recall@3	0.425
maxsim_recall@5	0.4843
maxsim_recall@10	0.5628
maxsim_ndcg@10	0.4339
maxsim_mrr@10	0.4159
maxsim_map@100	0.3931

Multi Vector Nano BEIR

Dataset: NanoBEIR_mean

Evaluated with MultiVectorNanoBEIREvaluator with these parameters:

{
    "dataset_names": [
        "climatefever",
        "dbpedia",
        "fever",
        "fiqa2018",
        "hotpotqa",
        "msmarco",
        "nfcorpus",
        "nq",
        "quoraretrieval",
        "scidocs",
        "arguana",
        "scifact",
        "touche2020"
    ],
    "dataset_id": "sentence-transformers/NanoBEIR-en"
}

Metric	Value
maxsim_accuracy@1	0.4345
maxsim_accuracy@3	0.6119
maxsim_accuracy@5	0.6767
maxsim_accuracy@10	0.7584
maxsim_precision@1	0.4345
maxsim_precision@3	0.271
maxsim_precision@5	0.2108
maxsim_precision@10	0.1488
maxsim_recall@1	0.2569
maxsim_recall@3	0.3922
maxsim_recall@5	0.4528
maxsim_recall@10	0.5266
maxsim_ndcg@10	0.4774
maxsim_mrr@10	0.5363
maxsim_map@100	0.4066

Training Details

Training Dataset

msmarco-bm25

Dataset: msmarco-bm25 at ce8a493
Size: 99,000 training samples
Columns: query, positive, and negative

Approximate statistics based on the first 100 samples:

	query	positive	negative
type	string	string	string
modality	text	text	text
details	min: 5 tokens mean: 9.4 tokens max: 18 tokens	min: 28 tokens mean: 83.3 tokens max: 197 tokens	min: 24 tokens mean: 76.0 tokens max: 226 tokens

Samples:

query	positive	negative
`what is an agate made of`	`Agate is the name given to a group of silicate minerals that are made up primarily of chalcedony. Chalcedony is a member of the quartz family of minerals.`	`What is an agate, though, and what are the properties and feng shui meaning of agate? Let's find out. WHAT IS THE MEANING OF AGATE? As a form of chalcedony (type of quartz) agate exhibits a variety of colours , shapes, as well as an often present gentle iridescence.`
`what is the socratic method?`	`The Socratic Learning Method (SLM) is a constructivist learning approach consisting of four key. steps: eliciting relevant preconceptions, clarifying preconceptions, testing ones own. hypotheses or encountered propositions, and deciding whether to accept the hypotheses or. propositions.`	`The Socratic Learning Method and the Inquiry-Based Learning Method. Since the Socratic dialogues are among the earliest documented instances of learning through. inquiry, it is reasonable to argue that what is now known as inquiry-based learning can trace its. origin to the Socratic Learning Method.`
`what ditto means`	`â¢ DITTO (noun). The noun DITTO has 1 sense: 1. a mark used to indicate the word above it should be repeated. Familiarity information: DITTO used as a noun is very rare. â¢ DITTO (verb). The verb DITTO has 1 sense: 1. repeat an action or statement. Familiarity information: DITTO used as a verb is very rare.`	`Valerie Hill 0 I had texted one of my Auntie's and she sent the word Ditto back to me I was like what do that mean. Now I know!!! Acronym. DITTO in text means the same, or me too, or I agree.`

Loss: CachedMultiVectorMultipleNegativesRankingLoss with these parameters:

{
    "score_metric": "colbert_scores",
    "mini_batch_size": 32,
    "mini_batch_num_tokens": null,
    "score_mini_batch_size": 32,
    "scale": 1.0,
    "size_average": true,
    "gather_across_devices": false
}

Evaluation Dataset

msmarco-bm25

Dataset: msmarco-bm25 at ce8a493
Size: 1,000 evaluation samples
Columns: query, positive, and negative

Approximate statistics based on the first 100 samples:

	query	positive	negative
type	string	string	string
modality	text	text	text
details	min: 4 tokens mean: 9.18 tokens max: 21 tokens	min: 22 tokens mean: 76.76 tokens max: 210 tokens	min: 30 tokens mean: 84.36 tokens max: 210 tokens

Samples:

query	positive	negative
`expected return is a function of`	`security market line Security market line (SML) is the representation of the capital asset pricing model. It displays the expected rate of return of an individual security as a function of systematic, non-diversifiable risk (its beta). beta Average sensitivity of a security's price to overall securities market prices.`	`A portfolio's expected return is the sum of the weighted average of each asset's expected return. Calculate a portfolio's expected return. To calculate the expected return of a portfolio, you need to know the expected return and weight of each asset in a portfolio.`
`what smoker temperature for barbecue chicken`	`Chickens smoked hot and fast over indirect heat on the grill. But for pulled chicken, I wanted a slightly more intense smokiness to balance the barbecue sauce. I decided to do a side-by-side comparison, cooking one on the grill over indirect heat at around 375Â°F, and one in the smoker at 225Â°F.`	`Cook for an hour and 15 minutes. Check your smoked chicken breasts to make sure the temperature is still holding at about 250 degrees. Flip the chicken breasts and close the lid again. After 30 minutes, check the internal temperature of the chicken with a meat thermometer. You are looking for a temperature of 160 degrees before you can pull them off the smoker. Serve your smoked chicken breasts with a side of barbecue sauce for dipping.`
`in classification of matter what is an element`	`Matter can be in the same phase or in two different phases for this separation to take place. Key Terms. mixture: Something that consists of diverse, non-bonded elements or molecules. element: A chemical substance that is made up of a particular kind of atom and cannot be broken down or transformed by a chemical reaction.`	2. What four elements make up 96% of all living matter? The four elements that make up 96% of all living matter are oxygen, carbon, hydrogen and nitrogen. 3. What is the difference between an essential element and a trace element? An essential element is an element that an organism needs to live a healthy life and reproduce. A trace element is required by an organism in only minute quantities. Section 2 4. Sketch a model of an atom of helium, showing the electrons, protons, neutrons, and atomic nucleus. Neutrons Protons Electrons 5.

Loss: CachedMultiVectorMultipleNegativesRankingLoss with these parameters:

{
    "score_metric": "colbert_scores",
    "mini_batch_size": 32,
    "mini_batch_num_tokens": null,
    "score_mini_batch_size": 32,
    "scale": 1.0,
    "size_average": true,
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 256
num_train_epochs: 1
learning_rate: 3e-05
warmup_steps: 0.05
bf16: True
per_device_eval_batch_size: 32
load_best_model_at_end: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

per_device_train_batch_size: 256
num_train_epochs: 1
max_steps: -1
learning_rate: 3e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 0.05
optim: adamw_torch_fused
optim_args: None
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: True
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: None
trackio_bucket_id: None
trackio_static_space_id: None
per_device_eval_batch_size: 32
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: True
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_static_graph: None
ddp_backend: None
ddp_timeout: 1800
fsdp: None
fsdp_config: None
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}
max_length: None

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_maxsim_ndcg@10	NanoNQ_maxsim_ndcg@10	NanoFiQA2018_maxsim_ndcg@10	NanoBEIR_mean_maxsim_ndcg@10	NanoClimateFEVER_maxsim_ndcg@10	NanoDBPedia_maxsim_ndcg@10	NanoFEVER_maxsim_ndcg@10	NanoHotpotQA_maxsim_ndcg@10	NanoNFCorpus_maxsim_ndcg@10	NanoQuoraRetrieval_maxsim_ndcg@10	NanoSCIDOCS_maxsim_ndcg@10	NanoArguAna_maxsim_ndcg@10	NanoSciFact_maxsim_ndcg@10	NanoTouche2020_maxsim_ndcg@10
-1	-1	-	-	0.1572	0.1851	0.1355	0.1593	-	-	-	-	-	-	-	-	-	-
0.0103	4	5.9065	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0207	8	5.7869	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0310	12	5.5644	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0413	16	5.2117	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0517	20	4.7197	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0620	24	4.0890	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0724	28	3.5701	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0827	32	3.2990	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0930	36	3.0834	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1008	39	-	1.5038	0.3511	0.2964	0.2598	0.3025	-	-	-	-	-	-	-	-	-	-
0.1034	40	2.9494	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1137	44	2.7276	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1240	48	2.4742	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1344	52	2.3246	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1447	56	2.2679	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1550	60	2.1709	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1654	64	2.1077	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1757	68	2.0022	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1860	72	1.9609	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1964	76	1.9054	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2016	78	-	1.0499	0.4417	0.4054	0.2641	0.3704	-	-	-	-	-	-	-	-	-	-
0.2067	80	1.9625	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2171	84	1.9317	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2274	88	1.9280	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2377	92	1.7935	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2481	96	1.8347	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2584	100	1.8208	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2687	104	1.8619	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2791	108	1.8128	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2894	112	1.7879	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2997	116	1.7014	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3023	117	-	0.9225	0.4534	0.4303	0.3673	0.4170	-	-	-	-	-	-	-	-	-	-
0.3101	120	1.6731	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3204	124	1.6961	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3307	128	1.7301	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3411	132	1.6529	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3514	136	1.6703	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3618	140	1.6882	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3721	144	1.6177	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3824	148	1.6462	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3928	152	1.6391	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4031	156	1.5817	0.8342	0.4936	0.4750	0.3217	0.4301	-	-	-	-	-	-	-	-	-	-
0.4134	160	1.6026	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4238	164	1.6237	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4341	168	1.5942	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4444	172	1.6211	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4548	176	1.6076	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4651	180	1.5197	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4755	184	1.5760	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4858	188	1.5535	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4961	192	1.5019	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5039	195	-	0.8004	0.5052	0.4891	0.3107	0.4350	-	-	-	-	-	-	-	-	-	-
0.5065	196	1.5746	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5168	200	1.5618	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5271	204	1.5410	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5375	208	1.5333	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5478	212	1.5448	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5581	216	1.5221	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5685	220	1.5687	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5788	224	1.5448	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5891	228	1.4613	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5995	232	1.5090	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6047	234	-	0.7690	0.4905	0.4702	0.3187	0.4265	-	-	-	-	-	-	-	-	-	-
0.6098	236	1.4572	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6202	240	1.5143	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6305	244	1.4898	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6408	248	1.4234	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6512	252	1.3648	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6615	256	1.3490	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6718	260	1.3034	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6822	264	1.3108	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6925	268	1.2706	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7028	272	1.3064	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7054	273	-	0.6826	0.4836	0.4642	0.2856	0.4112	-	-	-	-	-	-	-	-	-	-
0.7132	276	1.2720	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7235	280	1.2861	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7339	284	1.2676	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7442	288	1.2693	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7545	292	1.3310	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7649	296	1.3174	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7752	300	1.2657	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7855	304	1.2438	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7959	308	1.2717	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8062	312	1.2444	0.6657	0.5069	0.4660	0.3159	0.4296	-	-	-	-	-	-	-	-	-	-
0.8165	316	1.2162	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8269	320	1.2765	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8372	324	1.2029	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8475	328	1.2716	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8579	332	1.2241	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8682	336	1.2276	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8786	340	1.2421	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8889	344	1.2431	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8992	348	1.2105	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.907	351	-	0.6562	0.5197	0.4985	0.3225	0.4469	-	-	-	-	-	-	-	-	-	-
0.9096	352	1.2252	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9199	356	1.2255	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9302	360	1.2393	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9406	364	1.1916	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9509	368	1.1789	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9612	372	1.2024	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9716	376	1.2324	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9819	380	1.2035	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9922	384	1.2166	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.0	387	-	0.6495	0.5036	0.4702	0.3279	0.4339	-	-	-	-	-	-	-	-	-	-
-1	-1	-	-	0.5197	0.4985	0.3225	0.4774	0.2154	0.4666	0.7635	0.5294	0.3066	0.8471	0.2974	0.4229	0.5832	0.4341

The bold row denotes the saved checkpoint.

Training Time

Training: 41.5 minutes
Evaluation: 7.2 minutes
Total: 48.7 minutes

Framework Versions

Python: 3.11.6
Sentence Transformers: 5.7.0.dev0
Transformers: 5.13.1
PyTorch: 2.10.0+cu128
Accelerate: 1.14.0
Datasets: 4.8.4
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultiVectorMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}