Add new MultiVectorEncoder model

314389c verified 3 days ago

114 kB

language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - multi-vector
  - colbert
  - late-interaction
  - generated_from_trainer
  - dataset_size:50000
  - loss:MultiVectorMultipleNegativesRankingLoss
base_model: answerdotai/ModernBERT-base
widget:
  - text: >-
      The job of a physical education teacher is to promote students' physical
      fitness through exercise and sport activities. A bachelor's degree in
      physical education is required along with hands-on student teaching
      experience. Those who teach in public schools must become licensed. Show
      me 10 popular schools.
  - text: >-
      The Handheld Braille Labeler, model 2891, is a braille labeler designed
      for use by individuals who are blind or deaf blind or have low vision.
      This lightweight plastic braille label gun has both braille and print
      characters on its character selection dial.
  - text: surname meaning, fontana
  - text: >-
      garlic did originate near Siberia, Russia so powder was made also at
      russia at american factory. Garlic is native to central Asia, but its use
      spread across the world more than 5000 years ago, before recorded history.
      It was worshipped by the Egyptians and fed to workers building the Gread
      Pyramid at Giza, about 2600 BC. Greek athletes ate it to build their
      strength.
  - text: "With a 50-50 ratio of marijuana to tobacco, the cost of producing a pack of 20 pre-rolled joints could be brought down to just a little more than $20â\x80\x94so a $40 pack at the store. It isnâ\x80\x99t as easy as it seems, though. The government has a vested interest in producing income from the selling of marijuana.nother solution: mix the marijuana with tobacco. If marijuana cigarettes were to be mixed with tobacco, at a 50-50 ratio, it would bring the cost down significantly. Many tobacco farmers will wholesale a pound of their product for less than $2."
datasets:
  - sentence-transformers/msmarco-bm25
pipeline_tag: feature-extraction
library_name: sentence-transformers
metrics:
  - maxsim_accuracy@1
  - maxsim_accuracy@3
  - maxsim_accuracy@5
  - maxsim_accuracy@10
  - maxsim_precision@1
  - maxsim_precision@3
  - maxsim_precision@5
  - maxsim_precision@10
  - maxsim_recall@1
  - maxsim_recall@3
  - maxsim_recall@5
  - maxsim_recall@10
  - maxsim_ndcg@10
  - maxsim_mrr@10
  - maxsim_map@100
model-index:
  - name: ColBERT ModernBERT-base trained on MS MARCO triplets
    results:
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoMSMARCO
          type: NanoMSMARCO
        metrics:
          - type: maxsim_accuracy@1
            value: 0.3
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.4
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.5
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.66
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.3
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.13333333333333333
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.1
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.06600000000000002
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.3
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.4
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.5
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.66
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.4527581784932048
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.39018253968253963
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.40322961689375353
            name: Maxsim Map@100
          - type: maxsim_accuracy@1
            value: 0.32
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.46
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.54
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.6
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.32
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.15333333333333332
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.10800000000000001
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.06000000000000001
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.32
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.46
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.54
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.6
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.45502112234851405
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.4092777777777778
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.43050860789683937
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoNQ
          type: NanoNQ
        metrics:
          - type: maxsim_accuracy@1
            value: 0.22
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.48
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.58
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.74
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.22
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.15999999999999998
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.12
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07600000000000001
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.2
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.44
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.53
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.68
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.44041050127975234
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.3815555555555556
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.3659063680027385
            name: Maxsim Map@100
          - type: maxsim_accuracy@1
            value: 0.28
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.52
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.6
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.74
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.28
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.18
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.128
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.08
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.26
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.48
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.57
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.7
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.4738978154793734
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.414015873015873
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.40376142487928957
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoFiQA2018
          type: NanoFiQA2018
        metrics:
          - type: maxsim_accuracy@1
            value: 0.28
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.42
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.44
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.54
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.28
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.16666666666666669
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.12000000000000002
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07600000000000001
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.17085714285714285
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.2609047619047619
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.2989047619047619
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.36923809523809525
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.30837985877391133
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.3637222222222222
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.26018546740220616
            name: Maxsim Map@100
          - type: maxsim_accuracy@1
            value: 0.3
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.36
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.46
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.5
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.3
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.15333333333333332
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.124
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.068
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.19085714285714286
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.24957142857142856
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.3074047619047619
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.34140476190476193
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.295377235646036
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.34672222222222215
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.2605135618174357
            name: Maxsim Map@100
      - task:
          type: multi-vector-nano-beir
          name: Multi Vector Nano BEIR
        dataset:
          name: NanoBEIR mean
          type: NanoBEIR_mean
        metrics:
          - type: maxsim_accuracy@1
            value: 0.26666666666666666
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.43333333333333335
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.5066666666666667
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.6466666666666666
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.26666666666666666
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.15333333333333335
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.11333333333333334
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07266666666666667
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.2236190476190476
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.366968253968254
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.442968253968254
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.5697460317460318
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.40051617951562285
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.3784867724867725
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.3431071507662327
            name: Maxsim Map@100
          - type: maxsim_accuracy@1
            value: 0.39908948194662486
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.581098901098901
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.6536577708006279
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.7491679748822607
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.39908948194662486
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.25862899005756146
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.203723704866562
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.14720251177394034
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.23062992515826716
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.37294339343615923
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.4321351116772938
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.519316239197735
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.4580842641524777
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.507998205875757
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.3854174628797499
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoClimateFEVER
          type: NanoClimateFEVER
        metrics:
          - type: maxsim_accuracy@1
            value: 0.16
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.22
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.36
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.6
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.16
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.07333333333333333
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.07600000000000001
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07400000000000001
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.065
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.09666666666666666
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.1733333333333333
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.3123333333333333
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.20328688856846505
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.25064285714285706
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.14046408140724229
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoDBPedia
          type: NanoDBPedia
        metrics:
          - type: maxsim_accuracy@1
            value: 0.58
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.72
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.8
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.84
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.58
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.4599999999999999
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.4159999999999999
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.376
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.07193848165362521
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.12336786289481484
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.1697367247949505
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.23906302016037423
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.46049338115280875
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.6635555555555556
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.31549502875205054
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoFEVER
          type: NanoFEVER
        metrics:
          - type: maxsim_accuracy@1
            value: 0.56
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.88
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.92
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.98
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.56
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.30666666666666664
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.19199999999999995
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.10399999999999998
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.5166666666666666
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.8433333333333333
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.8733333333333333
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.9433333333333332
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.7533164734382702
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.711
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.6819194884124758
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoHotpotQA
          type: NanoHotpotQA
        metrics:
          - type: maxsim_accuracy@1
            value: 0.68
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.8
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.82
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.88
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.68
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.3
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.204
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.11599999999999998
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.34
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.45
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.51
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.58
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.5518930389479023
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.7415555555555555
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.45883358284049924
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoNFCorpus
          type: NanoNFCorpus
        metrics:
          - type: maxsim_accuracy@1
            value: 0.32
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.46
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.46
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.58
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.32
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.32
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.28
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.22599999999999998
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.020608387079507632
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.05831411005123257
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.07257884522927437
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.1022383828974261
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.26736600238438624
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.3999365079365079
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.11817323882706855
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoQuoraRetrieval
          type: NanoQuoraRetrieval
        metrics:
          - type: maxsim_accuracy@1
            value: 0.72
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.84
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.9
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.94
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.72
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.3533333333333333
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.23199999999999996
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.12599999999999997
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.6106666666666666
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.812
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.8686666666666667
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.9259999999999999
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.8212615451918094
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.7992222222222223
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.7831050306138172
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoSCIDOCS
          type: NanoSCIDOCS
        metrics:
          - type: maxsim_accuracy@1
            value: 0.34
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.56
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.62
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.7
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.34
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.26666666666666666
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.22
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.15
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.07166666666666666
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.16566666666666663
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.22866666666666663
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.30966666666666665
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.29635887359636653
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.464047619047619
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.22414235511948863
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoArguAna
          type: NanoArguAna
        metrics:
          - type: maxsim_accuracy@1
            value: 0.12
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.46
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.56
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.78
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.12
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.1533333333333333
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.11200000000000003
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07800000000000001
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.12
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.46
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.56
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.78
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.4323319619267822
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.3235793650793651
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.332455902068624
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoSciFact
          type: NanoSciFact
        metrics:
          - type: maxsim_accuracy@1
            value: 0.4
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.56
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.58
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.64
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.4
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.2
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.136
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.07400000000000001
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.375
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.54
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.58
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.64
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.5279305999807071
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.4907142857142857
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.5004343033880488
            name: Maxsim Map@100
      - task:
          type: multi-vector-information-retrieval
          name: Multi Vector Information Retrieval
        dataset:
          name: NanoTouche2020
          type: NanoTouche2020
        metrics:
          - type: maxsim_accuracy@1
            value: 0.40816326530612246
            name: Maxsim Accuracy@1
          - type: maxsim_accuracy@3
            value: 0.7142857142857143
            name: Maxsim Accuracy@3
          - type: maxsim_accuracy@5
            value: 0.8775510204081632
            name: Maxsim Accuracy@5
          - type: maxsim_accuracy@10
            value: 0.9591836734693877
            name: Maxsim Accuracy@10
          - type: maxsim_precision@1
            value: 0.40816326530612246
            name: Maxsim Precision@1
          - type: maxsim_precision@3
            value: 0.44217687074829926
            name: Maxsim Precision@3
          - type: maxsim_precision@5
            value: 0.42040816326530617
            name: Maxsim Precision@5
          - type: maxsim_precision@10
            value: 0.38163265306122446
            name: Maxsim Precision@10
          - type: maxsim_recall@1
            value: 0.03578501546719726
            name: Maxsim Recall@1
          - type: maxsim_recall@3
            value: 0.10934404648592841
            name: Maxsim Recall@3
          - type: maxsim_recall@5
            value: 0.16403611987583339
            name: Maxsim Recall@5
          - type: maxsim_recall@10
            value: 0.27707161127465835
            name: Maxsim Recall@10
          - type: maxsim_ndcg@10
            value: 0.4165604953207882
            name: Maxsim Ndcg@10
          - type: maxsim_mrr@10
            value: 0.5897068351149983
            name: Maxsim Mrr@10
          - type: maxsim_map@100
            value: 0.3606204114138682
            name: Maxsim Map@100

ColBERT ModernBERT-base trained on MS MARCO triplets

This is a Multi-Vector Encoder model finetuned from answerdotai/ModernBERT-base on the msmarco-bm25 dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

Model Type: Multi-Vector Encoder
Base model: answerdotai/ModernBERT-base
Maximum Sequence Length: 8192 tokens
Output Dimensionality: 128 dimensions
Similarity Function: maxsim
Supported Modality: Text
Training Dataset:
- msmarco-bm25
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Documentation: Multi-Vector Encoder Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (2): MultiVectorMask({'skiplist_words': [], 'keep_only_token_ids': None})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/multivector-ModernBERT-base-msmarco-contrastive")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'what does marijuana cost per joint',
]
documents = [
    'With a 50-50 ratio of marijuana to tobacco, the cost of producing a pack of 20 pre-rolled joints could be brought down to just a little more than $20â\x80\x94so a $40 pack at the store. It isnâ\x80\x99t as easy as it seems, though. The government has a vested interest in producing income from the selling of marijuana.nother solution: mix the marijuana with tobacco. If marijuana cigarettes were to be mixed with tobacco, at a 50-50 ratio, it would bring the cost down significantly. Many tobacco farmers will wholesale a pound of their product for less than $2.',
    'What does a dime,dub,eigth,quarter,and a zip of marijuana look like and cost?',
    'In January of 1980, residents decided to incorporate by an overwhelming margin. The Town of Farragut was incorporated on January 16, 1980, with the first board of Mayor and Alderman elected on April 1, 1980.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (8, 128) (129, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 7.4345,  3.4202, -0.0586]])

Evaluation

Metrics

Multi Vector Information Retrieval

Datasets: NanoMSMARCO, NanoNQ, NanoFiQA2018, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
Evaluated with MultiVectorInformationRetrievalEvaluator

Metric	NanoMSMARCO	NanoNQ	NanoFiQA2018	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoHotpotQA	NanoNFCorpus	NanoQuoraRetrieval	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
maxsim_accuracy@1	0.32	0.28	0.3	0.16	0.58	0.56	0.68	0.32	0.72	0.34	0.12	0.4	0.4082
maxsim_accuracy@3	0.46	0.52	0.36	0.22	0.72	0.88	0.8	0.46	0.84	0.56	0.46	0.56	0.7143
maxsim_accuracy@5	0.54	0.6	0.46	0.36	0.8	0.92	0.82	0.46	0.9	0.62	0.56	0.58	0.8776
maxsim_accuracy@10	0.6	0.74	0.5	0.6	0.84	0.98	0.88	0.58	0.94	0.7	0.78	0.64	0.9592
maxsim_precision@1	0.32	0.28	0.3	0.16	0.58	0.56	0.68	0.32	0.72	0.34	0.12	0.4	0.4082
maxsim_precision@3	0.1533	0.18	0.1533	0.0733	0.46	0.3067	0.3	0.32	0.3533	0.2667	0.1533	0.2	0.4422
maxsim_precision@5	0.108	0.128	0.124	0.076	0.416	0.192	0.204	0.28	0.232	0.22	0.112	0.136	0.4204
maxsim_precision@10	0.06	0.08	0.068	0.074	0.376	0.104	0.116	0.226	0.126	0.15	0.078	0.074	0.3816
maxsim_recall@1	0.32	0.26	0.1909	0.065	0.0719	0.5167	0.34	0.0206	0.6107	0.0717	0.12	0.375	0.0358
maxsim_recall@3	0.46	0.48	0.2496	0.0967	0.1234	0.8433	0.45	0.0583	0.812	0.1657	0.46	0.54	0.1093
maxsim_recall@5	0.54	0.57	0.3074	0.1733	0.1697	0.8733	0.51	0.0726	0.8687	0.2287	0.56	0.58	0.164
maxsim_recall@10	0.6	0.7	0.3414	0.3123	0.2391	0.9433	0.58	0.1022	0.926	0.3097	0.78	0.64	0.2771
maxsim_ndcg@10	0.455	0.4739	0.2954	0.2033	0.4605	0.7533	0.5519	0.2674	0.8213	0.2964	0.4323	0.5279	0.4166
maxsim_mrr@10	0.4093	0.414	0.3467	0.2506	0.6636	0.711	0.7416	0.3999	0.7992	0.464	0.3236	0.4907	0.5897
maxsim_map@100	0.4305	0.4038	0.2605	0.1405	0.3155	0.6819	0.4588	0.1182	0.7831	0.2241	0.3325	0.5004	0.3606

Multi Vector Nano BEIR

Dataset: NanoBEIR_mean

Evaluated with MultiVectorNanoBEIREvaluator with these parameters:

{
    "dataset_names": [
        "msmarco",
        "nq",
        "fiqa2018"
    ],
    "dataset_id": "sentence-transformers/NanoBEIR-en"
}

Metric	Value
maxsim_accuracy@1	0.2667
maxsim_accuracy@3	0.4333
maxsim_accuracy@5	0.5067
maxsim_accuracy@10	0.6467
maxsim_precision@1	0.2667
maxsim_precision@3	0.1533
maxsim_precision@5	0.1133
maxsim_precision@10	0.0727
maxsim_recall@1	0.2236
maxsim_recall@3	0.367
maxsim_recall@5	0.443
maxsim_recall@10	0.5697
maxsim_ndcg@10	0.4005
maxsim_mrr@10	0.3785
maxsim_map@100	0.3431

Multi Vector Nano BEIR

Dataset: NanoBEIR_mean

Evaluated with MultiVectorNanoBEIREvaluator with these parameters:

{
    "dataset_names": [
        "climatefever",
        "dbpedia",
        "fever",
        "fiqa2018",
        "hotpotqa",
        "msmarco",
        "nfcorpus",
        "nq",
        "quoraretrieval",
        "scidocs",
        "arguana",
        "scifact",
        "touche2020"
    ],
    "dataset_id": "sentence-transformers/NanoBEIR-en"
}

Metric	Value
maxsim_accuracy@1	0.3991
maxsim_accuracy@3	0.5811
maxsim_accuracy@5	0.6537
maxsim_accuracy@10	0.7492
maxsim_precision@1	0.3991
maxsim_precision@3	0.2586
maxsim_precision@5	0.2037
maxsim_precision@10	0.1472
maxsim_recall@1	0.2306
maxsim_recall@3	0.3729
maxsim_recall@5	0.4321
maxsim_recall@10	0.5193
maxsim_ndcg@10	0.4581
maxsim_mrr@10	0.508
maxsim_map@100	0.3854

Training Details

Training Dataset

msmarco-bm25

Dataset: msmarco-bm25 at ce8a493
Size: 50,000 training samples
Columns: query, positive, and negative

Approximate statistics based on the first 100 samples:

	query	positive	negative
type	string	string	string
modality	text	text	text
details	min: 5 tokens mean: 9.2 tokens max: 23 tokens	min: 24 tokens mean: 85.21 tokens max: 234 tokens	min: 24 tokens mean: 78.79 tokens max: 189 tokens

Samples:

query	positive	negative
`how many days to renew philippine passport to usa`	`The United States requires non-citizens to keep a foreign passport that is valid for six months beyond their date of departure. If you are in the United States legally, then you can renew your Philippine passport at the consulate general's office in Los Angeles.`	`How much does it cost to renew a Philippines passport? Philippine Passport Fees for Renewal is P 950 for 15 working days and 1,200 for 7 working days. This is according to the Department of Freign Affairs website.`
`which sexually transmitted diseases can lead to infections inside joint spaces?`	`Gonorrhea is a sexually transmitted disease (STD) that can infect both men and women. It can cause infections in the genitals, rectum, and throat.It is a very common infection, especially among young people ages 15-24 years.omen with gonorrhea are at risk of developing serious complications from the infection, even if they donât have any symptoms. Symptoms in women can include: 1 Painful or burning sensation when urinating; 2 Increased vaginal discharge; 3 Vaginal bleeding between periods.`	`STDs and Infertility. Sexually transmitted diseases, STDs, also called sexually transmitted infections or STIs, can cause immediate, annoying symptoms with long-lasting, serious repercussions. Few people realize that these sexually transmitted diseases can cause damage that may eventually lead to infertility.`
`when was eviva amore constructed`	`Nasher Sculpture Center Press Images Back of the garden, Nasher Sculpture Center; photo by Tim Hursley. Mark di Suvero, Eviva Amore, 2001 in gardens of Nasher Sculpture Center; photo by Tim Hursley. Jaume Plensa, The Long Night (From Ausias March to Vincent Andres AndrÃ©s) , estelles, estellÃ©s 2007 At Nasher; sculpture center Photo By. tim hursley`	`Richard Serra, My Curves Are Not Mad, 1987 and Augustus Rodin, Eve, 1881 (cast before 1932) at Nasher Sculpture Center; photo by Tim Hursley. Mark di Suvero, Eviva Amore, 2001 at dusk in gardens of Nasher Sculpture Center; photo by Tim Hursley. Jeremy Strick, Director of the Nasher Sculpture.`

Loss: MultiVectorMultipleNegativesRankingLoss with these parameters:

{
    "score_metric": "colbert_scores",
    "scale": 1.0,
    "score_mini_batch_size": null,
    "size_average": true,
    "gather_across_devices": false
}

Evaluation Dataset

msmarco-bm25

Dataset: msmarco-bm25 at ce8a493
Size: 1,000 evaluation samples
Columns: query, positive, and negative

Approximate statistics based on the first 100 samples:

	query	positive	negative
type	string	string	string
modality	text	text	text
details	min: 4 tokens mean: 9.15 tokens max: 19 tokens	min: 39 tokens mean: 82.03 tokens max: 199 tokens	min: 27 tokens mean: 81.62 tokens max: 157 tokens

Samples:

query	positive	negative
`what is chor means`	`â¢ CHORE (noun) The noun CHORE has 1 sense: 1. a specific piece of work required to be done as a duty or for a specific fee. Familiarity information: CHORE used as a noun is very rare.`	`Any two different languages and not just English and other language. Example 1. Chore (pronounced as cHor) means 'a routine task' in English language. Whereas Chor {à¤à¥à¤°} (also pronounced as CHor) means a thief or a burglar in both Hindi and Marathi language.`
`how is gravity measured`	`The gravity of Earth, which is denoted by g, refers to the acceleration that the Earth imparts to objects on or near its surface due to gravity. In SI units this acceleration is measured in metres per second squared (in symbols, m/s2 or mÂ·sâ2) or equivalently in newtons per kilogram (N/kg or NÂ·kgâ1).`	`When the wort is first added to the yeast, the specific gravity of the mixture is measured. Later, the specific gravity may be measured again to determine how much alcohol is in the beer, and to know when to stop the fermentation.hen the wort is first added to the yeast, the specific gravity of the mixture is measured. Later, the specific gravity may be measured again to determine how much alcohol is in the beer, and to know when to stop the fermentation.`
`salary of doctor during fellowship`	`Average fellowship salary and wage. The median expected salary for a Fellowship physician in the United States averages to about $150,353 per annum and an average hourly wage is around $20 per hour. fellowship physician in USA receives an average yearly salary ranging from between $34,225 â $59,542. In addition, a yearly bonus of around $4,888 will be included as part of the annual salary package.`	Medical Fellowship Salary. Medical Fellowship average salary is $55,008, median salary is $- with a salary range from $- to $-.Medical Fellowship salaries are collected from government agencies and companies. Each salary is associated with a real job position.Medical Fellowship salary statistics is not exclusive and is for reference only.They are presented as is and updated regularly.edical Fellowship salaries are collected from government agencies and companies. Each salary is associated with a real job position. Medical Fellowship salary statistics is not exclusive and is for reference only. They are presented as is and updated regularly.

Loss: MultiVectorMultipleNegativesRankingLoss with these parameters:

{
    "score_metric": "colbert_scores",
    "scale": 1.0,
    "score_mini_batch_size": null,
    "size_average": true,
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 32
num_train_epochs: 1
learning_rate: 3e-05
warmup_steps: 0.05
bf16: True
per_device_eval_batch_size: 32
load_best_model_at_end: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

per_device_train_batch_size: 32
num_train_epochs: 1
max_steps: -1
learning_rate: 3e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 0.05
optim: adamw_torch_fused
optim_args: None
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: True
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: None
trackio_bucket_id: None
trackio_static_space_id: None
per_device_eval_batch_size: 32
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: True
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_static_graph: None
ddp_backend: None
ddp_timeout: 1800
fsdp: None
fsdp_config: None
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}
max_length: None

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_maxsim_ndcg@10	NanoNQ_maxsim_ndcg@10	NanoFiQA2018_maxsim_ndcg@10	NanoBEIR_mean_maxsim_ndcg@10	NanoClimateFEVER_maxsim_ndcg@10	NanoDBPedia_maxsim_ndcg@10	NanoFEVER_maxsim_ndcg@10	NanoHotpotQA_maxsim_ndcg@10	NanoNFCorpus_maxsim_ndcg@10	NanoQuoraRetrieval_maxsim_ndcg@10	NanoSCIDOCS_maxsim_ndcg@10	NanoArguAna_maxsim_ndcg@10	NanoSciFact_maxsim_ndcg@10	NanoTouche2020_maxsim_ndcg@10
-1	-1	-	-	0.1739	0.1797	0.1710	0.1749	-	-	-	-	-	-	-	-	-	-
0.0102	16	3.7538	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0205	32	3.3661	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0307	48	2.5871	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0409	64	1.9409	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0512	80	1.5994	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0614	96	1.4578	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0717	112	1.3978	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0819	128	1.2065	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0921	144	1.1288	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1004	157	-	1.0105	0.3962	0.3703	0.2569	0.3412	-	-	-	-	-	-	-	-	-	-
0.1024	160	1.0619	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1126	176	1.0149	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1228	192	0.9993	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1331	208	0.9477	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1433	224	0.8920	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1536	240	0.9153	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1638	256	0.8994	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1740	272	0.8754	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1843	288	0.8445	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1945	304	0.8697	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2009	314	-	0.8284	0.4284	0.4444	0.2827	0.3852	-	-	-	-	-	-	-	-	-	-
0.2047	320	0.8569	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2150	336	0.8456	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2252	352	0.8400	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2354	368	0.8554	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2457	384	0.8155	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2559	400	0.8120	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2662	416	0.7631	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2764	432	0.8388	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2866	448	0.8001	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2969	464	0.7797	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3013	471	-	0.7752	0.4347	0.3954	0.2937	0.3746	-	-	-	-	-	-	-	-	-	-
0.3071	480	0.7782	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3173	496	0.7381	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3276	512	0.7863	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3378	528	0.8055	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3480	544	0.7434	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3583	560	0.7354	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3685	576	0.7944	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3788	592	0.7828	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3890	608	0.7276	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3992	624	0.7484	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4018	628	-	0.7316	0.4197	0.4419	0.2871	0.3829	-	-	-	-	-	-	-	-	-	-
0.4095	640	0.7728	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4197	656	0.7389	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4299	672	0.7296	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4402	688	0.7094	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4504	704	0.7304	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4607	720	0.6726	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4709	736	0.6522	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4811	752	0.6913	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4914	768	0.6558	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5016	784	0.6472	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5022	785	-	0.6360	0.3996	0.3904	0.3067	0.3656	-	-	-	-	-	-	-	-	-	-
0.5118	800	0.6718	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5221	816	0.6167	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5323	832	0.6957	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5425	848	0.6781	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5528	864	0.6126	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5630	880	0.7008	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5733	896	0.5734	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5835	912	0.6099	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5937	928	0.6329	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6027	942	-	0.6093	0.4213	0.4733	0.3191	0.4045	-	-	-	-	-	-	-	-	-	-
0.6040	944	0.5956	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6142	960	0.5947	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6244	976	0.5665	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6347	992	0.6213	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6449	1008	0.5896	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6552	1024	0.5584	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6654	1040	0.6183	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6756	1056	0.5957	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6859	1072	0.5814	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6961	1088	0.5643	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7031	1099	-	0.5899	0.455	0.4739	0.2954	0.4081	-	-	-	-	-	-	-	-	-	-
0.7063	1104	0.6579	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7166	1120	0.5351	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7268	1136	0.6088	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7370	1152	0.5648	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7473	1168	0.5743	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7575	1184	0.5905	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7678	1200	0.5474	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7780	1216	0.6153	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7882	1232	0.6152	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7985	1248	0.5790	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8036	1256	-	0.5776	0.4383	0.4559	0.3231	0.4058	-	-	-	-	-	-	-	-	-	-
0.8087	1264	0.6067	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8189	1280	0.6050	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8292	1296	0.5423	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8394	1312	0.5930	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8496	1328	0.5682	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8599	1344	0.5487	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8701	1360	0.5960	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8804	1376	0.5687	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8906	1392	0.5542	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9008	1408	0.5451	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9040	1413	-	0.5711	0.4252	0.4364	0.3032	0.3883	-	-	-	-	-	-	-	-	-	-
0.9111	1424	0.5537	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9213	1440	0.6169	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9315	1456	0.5889	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9418	1472	0.5735	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9520	1488	0.5574	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9623	1504	0.5436	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9725	1520	0.5563	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9827	1536	0.5849	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9930	1552	0.5657	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.0	1563	-	0.5687	0.4528	0.4404	0.3084	0.4005	-	-	-	-	-	-	-	-	-	-
-1	-1	-	-	0.4550	0.4739	0.2954	0.4581	0.2033	0.4605	0.7533	0.5519	0.2674	0.8213	0.2964	0.4323	0.5279	0.4166

The bold row denotes the saved checkpoint.

Training Time

Training: 21.9 minutes
Evaluation: 8.1 minutes
Total: 30.0 minutes

Framework Versions

Python: 3.11.6
Sentence Transformers: 5.7.0.dev0
Transformers: 5.13.1
PyTorch: 2.10.0+cu128
Accelerate: 1.14.0
Datasets: 4.8.4
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultiVectorMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}