Hugging Face's logo

multi-vector-encoder-testing
/
bert-tiny-multi-vector

Feature Extraction
sentence-transformers
Safetensors
English
bert
multi-vector
colbert
late-interaction
Generated from Trainer
dataset_size:501907
loss:MultiVectorMultipleNegativesRankingLoss
Eval Results (legacy)
text-embeddings-inference
Model card Files
xet
 Community
tomaarsen HF Staff commited on
Commit
a0b72ca
·
verified ·
1 Parent(s): b8bd69c

Add new MultiVectorEncoder model

Browse files
Files changed (15) hide show
  1. 1_Dense/config.json +8 -0
  2. 1_Dense/model.safetensors +3 -0
  3. 2_MultiVectorMask/config.json +7 -0
  4. 3_Normalize/config.json +4 -0
  5. README.md +0 -0
  6. RUN_SUMMARY.md +45 -0
  7. config.json +28 -0
  8. config_sentence_transformers.json +14 -0
  9. model.safetensors +3 -0
  10. modules.json +26 -0
  11. results.json +2443 -0
  12. sentence_bert_config.json +18 -0
  13. tokenizer.json +0 -0
  14. tokenizer_config.json +22 -0
  15. train.py +215 -0
1_Dense/config.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "in_features": 128,
3
+ "out_features": 128,
4
+ "bias": false,
5
+ "activation_function": "torch.nn.modules.linear.Identity",
6
+ "module_input_name": "token_embeddings",
7
+ "module_output_name": "token_embeddings"
8
+ }
1_Dense/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dff3a9b4a56a4e6adb29807bab01900d8465f490ad23cf6934a7e0b2536ade2c
3
+ size 65624
2_MultiVectorMask/config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "skiplist_words": [],
3
+ "skiplist_tasks": [
4
+ "document"
5
+ ],
6
+ "keep_only_token_ids": null
7
+ }
3_Normalize/config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "module_input_name": "token_embeddings",
3
+ "module_output_name": "token_embeddings"
4
+ }
README.md ADDED
The diff for this file is too large to render. See raw diff
 
RUN_SUMMARY.md ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Longer BERT tiny training run
2
+
3
+ Continued multi-vector-encoder-testing/bert-tiny-msmarco at revision 81c5b4e78ac3bdbb01606e60e82bc34d86ed897b.
4
+
5
+ Trained for 10,000 additional steps with batch size 128 on 501,907 MS MARCO BM25 triplets, with 1,024 held-out rows for evaluation loss. This processed 1.28 million triplets over about 2.55 epochs. Learning rate was 1e-5, with 5% warmup and linear decay. Training used bf16 on one RTX 3090 and took 15.6 minutes including periodic evaluation.
6
+
7
+ Selected step 6,000 using mean nDCG@10 on NanoMSMARCO, NanoNQ, and NanoFiQA2018, evaluated every 2,000 steps. The remaining ten datasets were evaluated only before and after training.
8
+
9
+ ## Mean nDCG@10
10
+
11
+ | Evaluation group | Initial model | Longer run |
12
+ | --- | ---: | ---: |
13
+ | Three selection datasets | 0.3326 | 0.3729 |
14
+ | All 13 datasets | 0.3999 | 0.4468 |
15
+ | Ten additional datasets | 0.4201 | 0.4690 |
16
+
17
+ ## Per-dataset nDCG@10
18
+
19
+ | Dataset | Initial model | Longer run | Change |
20
+ | --- | ---: | ---: | ---: |
21
+ | NanoClimateFEVER | 0.1431 | 0.1919 | +0.0489 |
22
+ | NanoDBPedia | 0.3914 | 0.4820 | +0.0907 |
23
+ | NanoFEVER | 0.6310 | 0.6793 | +0.0483 |
24
+ | NanoFiQA2018 | 0.2658 | 0.3027 | +0.0369 |
25
+ | NanoHotpotQA | 0.5766 | 0.6840 | +0.1073 |
26
+ | NanoMSMARCO | 0.4328 | 0.3859 | -0.0469 |
27
+ | NanoNFCorpus | 0.2569 | 0.2852 | +0.0283 |
28
+ | NanoNQ | 0.2992 | 0.4300 | +0.1307 |
29
+ | NanoQuoraRetrieval | 0.7976 | 0.8355 | +0.0379 |
30
+ | NanoSCIDOCS | 0.2029 | 0.2217 | +0.0188 |
31
+ | NanoArguAna | 0.2863 | 0.3539 | +0.0676 |
32
+ | NanoSciFact | 0.5060 | 0.5650 | +0.0590 |
33
+ | NanoTouche2020 | 0.4096 | 0.3920 | -0.0176 |
34
+
35
+ Improved on 11 of 13 datasets. NanoMSMARCO and NanoTouche2020 declined.
36
+
37
+ ## Reproduce
38
+
39
+ Run with a compatible Sentence Transformers checkout and its training dependencies:
40
+
41
+ ```bash
42
+ python train.py --long-run
43
+ ```
44
+
45
+ The script pins the initial model revision. Training arguments, every evaluation, and per-dataset before-and-after results are included in results.json.
config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_cross_attention": false,
3
+ "architectures": [
4
+ "BertModel"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "bos_token_id": null,
8
+ "classifier_dropout": null,
9
+ "dtype": "float32",
10
+ "eos_token_id": null,
11
+ "hidden_act": "gelu",
12
+ "hidden_dropout_prob": 0.1,
13
+ "hidden_size": 128,
14
+ "initializer_range": 0.02,
15
+ "intermediate_size": 512,
16
+ "is_decoder": false,
17
+ "layer_norm_eps": 1e-12,
18
+ "max_position_embeddings": 512,
19
+ "model_type": "bert",
20
+ "num_attention_heads": 2,
21
+ "num_hidden_layers": 2,
22
+ "pad_token_id": 0,
23
+ "tie_word_embeddings": true,
24
+ "transformers_version": "5.16.1",
25
+ "type_vocab_size": 2,
26
+ "use_cache": false,
27
+ "vocab_size": 30522
28
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "pytorch": "2.10.0+cu128",
4
+ "sentence_transformers": "6.1.0.dev0",
5
+ "transformers": "5.16.1"
6
+ },
7
+ "default_prompt_name": null,
8
+ "model_type": "MultiVectorEncoder",
9
+ "prompts": {
10
+ "document": "",
11
+ "query": ""
12
+ },
13
+ "similarity_fn_name": "maxsim"
14
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bd69e90a72afaa8fdbb8fb3d00955479f81dcaf325842b33a6405eca4550002d
3
+ size 17547912
modules.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.base.modules.transformer.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Dense",
12
+ "type": "sentence_transformers.base.modules.dense.Dense"
13
+ },
14
+ {
15
+ "idx": 2,
16
+ "name": "2",
17
+ "path": "2_MultiVectorMask",
18
+ "type": "sentence_transformers.multi_vector_encoder.modules.multi_vector_mask.MultiVectorMask"
19
+ },
20
+ {
21
+ "idx": 3,
22
+ "name": "3",
23
+ "path": "3_Normalize",
24
+ "type": "sentence_transformers.base.modules.normalize.Normalize"
25
+ }
26
+ ]
results.json ADDED
@@ -0,0 +1,2443 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "baseline": {
3
+ "NanoMSMARCO_maxsim_accuracy@1": 0.26,
4
+ "NanoMSMARCO_maxsim_accuracy@3": 0.42,
5
+ "NanoMSMARCO_maxsim_accuracy@5": 0.48,
6
+ "NanoMSMARCO_maxsim_accuracy@10": 0.66,
7
+ "NanoMSMARCO_maxsim_precision@1": 0.26,
8
+ "NanoMSMARCO_maxsim_precision@3": 0.13999999999999999,
9
+ "NanoMSMARCO_maxsim_precision@5": 0.09600000000000002,
10
+ "NanoMSMARCO_maxsim_precision@10": 0.066,
11
+ "NanoMSMARCO_maxsim_recall@1": 0.26,
12
+ "NanoMSMARCO_maxsim_recall@3": 0.42,
13
+ "NanoMSMARCO_maxsim_recall@5": 0.48,
14
+ "NanoMSMARCO_maxsim_recall@10": 0.66,
15
+ "NanoMSMARCO_maxsim_ndcg@10": 0.4328151442419565,
16
+ "NanoMSMARCO_maxsim_mrr@10": 0.36385714285714277,
17
+ "NanoMSMARCO_maxsim_map@100": 0.37752742505056863,
18
+ "NanoNQ_maxsim_accuracy@1": 0.16,
19
+ "NanoNQ_maxsim_accuracy@3": 0.3,
20
+ "NanoNQ_maxsim_accuracy@5": 0.36,
21
+ "NanoNQ_maxsim_accuracy@10": 0.52,
22
+ "NanoNQ_maxsim_precision@1": 0.16,
23
+ "NanoNQ_maxsim_precision@3": 0.1,
24
+ "NanoNQ_maxsim_precision@5": 0.076,
25
+ "NanoNQ_maxsim_precision@10": 0.054000000000000006,
26
+ "NanoNQ_maxsim_recall@1": 0.15,
27
+ "NanoNQ_maxsim_recall@3": 0.28,
28
+ "NanoNQ_maxsim_recall@5": 0.35,
29
+ "NanoNQ_maxsim_recall@10": 0.49,
30
+ "NanoNQ_maxsim_ndcg@10": 0.29921814444748196,
31
+ "NanoNQ_maxsim_mrr@10": 0.2469365079365079,
32
+ "NanoNQ_maxsim_map@100": 0.24908347359322217,
33
+ "NanoFiQA2018_maxsim_accuracy@1": 0.22,
34
+ "NanoFiQA2018_maxsim_accuracy@3": 0.4,
35
+ "NanoFiQA2018_maxsim_accuracy@5": 0.46,
36
+ "NanoFiQA2018_maxsim_accuracy@10": 0.54,
37
+ "NanoFiQA2018_maxsim_precision@1": 0.22,
38
+ "NanoFiQA2018_maxsim_precision@3": 0.1533333333333333,
39
+ "NanoFiQA2018_maxsim_precision@5": 0.11600000000000002,
40
+ "NanoFiQA2018_maxsim_precision@10": 0.07400000000000001,
41
+ "NanoFiQA2018_maxsim_recall@1": 0.09552380952380952,
42
+ "NanoFiQA2018_maxsim_recall@3": 0.22885714285714287,
43
+ "NanoFiQA2018_maxsim_recall@5": 0.2935238095238095,
44
+ "NanoFiQA2018_maxsim_recall@10": 0.35904761904761906,
45
+ "NanoFiQA2018_maxsim_ndcg@10": 0.2657905839072199,
46
+ "NanoFiQA2018_maxsim_mrr@10": 0.3248888888888888,
47
+ "NanoFiQA2018_maxsim_map@100": 0.19617277017199142,
48
+ "NanoBEIR_mean_maxsim_accuracy@1": 0.21333333333333335,
49
+ "NanoBEIR_mean_maxsim_accuracy@3": 0.37333333333333335,
50
+ "NanoBEIR_mean_maxsim_accuracy@5": 0.43333333333333335,
51
+ "NanoBEIR_mean_maxsim_accuracy@10": 0.5733333333333334,
52
+ "NanoBEIR_mean_maxsim_precision@1": 0.21333333333333335,
53
+ "NanoBEIR_mean_maxsim_precision@3": 0.1311111111111111,
54
+ "NanoBEIR_mean_maxsim_precision@5": 0.09600000000000002,
55
+ "NanoBEIR_mean_maxsim_precision@10": 0.06466666666666666,
56
+ "NanoBEIR_mean_maxsim_recall@1": 0.16850793650793652,
57
+ "NanoBEIR_mean_maxsim_recall@3": 0.3096190476190476,
58
+ "NanoBEIR_mean_maxsim_recall@5": 0.37450793650793646,
59
+ "NanoBEIR_mean_maxsim_recall@10": 0.503015873015873,
60
+ "NanoBEIR_mean_maxsim_ndcg@10": 0.33260795753221944,
61
+ "NanoBEIR_mean_maxsim_mrr@10": 0.31189417989417983,
62
+ "NanoBEIR_mean_maxsim_map@100": 0.27426122293859406
63
+ },
64
+ "final": {
65
+ "NanoMSMARCO_maxsim_accuracy@1": 0.16,
66
+ "NanoMSMARCO_maxsim_accuracy@3": 0.32,
67
+ "NanoMSMARCO_maxsim_accuracy@5": 0.42,
68
+ "NanoMSMARCO_maxsim_accuracy@10": 0.7,
69
+ "NanoMSMARCO_maxsim_precision@1": 0.16,
70
+ "NanoMSMARCO_maxsim_precision@3": 0.10666666666666666,
71
+ "NanoMSMARCO_maxsim_precision@5": 0.084,
72
+ "NanoMSMARCO_maxsim_precision@10": 0.07,
73
+ "NanoMSMARCO_maxsim_recall@1": 0.16,
74
+ "NanoMSMARCO_maxsim_recall@3": 0.32,
75
+ "NanoMSMARCO_maxsim_recall@5": 0.42,
76
+ "NanoMSMARCO_maxsim_recall@10": 0.7,
77
+ "NanoMSMARCO_maxsim_ndcg@10": 0.3858968432351718,
78
+ "NanoMSMARCO_maxsim_mrr@10": 0.292015873015873,
79
+ "NanoMSMARCO_maxsim_map@100": 0.3048587220806822,
80
+ "NanoNQ_maxsim_accuracy@1": 0.28,
81
+ "NanoNQ_maxsim_accuracy@3": 0.4,
82
+ "NanoNQ_maxsim_accuracy@5": 0.48,
83
+ "NanoNQ_maxsim_accuracy@10": 0.66,
84
+ "NanoNQ_maxsim_precision@1": 0.28,
85
+ "NanoNQ_maxsim_precision@3": 0.13333333333333333,
86
+ "NanoNQ_maxsim_precision@5": 0.09600000000000002,
87
+ "NanoNQ_maxsim_precision@10": 0.066,
88
+ "NanoNQ_maxsim_recall@1": 0.27,
89
+ "NanoNQ_maxsim_recall@3": 0.39,
90
+ "NanoNQ_maxsim_recall@5": 0.46,
91
+ "NanoNQ_maxsim_recall@10": 0.61,
92
+ "NanoNQ_maxsim_ndcg@10": 0.42995107279160477,
93
+ "NanoNQ_maxsim_mrr@10": 0.3832698412698412,
94
+ "NanoNQ_maxsim_map@100": 0.3841358170885305,
95
+ "NanoFiQA2018_maxsim_accuracy@1": 0.26,
96
+ "NanoFiQA2018_maxsim_accuracy@3": 0.4,
97
+ "NanoFiQA2018_maxsim_accuracy@5": 0.48,
98
+ "NanoFiQA2018_maxsim_accuracy@10": 0.58,
99
+ "NanoFiQA2018_maxsim_precision@1": 0.26,
100
+ "NanoFiQA2018_maxsim_precision@3": 0.18,
101
+ "NanoFiQA2018_maxsim_precision@5": 0.132,
102
+ "NanoFiQA2018_maxsim_precision@10": 0.08,
103
+ "NanoFiQA2018_maxsim_recall@1": 0.12285714285714285,
104
+ "NanoFiQA2018_maxsim_recall@3": 0.251047619047619,
105
+ "NanoFiQA2018_maxsim_recall@5": 0.3117142857142857,
106
+ "NanoFiQA2018_maxsim_recall@10": 0.38704761904761903,
107
+ "NanoFiQA2018_maxsim_ndcg@10": 0.3027040168258662,
108
+ "NanoFiQA2018_maxsim_mrr@10": 0.36041269841269835,
109
+ "NanoFiQA2018_maxsim_map@100": 0.24465363135397997,
110
+ "NanoBEIR_mean_maxsim_accuracy@1": 0.23333333333333336,
111
+ "NanoBEIR_mean_maxsim_accuracy@3": 0.37333333333333335,
112
+ "NanoBEIR_mean_maxsim_accuracy@5": 0.45999999999999996,
113
+ "NanoBEIR_mean_maxsim_accuracy@10": 0.6466666666666666,
114
+ "NanoBEIR_mean_maxsim_precision@1": 0.23333333333333336,
115
+ "NanoBEIR_mean_maxsim_precision@3": 0.13999999999999999,
116
+ "NanoBEIR_mean_maxsim_precision@5": 0.10400000000000002,
117
+ "NanoBEIR_mean_maxsim_precision@10": 0.07200000000000001,
118
+ "NanoBEIR_mean_maxsim_recall@1": 0.1842857142857143,
119
+ "NanoBEIR_mean_maxsim_recall@3": 0.32034920634920633,
120
+ "NanoBEIR_mean_maxsim_recall@5": 0.3972380952380952,
121
+ "NanoBEIR_mean_maxsim_recall@10": 0.5656825396825397,
122
+ "NanoBEIR_mean_maxsim_ndcg@10": 0.37285064428421427,
123
+ "NanoBEIR_mean_maxsim_mrr@10": 0.34523280423280417,
124
+ "NanoBEIR_mean_maxsim_map@100": 0.31121605684106424
125
+ },
126
+ "best_checkpoint": "models\\bert-tiny-msmarco-long\\checkpoint-6000",
127
+ "history": [
128
+ {
129
+ "loss": 1.7376844882965088,
130
+ "grad_norm": 7.415415287017822,
131
+ "learning_rate": 0.0,
132
+ "epoch": 0.00025497195308516065,
133
+ "step": 1
134
+ },
135
+ {
136
+ "loss": 1.734242497658243,
137
+ "grad_norm": 6.916362762451172,
138
+ "learning_rate": 9.800000000000001e-07,
139
+ "epoch": 0.012748597654258032,
140
+ "step": 50
141
+ },
142
+ {
143
+ "loss": 1.7066943359375,
144
+ "grad_norm": 7.056603908538818,
145
+ "learning_rate": 1.98e-06,
146
+ "epoch": 0.025497195308516064,
147
+ "step": 100
148
+ },
149
+ {
150
+ "loss": 1.6563580322265625,
151
+ "grad_norm": 6.4600114822387695,
152
+ "learning_rate": 2.9800000000000003e-06,
153
+ "epoch": 0.0382457929627741,
154
+ "step": 150
155
+ },
156
+ {
157
+ "loss": 1.6342591857910156,
158
+ "grad_norm": 5.999237060546875,
159
+ "learning_rate": 3.980000000000001e-06,
160
+ "epoch": 0.05099439061703213,
161
+ "step": 200
162
+ },
163
+ {
164
+ "loss": 1.6082415771484375,
165
+ "grad_norm": 6.2727789878845215,
166
+ "learning_rate": 4.980000000000001e-06,
167
+ "epoch": 0.06374298827129016,
168
+ "step": 250
169
+ },
170
+ {
171
+ "loss": 1.5952156066894532,
172
+ "grad_norm": 6.114912986755371,
173
+ "learning_rate": 5.98e-06,
174
+ "epoch": 0.0764915859255482,
175
+ "step": 300
176
+ },
177
+ {
178
+ "loss": 1.5606399536132813,
179
+ "grad_norm": 6.401828765869141,
180
+ "learning_rate": 6.98e-06,
181
+ "epoch": 0.08924018357980622,
182
+ "step": 350
183
+ },
184
+ {
185
+ "loss": 1.5329177856445313,
186
+ "grad_norm": 5.98675012588501,
187
+ "learning_rate": 7.980000000000002e-06,
188
+ "epoch": 0.10198878123406425,
189
+ "step": 400
190
+ },
191
+ {
192
+ "loss": 1.4977052307128906,
193
+ "grad_norm": 5.7104811668396,
194
+ "learning_rate": 8.98e-06,
195
+ "epoch": 0.11473737888832228,
196
+ "step": 450
197
+ },
198
+ {
199
+ "loss": 1.5204747009277344,
200
+ "grad_norm": 5.618879318237305,
201
+ "learning_rate": 9.980000000000001e-06,
202
+ "epoch": 0.12748597654258031,
203
+ "step": 500
204
+ },
205
+ {
206
+ "loss": 1.4602200317382812,
207
+ "grad_norm": 5.667468070983887,
208
+ "learning_rate": 9.94842105263158e-06,
209
+ "epoch": 0.14023457419683835,
210
+ "step": 550
211
+ },
212
+ {
213
+ "loss": 1.4464675903320312,
214
+ "grad_norm": 5.8273606300354,
215
+ "learning_rate": 9.895789473684212e-06,
216
+ "epoch": 0.1529831718510964,
217
+ "step": 600
218
+ },
219
+ {
220
+ "loss": 1.4250950622558594,
221
+ "grad_norm": 5.368404388427734,
222
+ "learning_rate": 9.843157894736843e-06,
223
+ "epoch": 0.1657317695053544,
224
+ "step": 650
225
+ },
226
+ {
227
+ "loss": 1.3910955810546874,
228
+ "grad_norm": 6.110495567321777,
229
+ "learning_rate": 9.790526315789475e-06,
230
+ "epoch": 0.17848036715961244,
231
+ "step": 700
232
+ },
233
+ {
234
+ "loss": 1.3776310729980468,
235
+ "grad_norm": 5.904088020324707,
236
+ "learning_rate": 9.737894736842107e-06,
237
+ "epoch": 0.19122896481387047,
238
+ "step": 750
239
+ },
240
+ {
241
+ "loss": 1.3530450439453126,
242
+ "grad_norm": 5.835149765014648,
243
+ "learning_rate": 9.685263157894738e-06,
244
+ "epoch": 0.2039775624681285,
245
+ "step": 800
246
+ },
247
+ {
248
+ "loss": 1.329677734375,
249
+ "grad_norm": 5.804550647735596,
250
+ "learning_rate": 9.63263157894737e-06,
251
+ "epoch": 0.21672616012238655,
252
+ "step": 850
253
+ },
254
+ {
255
+ "loss": 1.3347909545898438,
256
+ "grad_norm": 5.69650936126709,
257
+ "learning_rate": 9.58e-06,
258
+ "epoch": 0.22947475777664456,
259
+ "step": 900
260
+ },
261
+ {
262
+ "loss": 1.342314453125,
263
+ "grad_norm": 6.2678680419921875,
264
+ "learning_rate": 9.527368421052631e-06,
265
+ "epoch": 0.2422233554309026,
266
+ "step": 950
267
+ },
268
+ {
269
+ "loss": 1.2871664428710938,
270
+ "grad_norm": 5.946215629577637,
271
+ "learning_rate": 9.474736842105265e-06,
272
+ "epoch": 0.25497195308516063,
273
+ "step": 1000
274
+ },
275
+ {
276
+ "loss": 1.287988739013672,
277
+ "grad_norm": 5.850720405578613,
278
+ "learning_rate": 9.422105263157896e-06,
279
+ "epoch": 0.26772055073941864,
280
+ "step": 1050
281
+ },
282
+ {
283
+ "loss": 1.2762126159667968,
284
+ "grad_norm": 5.957949638366699,
285
+ "learning_rate": 9.369473684210528e-06,
286
+ "epoch": 0.2804691483936767,
287
+ "step": 1100
288
+ },
289
+ {
290
+ "loss": 1.2545075225830078,
291
+ "grad_norm": 6.1128997802734375,
292
+ "learning_rate": 9.316842105263158e-06,
293
+ "epoch": 0.2932177460479347,
294
+ "step": 1150
295
+ },
296
+ {
297
+ "loss": 1.246324462890625,
298
+ "grad_norm": 5.8769121170043945,
299
+ "learning_rate": 9.26421052631579e-06,
300
+ "epoch": 0.3059663437021928,
301
+ "step": 1200
302
+ },
303
+ {
304
+ "loss": 1.2491684722900391,
305
+ "grad_norm": 5.3106255531311035,
306
+ "learning_rate": 9.211578947368421e-06,
307
+ "epoch": 0.3187149413564508,
308
+ "step": 1250
309
+ },
310
+ {
311
+ "loss": 1.2145277404785155,
312
+ "grad_norm": 6.250970363616943,
313
+ "learning_rate": 9.158947368421055e-06,
314
+ "epoch": 0.3314635390107088,
315
+ "step": 1300
316
+ },
317
+ {
318
+ "loss": 1.236901168823242,
319
+ "grad_norm": 5.616320610046387,
320
+ "learning_rate": 9.106315789473684e-06,
321
+ "epoch": 0.34421213666496686,
322
+ "step": 1350
323
+ },
324
+ {
325
+ "loss": 1.1994841003417969,
326
+ "grad_norm": 6.206327438354492,
327
+ "learning_rate": 9.053684210526316e-06,
328
+ "epoch": 0.35696073431922487,
329
+ "step": 1400
330
+ },
331
+ {
332
+ "loss": 1.1949559020996094,
333
+ "grad_norm": 5.921530246734619,
334
+ "learning_rate": 9.001052631578948e-06,
335
+ "epoch": 0.36970933197348294,
336
+ "step": 1450
337
+ },
338
+ {
339
+ "loss": 1.2016219329833984,
340
+ "grad_norm": 6.331475257873535,
341
+ "learning_rate": 8.94842105263158e-06,
342
+ "epoch": 0.38245792962774094,
343
+ "step": 1500
344
+ },
345
+ {
346
+ "loss": 1.1622018432617187,
347
+ "grad_norm": 6.142783164978027,
348
+ "learning_rate": 8.895789473684211e-06,
349
+ "epoch": 0.39520652728199895,
350
+ "step": 1550
351
+ },
352
+ {
353
+ "loss": 1.1885588073730469,
354
+ "grad_norm": 5.811583518981934,
355
+ "learning_rate": 8.843157894736843e-06,
356
+ "epoch": 0.407955124936257,
357
+ "step": 1600
358
+ },
359
+ {
360
+ "loss": 1.1866447448730468,
361
+ "grad_norm": 6.205753326416016,
362
+ "learning_rate": 8.790526315789474e-06,
363
+ "epoch": 0.42070372259051503,
364
+ "step": 1650
365
+ },
366
+ {
367
+ "loss": 1.1774078369140626,
368
+ "grad_norm": 6.716350078582764,
369
+ "learning_rate": 8.737894736842106e-06,
370
+ "epoch": 0.4334523202447731,
371
+ "step": 1700
372
+ },
373
+ {
374
+ "loss": 1.1461508178710937,
375
+ "grad_norm": 5.739165782928467,
376
+ "learning_rate": 8.685263157894738e-06,
377
+ "epoch": 0.4462009178990311,
378
+ "step": 1750
379
+ },
380
+ {
381
+ "loss": 1.1655081939697265,
382
+ "grad_norm": 5.719452857971191,
383
+ "learning_rate": 8.63263157894737e-06,
384
+ "epoch": 0.4589495155532891,
385
+ "step": 1800
386
+ },
387
+ {
388
+ "loss": 1.1309088897705077,
389
+ "grad_norm": 6.0798749923706055,
390
+ "learning_rate": 8.580000000000001e-06,
391
+ "epoch": 0.4716981132075472,
392
+ "step": 1850
393
+ },
394
+ {
395
+ "loss": 1.1612559509277345,
396
+ "grad_norm": 6.256891250610352,
397
+ "learning_rate": 8.527368421052632e-06,
398
+ "epoch": 0.4844467108618052,
399
+ "step": 1900
400
+ },
401
+ {
402
+ "loss": 1.1465477752685547,
403
+ "grad_norm": 5.919016361236572,
404
+ "learning_rate": 8.474736842105264e-06,
405
+ "epoch": 0.49719530851606325,
406
+ "step": 1950
407
+ },
408
+ {
409
+ "loss": 1.1740735626220704,
410
+ "grad_norm": 5.964484691619873,
411
+ "learning_rate": 8.422105263157896e-06,
412
+ "epoch": 0.5099439061703213,
413
+ "step": 2000
414
+ },
415
+ {
416
+ "eval_loss": 0.8206618428230286,
417
+ "eval_NanoMSMARCO_maxsim_accuracy@1": 0.14,
418
+ "eval_NanoMSMARCO_maxsim_accuracy@3": 0.3,
419
+ "eval_NanoMSMARCO_maxsim_accuracy@5": 0.44,
420
+ "eval_NanoMSMARCO_maxsim_accuracy@10": 0.6,
421
+ "eval_NanoMSMARCO_maxsim_precision@1": 0.14,
422
+ "eval_NanoMSMARCO_maxsim_precision@3": 0.1,
423
+ "eval_NanoMSMARCO_maxsim_precision@5": 0.08800000000000001,
424
+ "eval_NanoMSMARCO_maxsim_precision@10": 0.06,
425
+ "eval_NanoMSMARCO_maxsim_recall@1": 0.14,
426
+ "eval_NanoMSMARCO_maxsim_recall@3": 0.3,
427
+ "eval_NanoMSMARCO_maxsim_recall@5": 0.44,
428
+ "eval_NanoMSMARCO_maxsim_recall@10": 0.6,
429
+ "eval_NanoMSMARCO_maxsim_ndcg@10": 0.34600189291450606,
430
+ "eval_NanoMSMARCO_maxsim_mrr@10": 0.26735714285714285,
431
+ "eval_NanoMSMARCO_maxsim_map@100": 0.29039923269025436,
432
+ "eval_NanoNQ_maxsim_accuracy@1": 0.22,
433
+ "eval_NanoNQ_maxsim_accuracy@3": 0.38,
434
+ "eval_NanoNQ_maxsim_accuracy@5": 0.5,
435
+ "eval_NanoNQ_maxsim_accuracy@10": 0.68,
436
+ "eval_NanoNQ_maxsim_precision@1": 0.22,
437
+ "eval_NanoNQ_maxsim_precision@3": 0.12666666666666665,
438
+ "eval_NanoNQ_maxsim_precision@5": 0.1,
439
+ "eval_NanoNQ_maxsim_precision@10": 0.07,
440
+ "eval_NanoNQ_maxsim_recall@1": 0.21,
441
+ "eval_NanoNQ_maxsim_recall@3": 0.37,
442
+ "eval_NanoNQ_maxsim_recall@5": 0.48,
443
+ "eval_NanoNQ_maxsim_recall@10": 0.65,
444
+ "eval_NanoNQ_maxsim_ndcg@10": 0.4138066133720556,
445
+ "eval_NanoNQ_maxsim_mrr@10": 0.34757142857142853,
446
+ "eval_NanoNQ_maxsim_map@100": 0.3463553396303396,
447
+ "eval_NanoFiQA2018_maxsim_accuracy@1": 0.22,
448
+ "eval_NanoFiQA2018_maxsim_accuracy@3": 0.34,
449
+ "eval_NanoFiQA2018_maxsim_accuracy@5": 0.46,
450
+ "eval_NanoFiQA2018_maxsim_accuracy@10": 0.58,
451
+ "eval_NanoFiQA2018_maxsim_precision@1": 0.22,
452
+ "eval_NanoFiQA2018_maxsim_precision@3": 0.16,
453
+ "eval_NanoFiQA2018_maxsim_precision@5": 0.132,
454
+ "eval_NanoFiQA2018_maxsim_precision@10": 0.08,
455
+ "eval_NanoFiQA2018_maxsim_recall@1": 0.09885714285714287,
456
+ "eval_NanoFiQA2018_maxsim_recall@3": 0.1963809523809524,
457
+ "eval_NanoFiQA2018_maxsim_recall@5": 0.29304761904761906,
458
+ "eval_NanoFiQA2018_maxsim_recall@10": 0.38704761904761903,
459
+ "eval_NanoFiQA2018_maxsim_ndcg@10": 0.2792292147240143,
460
+ "eval_NanoFiQA2018_maxsim_mrr@10": 0.32299999999999995,
461
+ "eval_NanoFiQA2018_maxsim_map@100": 0.2137404159684256,
462
+ "eval_NanoBEIR_mean_maxsim_accuracy@1": 0.19333333333333333,
463
+ "eval_NanoBEIR_mean_maxsim_accuracy@3": 0.34,
464
+ "eval_NanoBEIR_mean_maxsim_accuracy@5": 0.4666666666666666,
465
+ "eval_NanoBEIR_mean_maxsim_accuracy@10": 0.62,
466
+ "eval_NanoBEIR_mean_maxsim_precision@1": 0.19333333333333333,
467
+ "eval_NanoBEIR_mean_maxsim_precision@3": 0.1288888888888889,
468
+ "eval_NanoBEIR_mean_maxsim_precision@5": 0.10666666666666667,
469
+ "eval_NanoBEIR_mean_maxsim_precision@10": 0.07,
470
+ "eval_NanoBEIR_mean_maxsim_recall@1": 0.14961904761904762,
471
+ "eval_NanoBEIR_mean_maxsim_recall@3": 0.28879365079365077,
472
+ "eval_NanoBEIR_mean_maxsim_recall@5": 0.4043492063492063,
473
+ "eval_NanoBEIR_mean_maxsim_recall@10": 0.5456825396825397,
474
+ "eval_NanoBEIR_mean_maxsim_ndcg@10": 0.3463459070035253,
475
+ "eval_NanoBEIR_mean_maxsim_mrr@10": 0.3126428571428571,
476
+ "eval_NanoBEIR_mean_maxsim_map@100": 0.2834983294296732,
477
+ "eval_runtime": 4.0747,
478
+ "eval_samples_per_second": 251.305,
479
+ "eval_steps_per_second": 7.853,
480
+ "epoch": 0.5099439061703213,
481
+ "step": 2000
482
+ },
483
+ {
484
+ "loss": 1.1287294006347657,
485
+ "grad_norm": 5.990379333496094,
486
+ "learning_rate": 8.369473684210527e-06,
487
+ "epoch": 0.5226925038245793,
488
+ "step": 2050
489
+ },
490
+ {
491
+ "loss": 1.1324900054931641,
492
+ "grad_norm": 5.922431468963623,
493
+ "learning_rate": 8.316842105263159e-06,
494
+ "epoch": 0.5354411014788373,
495
+ "step": 2100
496
+ },
497
+ {
498
+ "loss": 1.1120962524414062,
499
+ "grad_norm": 5.448898792266846,
500
+ "learning_rate": 8.26421052631579e-06,
501
+ "epoch": 0.5481896991330953,
502
+ "step": 2150
503
+ },
504
+ {
505
+ "loss": 1.104578628540039,
506
+ "grad_norm": 5.152342319488525,
507
+ "learning_rate": 8.211578947368422e-06,
508
+ "epoch": 0.5609382967873534,
509
+ "step": 2200
510
+ },
511
+ {
512
+ "loss": 1.1026490783691407,
513
+ "grad_norm": 5.983816623687744,
514
+ "learning_rate": 8.158947368421052e-06,
515
+ "epoch": 0.5736868944416115,
516
+ "step": 2250
517
+ },
518
+ {
519
+ "loss": 1.124510269165039,
520
+ "grad_norm": 6.402281284332275,
521
+ "learning_rate": 8.106315789473684e-06,
522
+ "epoch": 0.5864354920958694,
523
+ "step": 2300
524
+ },
525
+ {
526
+ "loss": 1.1252516937255859,
527
+ "grad_norm": 6.615288257598877,
528
+ "learning_rate": 8.053684210526317e-06,
529
+ "epoch": 0.5991840897501275,
530
+ "step": 2350
531
+ },
532
+ {
533
+ "loss": 1.1050936126708983,
534
+ "grad_norm": 6.889590740203857,
535
+ "learning_rate": 8.001052631578949e-06,
536
+ "epoch": 0.6119326874043856,
537
+ "step": 2400
538
+ },
539
+ {
540
+ "loss": 1.0870243835449218,
541
+ "grad_norm": 6.245548725128174,
542
+ "learning_rate": 7.94842105263158e-06,
543
+ "epoch": 0.6246812850586435,
544
+ "step": 2450
545
+ },
546
+ {
547
+ "loss": 1.077528839111328,
548
+ "grad_norm": 5.783371448516846,
549
+ "learning_rate": 7.89578947368421e-06,
550
+ "epoch": 0.6374298827129016,
551
+ "step": 2500
552
+ },
553
+ {
554
+ "loss": 1.0713024139404297,
555
+ "grad_norm": 5.964858531951904,
556
+ "learning_rate": 7.843157894736842e-06,
557
+ "epoch": 0.6501784803671596,
558
+ "step": 2550
559
+ },
560
+ {
561
+ "loss": 1.0945580291748047,
562
+ "grad_norm": 5.975121974945068,
563
+ "learning_rate": 7.790526315789474e-06,
564
+ "epoch": 0.6629270780214176,
565
+ "step": 2600
566
+ },
567
+ {
568
+ "loss": 1.0681832885742188,
569
+ "grad_norm": 6.2195611000061035,
570
+ "learning_rate": 7.737894736842105e-06,
571
+ "epoch": 0.6756756756756757,
572
+ "step": 2650
573
+ },
574
+ {
575
+ "loss": 1.0537023162841797,
576
+ "grad_norm": 5.614584922790527,
577
+ "learning_rate": 7.685263157894739e-06,
578
+ "epoch": 0.6884242733299337,
579
+ "step": 2700
580
+ },
581
+ {
582
+ "loss": 1.0500063323974609,
583
+ "grad_norm": 5.860501289367676,
584
+ "learning_rate": 7.632631578947369e-06,
585
+ "epoch": 0.7011728709841918,
586
+ "step": 2750
587
+ },
588
+ {
589
+ "loss": 1.0920198059082031,
590
+ "grad_norm": 6.509205341339111,
591
+ "learning_rate": 7.58e-06,
592
+ "epoch": 0.7139214686384497,
593
+ "step": 2800
594
+ },
595
+ {
596
+ "loss": 1.0873284912109376,
597
+ "grad_norm": 5.848539352416992,
598
+ "learning_rate": 7.527368421052632e-06,
599
+ "epoch": 0.7266700662927078,
600
+ "step": 2850
601
+ },
602
+ {
603
+ "loss": 1.059793701171875,
604
+ "grad_norm": 7.743218898773193,
605
+ "learning_rate": 7.4747368421052635e-06,
606
+ "epoch": 0.7394186639469659,
607
+ "step": 2900
608
+ },
609
+ {
610
+ "loss": 1.0745486450195312,
611
+ "grad_norm": 5.6027607917785645,
612
+ "learning_rate": 7.422105263157895e-06,
613
+ "epoch": 0.7521672616012238,
614
+ "step": 2950
615
+ },
616
+ {
617
+ "loss": 1.0586804962158203,
618
+ "grad_norm": 5.33534574508667,
619
+ "learning_rate": 7.369473684210528e-06,
620
+ "epoch": 0.7649158592554819,
621
+ "step": 3000
622
+ },
623
+ {
624
+ "loss": 1.0532286834716797,
625
+ "grad_norm": 5.572927951812744,
626
+ "learning_rate": 7.3168421052631585e-06,
627
+ "epoch": 0.77766445690974,
628
+ "step": 3050
629
+ },
630
+ {
631
+ "loss": 1.0608716583251954,
632
+ "grad_norm": 5.5701775550842285,
633
+ "learning_rate": 7.26421052631579e-06,
634
+ "epoch": 0.7904130545639979,
635
+ "step": 3100
636
+ },
637
+ {
638
+ "loss": 1.061068344116211,
639
+ "grad_norm": 5.325122356414795,
640
+ "learning_rate": 7.211578947368422e-06,
641
+ "epoch": 0.803161652218256,
642
+ "step": 3150
643
+ },
644
+ {
645
+ "loss": 1.059212646484375,
646
+ "grad_norm": 5.75468111038208,
647
+ "learning_rate": 7.158947368421053e-06,
648
+ "epoch": 0.815910249872514,
649
+ "step": 3200
650
+ },
651
+ {
652
+ "loss": 1.041295928955078,
653
+ "grad_norm": 6.296924591064453,
654
+ "learning_rate": 7.106315789473684e-06,
655
+ "epoch": 0.8286588475267721,
656
+ "step": 3250
657
+ },
658
+ {
659
+ "loss": 1.0224284362792968,
660
+ "grad_norm": 5.157304763793945,
661
+ "learning_rate": 7.053684210526316e-06,
662
+ "epoch": 0.8414074451810301,
663
+ "step": 3300
664
+ },
665
+ {
666
+ "loss": 1.0371504974365235,
667
+ "grad_norm": 6.586690902709961,
668
+ "learning_rate": 7.001052631578948e-06,
669
+ "epoch": 0.8541560428352881,
670
+ "step": 3350
671
+ },
672
+ {
673
+ "loss": 1.054374008178711,
674
+ "grad_norm": 6.449936389923096,
675
+ "learning_rate": 6.94842105263158e-06,
676
+ "epoch": 0.8669046404895462,
677
+ "step": 3400
678
+ },
679
+ {
680
+ "loss": 0.9921880340576172,
681
+ "grad_norm": 5.447681903839111,
682
+ "learning_rate": 6.8957894736842116e-06,
683
+ "epoch": 0.8796532381438041,
684
+ "step": 3450
685
+ },
686
+ {
687
+ "loss": 1.0214737701416015,
688
+ "grad_norm": 6.658762454986572,
689
+ "learning_rate": 6.843157894736842e-06,
690
+ "epoch": 0.8924018357980622,
691
+ "step": 3500
692
+ },
693
+ {
694
+ "loss": 1.0051663970947267,
695
+ "grad_norm": 6.1249680519104,
696
+ "learning_rate": 6.790526315789474e-06,
697
+ "epoch": 0.9051504334523203,
698
+ "step": 3550
699
+ },
700
+ {
701
+ "loss": 1.0060914611816407,
702
+ "grad_norm": 6.070328712463379,
703
+ "learning_rate": 6.737894736842106e-06,
704
+ "epoch": 0.9178990311065782,
705
+ "step": 3600
706
+ },
707
+ {
708
+ "loss": 1.0136323547363282,
709
+ "grad_norm": 5.561135292053223,
710
+ "learning_rate": 6.685263157894737e-06,
711
+ "epoch": 0.9306476287608363,
712
+ "step": 3650
713
+ },
714
+ {
715
+ "loss": 1.0235866546630858,
716
+ "grad_norm": 6.205555438995361,
717
+ "learning_rate": 6.63263157894737e-06,
718
+ "epoch": 0.9433962264150944,
719
+ "step": 3700
720
+ },
721
+ {
722
+ "loss": 0.993685302734375,
723
+ "grad_norm": 5.501027584075928,
724
+ "learning_rate": 6.5800000000000005e-06,
725
+ "epoch": 0.9561448240693524,
726
+ "step": 3750
727
+ },
728
+ {
729
+ "loss": 1.012783203125,
730
+ "grad_norm": 6.655802249908447,
731
+ "learning_rate": 6.527368421052632e-06,
732
+ "epoch": 0.9688934217236104,
733
+ "step": 3800
734
+ },
735
+ {
736
+ "loss": 1.0399366760253905,
737
+ "grad_norm": 6.5160956382751465,
738
+ "learning_rate": 6.474736842105264e-06,
739
+ "epoch": 0.9816420193778684,
740
+ "step": 3850
741
+ },
742
+ {
743
+ "loss": 0.9839756774902344,
744
+ "grad_norm": 6.4597039222717285,
745
+ "learning_rate": 6.4221052631578954e-06,
746
+ "epoch": 0.9943906170321265,
747
+ "step": 3900
748
+ },
749
+ {
750
+ "loss": 0.9999301910400391,
751
+ "grad_norm": 5.411151885986328,
752
+ "learning_rate": 6.369473684210526e-06,
753
+ "epoch": 1.0071392146863845,
754
+ "step": 3950
755
+ },
756
+ {
757
+ "loss": 1.0204017639160157,
758
+ "grad_norm": 5.2421112060546875,
759
+ "learning_rate": 6.316842105263158e-06,
760
+ "epoch": 1.0198878123406425,
761
+ "step": 4000
762
+ },
763
+ {
764
+ "eval_loss": 0.7237886190414429,
765
+ "eval_NanoMSMARCO_maxsim_accuracy@1": 0.16,
766
+ "eval_NanoMSMARCO_maxsim_accuracy@3": 0.34,
767
+ "eval_NanoMSMARCO_maxsim_accuracy@5": 0.42,
768
+ "eval_NanoMSMARCO_maxsim_accuracy@10": 0.68,
769
+ "eval_NanoMSMARCO_maxsim_precision@1": 0.16,
770
+ "eval_NanoMSMARCO_maxsim_precision@3": 0.11333333333333333,
771
+ "eval_NanoMSMARCO_maxsim_precision@5": 0.084,
772
+ "eval_NanoMSMARCO_maxsim_precision@10": 0.068,
773
+ "eval_NanoMSMARCO_maxsim_recall@1": 0.16,
774
+ "eval_NanoMSMARCO_maxsim_recall@3": 0.34,
775
+ "eval_NanoMSMARCO_maxsim_recall@5": 0.42,
776
+ "eval_NanoMSMARCO_maxsim_recall@10": 0.68,
777
+ "eval_NanoMSMARCO_maxsim_ndcg@10": 0.3782341818236686,
778
+ "eval_NanoMSMARCO_maxsim_mrr@10": 0.2877619047619047,
779
+ "eval_NanoMSMARCO_maxsim_map@100": 0.30340628302046213,
780
+ "eval_NanoNQ_maxsim_accuracy@1": 0.28,
781
+ "eval_NanoNQ_maxsim_accuracy@3": 0.38,
782
+ "eval_NanoNQ_maxsim_accuracy@5": 0.5,
783
+ "eval_NanoNQ_maxsim_accuracy@10": 0.7,
784
+ "eval_NanoNQ_maxsim_precision@1": 0.28,
785
+ "eval_NanoNQ_maxsim_precision@3": 0.12666666666666665,
786
+ "eval_NanoNQ_maxsim_precision@5": 0.1,
787
+ "eval_NanoNQ_maxsim_precision@10": 0.07,
788
+ "eval_NanoNQ_maxsim_recall@1": 0.27,
789
+ "eval_NanoNQ_maxsim_recall@3": 0.37,
790
+ "eval_NanoNQ_maxsim_recall@5": 0.48,
791
+ "eval_NanoNQ_maxsim_recall@10": 0.65,
792
+ "eval_NanoNQ_maxsim_ndcg@10": 0.4405203635714057,
793
+ "eval_NanoNQ_maxsim_mrr@10": 0.38526984126984126,
794
+ "eval_NanoNQ_maxsim_map@100": 0.3829180996894442,
795
+ "eval_NanoFiQA2018_maxsim_accuracy@1": 0.24,
796
+ "eval_NanoFiQA2018_maxsim_accuracy@3": 0.4,
797
+ "eval_NanoFiQA2018_maxsim_accuracy@5": 0.46,
798
+ "eval_NanoFiQA2018_maxsim_accuracy@10": 0.58,
799
+ "eval_NanoFiQA2018_maxsim_precision@1": 0.24,
800
+ "eval_NanoFiQA2018_maxsim_precision@3": 0.18,
801
+ "eval_NanoFiQA2018_maxsim_precision@5": 0.132,
802
+ "eval_NanoFiQA2018_maxsim_precision@10": 0.08,
803
+ "eval_NanoFiQA2018_maxsim_recall@1": 0.09952380952380953,
804
+ "eval_NanoFiQA2018_maxsim_recall@3": 0.251047619047619,
805
+ "eval_NanoFiQA2018_maxsim_recall@5": 0.303047619047619,
806
+ "eval_NanoFiQA2018_maxsim_recall@10": 0.38704761904761903,
807
+ "eval_NanoFiQA2018_maxsim_ndcg@10": 0.28811105912017293,
808
+ "eval_NanoFiQA2018_maxsim_mrr@10": 0.34024603174603174,
809
+ "eval_NanoFiQA2018_maxsim_map@100": 0.22545265942194526,
810
+ "eval_NanoBEIR_mean_maxsim_accuracy@1": 0.22666666666666668,
811
+ "eval_NanoBEIR_mean_maxsim_accuracy@3": 0.37333333333333335,
812
+ "eval_NanoBEIR_mean_maxsim_accuracy@5": 0.45999999999999996,
813
+ "eval_NanoBEIR_mean_maxsim_accuracy@10": 0.6533333333333333,
814
+ "eval_NanoBEIR_mean_maxsim_precision@1": 0.22666666666666668,
815
+ "eval_NanoBEIR_mean_maxsim_precision@3": 0.13999999999999999,
816
+ "eval_NanoBEIR_mean_maxsim_precision@5": 0.10533333333333333,
817
+ "eval_NanoBEIR_mean_maxsim_precision@10": 0.07266666666666667,
818
+ "eval_NanoBEIR_mean_maxsim_recall@1": 0.17650793650793653,
819
+ "eval_NanoBEIR_mean_maxsim_recall@3": 0.32034920634920633,
820
+ "eval_NanoBEIR_mean_maxsim_recall@5": 0.401015873015873,
821
+ "eval_NanoBEIR_mean_maxsim_recall@10": 0.5723492063492064,
822
+ "eval_NanoBEIR_mean_maxsim_ndcg@10": 0.3689552015050824,
823
+ "eval_NanoBEIR_mean_maxsim_mrr@10": 0.33775925925925926,
824
+ "eval_NanoBEIR_mean_maxsim_map@100": 0.3039256807106172,
825
+ "eval_runtime": 3.7291,
826
+ "eval_samples_per_second": 274.6,
827
+ "eval_steps_per_second": 8.581,
828
+ "epoch": 1.0198878123406425,
829
+ "step": 4000
830
+ },
831
+ {
832
+ "loss": 1.0074549865722657,
833
+ "grad_norm": 5.800049781799316,
834
+ "learning_rate": 6.26421052631579e-06,
835
+ "epoch": 1.0326364099949006,
836
+ "step": 4050
837
+ },
838
+ {
839
+ "loss": 1.0111343383789062,
840
+ "grad_norm": 5.648647308349609,
841
+ "learning_rate": 6.211578947368422e-06,
842
+ "epoch": 1.0453850076491586,
843
+ "step": 4100
844
+ },
845
+ {
846
+ "loss": 0.9972299194335937,
847
+ "grad_norm": 5.885046482086182,
848
+ "learning_rate": 6.158947368421054e-06,
849
+ "epoch": 1.0581336053034167,
850
+ "step": 4150
851
+ },
852
+ {
853
+ "loss": 1.0143543243408204,
854
+ "grad_norm": 5.489906311035156,
855
+ "learning_rate": 6.106315789473684e-06,
856
+ "epoch": 1.0708822029576746,
857
+ "step": 4200
858
+ },
859
+ {
860
+ "loss": 1.0053844451904297,
861
+ "grad_norm": 5.163933277130127,
862
+ "learning_rate": 6.053684210526316e-06,
863
+ "epoch": 1.0836308006119326,
864
+ "step": 4250
865
+ },
866
+ {
867
+ "loss": 1.01227783203125,
868
+ "grad_norm": 5.415525913238525,
869
+ "learning_rate": 6.001052631578948e-06,
870
+ "epoch": 1.0963793982661907,
871
+ "step": 4300
872
+ },
873
+ {
874
+ "loss": 0.9975738525390625,
875
+ "grad_norm": 6.0239715576171875,
876
+ "learning_rate": 5.94842105263158e-06,
877
+ "epoch": 1.1091279959204488,
878
+ "step": 4350
879
+ },
880
+ {
881
+ "loss": 0.9797928619384766,
882
+ "grad_norm": 5.9564127922058105,
883
+ "learning_rate": 5.895789473684212e-06,
884
+ "epoch": 1.1218765935747068,
885
+ "step": 4400
886
+ },
887
+ {
888
+ "loss": 1.0313327026367187,
889
+ "grad_norm": 6.634435653686523,
890
+ "learning_rate": 5.843157894736843e-06,
891
+ "epoch": 1.1346251912289649,
892
+ "step": 4450
893
+ },
894
+ {
895
+ "loss": 0.9912808227539063,
896
+ "grad_norm": 5.924166679382324,
897
+ "learning_rate": 5.790526315789474e-06,
898
+ "epoch": 1.1473737888832227,
899
+ "step": 4500
900
+ },
901
+ {
902
+ "loss": 0.9911473846435547,
903
+ "grad_norm": 5.722446918487549,
904
+ "learning_rate": 5.737894736842106e-06,
905
+ "epoch": 1.1601223865374808,
906
+ "step": 4550
907
+ },
908
+ {
909
+ "loss": 0.9879852294921875,
910
+ "grad_norm": 5.746371269226074,
911
+ "learning_rate": 5.685263157894737e-06,
912
+ "epoch": 1.1728709841917389,
913
+ "step": 4600
914
+ },
915
+ {
916
+ "loss": 0.9833634948730469,
917
+ "grad_norm": 5.92328405380249,
918
+ "learning_rate": 5.632631578947368e-06,
919
+ "epoch": 1.185619581845997,
920
+ "step": 4650
921
+ },
922
+ {
923
+ "loss": 0.9488947296142578,
924
+ "grad_norm": 5.607302665710449,
925
+ "learning_rate": 5.580000000000001e-06,
926
+ "epoch": 1.198368179500255,
927
+ "step": 4700
928
+ },
929
+ {
930
+ "loss": 0.9462134552001953,
931
+ "grad_norm": 5.134405612945557,
932
+ "learning_rate": 5.527368421052632e-06,
933
+ "epoch": 1.211116777154513,
934
+ "step": 4750
935
+ },
936
+ {
937
+ "loss": 0.9669766235351562,
938
+ "grad_norm": 6.016522407531738,
939
+ "learning_rate": 5.474736842105264e-06,
940
+ "epoch": 1.2238653748087711,
941
+ "step": 4800
942
+ },
943
+ {
944
+ "loss": 0.9765520477294922,
945
+ "grad_norm": 5.564464092254639,
946
+ "learning_rate": 5.422105263157895e-06,
947
+ "epoch": 1.236613972463029,
948
+ "step": 4850
949
+ },
950
+ {
951
+ "loss": 0.9655083465576172,
952
+ "grad_norm": 5.814446449279785,
953
+ "learning_rate": 5.3694736842105265e-06,
954
+ "epoch": 1.249362570117287,
955
+ "step": 4900
956
+ },
957
+ {
958
+ "loss": 0.9524942779541016,
959
+ "grad_norm": 5.315156936645508,
960
+ "learning_rate": 5.316842105263158e-06,
961
+ "epoch": 1.262111167771545,
962
+ "step": 4950
963
+ },
964
+ {
965
+ "loss": 0.9818015289306641,
966
+ "grad_norm": 6.273362159729004,
967
+ "learning_rate": 5.26421052631579e-06,
968
+ "epoch": 1.2748597654258031,
969
+ "step": 5000
970
+ },
971
+ {
972
+ "loss": 0.9698784637451172,
973
+ "grad_norm": 5.575201988220215,
974
+ "learning_rate": 5.211578947368422e-06,
975
+ "epoch": 1.2876083630800612,
976
+ "step": 5050
977
+ },
978
+ {
979
+ "loss": 0.9678575134277344,
980
+ "grad_norm": 6.0221686363220215,
981
+ "learning_rate": 5.158947368421053e-06,
982
+ "epoch": 1.3003569607343193,
983
+ "step": 5100
984
+ },
985
+ {
986
+ "loss": 0.9810379791259766,
987
+ "grad_norm": 5.811202526092529,
988
+ "learning_rate": 5.106315789473685e-06,
989
+ "epoch": 1.3131055583885773,
990
+ "step": 5150
991
+ },
992
+ {
993
+ "loss": 0.9722916412353516,
994
+ "grad_norm": 5.7698211669921875,
995
+ "learning_rate": 5.053684210526316e-06,
996
+ "epoch": 1.3258541560428352,
997
+ "step": 5200
998
+ },
999
+ {
1000
+ "loss": 0.9659819793701172,
1001
+ "grad_norm": 6.167635440826416,
1002
+ "learning_rate": 5.001052631578948e-06,
1003
+ "epoch": 1.3386027536970933,
1004
+ "step": 5250
1005
+ },
1006
+ {
1007
+ "loss": 0.9688465118408203,
1008
+ "grad_norm": 5.632967948913574,
1009
+ "learning_rate": 4.94842105263158e-06,
1010
+ "epoch": 1.3513513513513513,
1011
+ "step": 5300
1012
+ },
1013
+ {
1014
+ "loss": 0.97010009765625,
1015
+ "grad_norm": 5.637306213378906,
1016
+ "learning_rate": 4.895789473684211e-06,
1017
+ "epoch": 1.3640999490056094,
1018
+ "step": 5350
1019
+ },
1020
+ {
1021
+ "loss": 0.9531510162353516,
1022
+ "grad_norm": 4.932260036468506,
1023
+ "learning_rate": 4.843157894736842e-06,
1024
+ "epoch": 1.3768485466598674,
1025
+ "step": 5400
1026
+ },
1027
+ {
1028
+ "loss": 0.9817030334472656,
1029
+ "grad_norm": 5.899550914764404,
1030
+ "learning_rate": 4.790526315789474e-06,
1031
+ "epoch": 1.3895971443141255,
1032
+ "step": 5450
1033
+ },
1034
+ {
1035
+ "loss": 0.9400790405273437,
1036
+ "grad_norm": 5.441213607788086,
1037
+ "learning_rate": 4.737894736842106e-06,
1038
+ "epoch": 1.4023457419683836,
1039
+ "step": 5500
1040
+ },
1041
+ {
1042
+ "loss": 0.9398882293701172,
1043
+ "grad_norm": 5.9854655265808105,
1044
+ "learning_rate": 4.685263157894737e-06,
1045
+ "epoch": 1.4150943396226414,
1046
+ "step": 5550
1047
+ },
1048
+ {
1049
+ "loss": 0.9505341339111328,
1050
+ "grad_norm": 5.776639938354492,
1051
+ "learning_rate": 4.6326315789473686e-06,
1052
+ "epoch": 1.4278429372768995,
1053
+ "step": 5600
1054
+ },
1055
+ {
1056
+ "loss": 0.9515079498291016,
1057
+ "grad_norm": 5.740902900695801,
1058
+ "learning_rate": 4.58e-06,
1059
+ "epoch": 1.4405915349311575,
1060
+ "step": 5650
1061
+ },
1062
+ {
1063
+ "loss": 0.9575539398193359,
1064
+ "grad_norm": 6.547975063323975,
1065
+ "learning_rate": 4.527368421052632e-06,
1066
+ "epoch": 1.4533401325854156,
1067
+ "step": 5700
1068
+ },
1069
+ {
1070
+ "loss": 0.9672400665283203,
1071
+ "grad_norm": 6.232046127319336,
1072
+ "learning_rate": 4.4747368421052635e-06,
1073
+ "epoch": 1.4660887302396737,
1074
+ "step": 5750
1075
+ },
1076
+ {
1077
+ "loss": 0.9414089965820313,
1078
+ "grad_norm": 5.383060455322266,
1079
+ "learning_rate": 4.422105263157895e-06,
1080
+ "epoch": 1.4788373278939317,
1081
+ "step": 5800
1082
+ },
1083
+ {
1084
+ "loss": 0.9354756927490234,
1085
+ "grad_norm": 5.175451278686523,
1086
+ "learning_rate": 4.369473684210527e-06,
1087
+ "epoch": 1.4915859255481898,
1088
+ "step": 5850
1089
+ },
1090
+ {
1091
+ "loss": 0.9615425872802734,
1092
+ "grad_norm": 6.005729675292969,
1093
+ "learning_rate": 4.316842105263158e-06,
1094
+ "epoch": 1.5043345232024476,
1095
+ "step": 5900
1096
+ },
1097
+ {
1098
+ "loss": 0.943619384765625,
1099
+ "grad_norm": 5.894584655761719,
1100
+ "learning_rate": 4.264210526315789e-06,
1101
+ "epoch": 1.5170831208567057,
1102
+ "step": 5950
1103
+ },
1104
+ {
1105
+ "loss": 0.964720458984375,
1106
+ "grad_norm": 5.264230251312256,
1107
+ "learning_rate": 4.211578947368422e-06,
1108
+ "epoch": 1.5298317185109638,
1109
+ "step": 6000
1110
+ },
1111
+ {
1112
+ "eval_loss": 0.6845850944519043,
1113
+ "eval_NanoMSMARCO_maxsim_accuracy@1": 0.16,
1114
+ "eval_NanoMSMARCO_maxsim_accuracy@3": 0.32,
1115
+ "eval_NanoMSMARCO_maxsim_accuracy@5": 0.42,
1116
+ "eval_NanoMSMARCO_maxsim_accuracy@10": 0.7,
1117
+ "eval_NanoMSMARCO_maxsim_precision@1": 0.16,
1118
+ "eval_NanoMSMARCO_maxsim_precision@3": 0.10666666666666666,
1119
+ "eval_NanoMSMARCO_maxsim_precision@5": 0.084,
1120
+ "eval_NanoMSMARCO_maxsim_precision@10": 0.07,
1121
+ "eval_NanoMSMARCO_maxsim_recall@1": 0.16,
1122
+ "eval_NanoMSMARCO_maxsim_recall@3": 0.32,
1123
+ "eval_NanoMSMARCO_maxsim_recall@5": 0.42,
1124
+ "eval_NanoMSMARCO_maxsim_recall@10": 0.7,
1125
+ "eval_NanoMSMARCO_maxsim_ndcg@10": 0.3858968432351718,
1126
+ "eval_NanoMSMARCO_maxsim_mrr@10": 0.292015873015873,
1127
+ "eval_NanoMSMARCO_maxsim_map@100": 0.3048587220806822,
1128
+ "eval_NanoNQ_maxsim_accuracy@1": 0.28,
1129
+ "eval_NanoNQ_maxsim_accuracy@3": 0.4,
1130
+ "eval_NanoNQ_maxsim_accuracy@5": 0.48,
1131
+ "eval_NanoNQ_maxsim_accuracy@10": 0.66,
1132
+ "eval_NanoNQ_maxsim_precision@1": 0.28,
1133
+ "eval_NanoNQ_maxsim_precision@3": 0.13333333333333333,
1134
+ "eval_NanoNQ_maxsim_precision@5": 0.09600000000000002,
1135
+ "eval_NanoNQ_maxsim_precision@10": 0.066,
1136
+ "eval_NanoNQ_maxsim_recall@1": 0.27,
1137
+ "eval_NanoNQ_maxsim_recall@3": 0.39,
1138
+ "eval_NanoNQ_maxsim_recall@5": 0.46,
1139
+ "eval_NanoNQ_maxsim_recall@10": 0.61,
1140
+ "eval_NanoNQ_maxsim_ndcg@10": 0.43040854986709853,
1141
+ "eval_NanoNQ_maxsim_mrr@10": 0.3837460317460317,
1142
+ "eval_NanoNQ_maxsim_map@100": 0.38461468064678456,
1143
+ "eval_NanoFiQA2018_maxsim_accuracy@1": 0.26,
1144
+ "eval_NanoFiQA2018_maxsim_accuracy@3": 0.4,
1145
+ "eval_NanoFiQA2018_maxsim_accuracy@5": 0.48,
1146
+ "eval_NanoFiQA2018_maxsim_accuracy@10": 0.58,
1147
+ "eval_NanoFiQA2018_maxsim_precision@1": 0.26,
1148
+ "eval_NanoFiQA2018_maxsim_precision@3": 0.18,
1149
+ "eval_NanoFiQA2018_maxsim_precision@5": 0.132,
1150
+ "eval_NanoFiQA2018_maxsim_precision@10": 0.08,
1151
+ "eval_NanoFiQA2018_maxsim_recall@1": 0.12285714285714285,
1152
+ "eval_NanoFiQA2018_maxsim_recall@3": 0.251047619047619,
1153
+ "eval_NanoFiQA2018_maxsim_recall@5": 0.3117142857142857,
1154
+ "eval_NanoFiQA2018_maxsim_recall@10": 0.38704761904761903,
1155
+ "eval_NanoFiQA2018_maxsim_ndcg@10": 0.3025488588070337,
1156
+ "eval_NanoFiQA2018_maxsim_mrr@10": 0.36041269841269835,
1157
+ "eval_NanoFiQA2018_maxsim_map@100": 0.24423869572488793,
1158
+ "eval_NanoBEIR_mean_maxsim_accuracy@1": 0.23333333333333336,
1159
+ "eval_NanoBEIR_mean_maxsim_accuracy@3": 0.37333333333333335,
1160
+ "eval_NanoBEIR_mean_maxsim_accuracy@5": 0.45999999999999996,
1161
+ "eval_NanoBEIR_mean_maxsim_accuracy@10": 0.6466666666666666,
1162
+ "eval_NanoBEIR_mean_maxsim_precision@1": 0.23333333333333336,
1163
+ "eval_NanoBEIR_mean_maxsim_precision@3": 0.13999999999999999,
1164
+ "eval_NanoBEIR_mean_maxsim_precision@5": 0.10400000000000002,
1165
+ "eval_NanoBEIR_mean_maxsim_precision@10": 0.07200000000000001,
1166
+ "eval_NanoBEIR_mean_maxsim_recall@1": 0.1842857142857143,
1167
+ "eval_NanoBEIR_mean_maxsim_recall@3": 0.32034920634920633,
1168
+ "eval_NanoBEIR_mean_maxsim_recall@5": 0.3972380952380952,
1169
+ "eval_NanoBEIR_mean_maxsim_recall@10": 0.5656825396825397,
1170
+ "eval_NanoBEIR_mean_maxsim_ndcg@10": 0.3729514173031014,
1171
+ "eval_NanoBEIR_mean_maxsim_mrr@10": 0.34539153439153436,
1172
+ "eval_NanoBEIR_mean_maxsim_map@100": 0.31123736615078484,
1173
+ "eval_runtime": 3.7136,
1174
+ "eval_samples_per_second": 275.74,
1175
+ "eval_steps_per_second": 8.617,
1176
+ "epoch": 1.5298317185109638,
1177
+ "step": 6000
1178
+ },
1179
+ {
1180
+ "loss": 0.9527646636962891,
1181
+ "grad_norm": 5.419942855834961,
1182
+ "learning_rate": 4.158947368421053e-06,
1183
+ "epoch": 1.5425803161652218,
1184
+ "step": 6050
1185
+ },
1186
+ {
1187
+ "loss": 0.9358503723144531,
1188
+ "grad_norm": 5.1898298263549805,
1189
+ "learning_rate": 4.106315789473684e-06,
1190
+ "epoch": 1.55532891381948,
1191
+ "step": 6100
1192
+ },
1193
+ {
1194
+ "loss": 0.9409062957763672,
1195
+ "grad_norm": 5.944079875946045,
1196
+ "learning_rate": 4.0536842105263166e-06,
1197
+ "epoch": 1.5680775114737378,
1198
+ "step": 6150
1199
+ },
1200
+ {
1201
+ "loss": 0.9534385681152344,
1202
+ "grad_norm": 5.932693958282471,
1203
+ "learning_rate": 4.001052631578947e-06,
1204
+ "epoch": 1.580826109127996,
1205
+ "step": 6200
1206
+ },
1207
+ {
1208
+ "loss": 0.9594136047363281,
1209
+ "grad_norm": 5.8394389152526855,
1210
+ "learning_rate": 3.948421052631579e-06,
1211
+ "epoch": 1.5935747067822539,
1212
+ "step": 6250
1213
+ },
1214
+ {
1215
+ "loss": 0.9496177673339844,
1216
+ "grad_norm": 5.904182434082031,
1217
+ "learning_rate": 3.8957894736842115e-06,
1218
+ "epoch": 1.606323304436512,
1219
+ "step": 6300
1220
+ },
1221
+ {
1222
+ "loss": 0.9397685241699218,
1223
+ "grad_norm": 6.152402877807617,
1224
+ "learning_rate": 3.843157894736842e-06,
1225
+ "epoch": 1.61907190209077,
1226
+ "step": 6350
1227
+ },
1228
+ {
1229
+ "loss": 0.9368280029296875,
1230
+ "grad_norm": 5.958895683288574,
1231
+ "learning_rate": 3.790526315789474e-06,
1232
+ "epoch": 1.631820499745028,
1233
+ "step": 6400
1234
+ },
1235
+ {
1236
+ "loss": 0.9481920623779296,
1237
+ "grad_norm": 5.238616466522217,
1238
+ "learning_rate": 3.7378947368421055e-06,
1239
+ "epoch": 1.6445690973992861,
1240
+ "step": 6450
1241
+ },
1242
+ {
1243
+ "loss": 0.9392526245117188,
1244
+ "grad_norm": 6.374394416809082,
1245
+ "learning_rate": 3.685263157894737e-06,
1246
+ "epoch": 1.657317695053544,
1247
+ "step": 6500
1248
+ },
1249
+ {
1250
+ "loss": 0.9600529479980469,
1251
+ "grad_norm": 6.4856276512146,
1252
+ "learning_rate": 3.632631578947369e-06,
1253
+ "epoch": 1.6700662927078023,
1254
+ "step": 6550
1255
+ },
1256
+ {
1257
+ "loss": 0.9296820068359375,
1258
+ "grad_norm": 5.436391830444336,
1259
+ "learning_rate": 3.58e-06,
1260
+ "epoch": 1.6828148903620601,
1261
+ "step": 6600
1262
+ },
1263
+ {
1264
+ "loss": 0.9262808227539062,
1265
+ "grad_norm": 5.286633014678955,
1266
+ "learning_rate": 3.527368421052632e-06,
1267
+ "epoch": 1.6955634880163182,
1268
+ "step": 6650
1269
+ },
1270
+ {
1271
+ "loss": 0.9391435241699219,
1272
+ "grad_norm": 6.064976215362549,
1273
+ "learning_rate": 3.4747368421052637e-06,
1274
+ "epoch": 1.7083120856705762,
1275
+ "step": 6700
1276
+ },
1277
+ {
1278
+ "loss": 0.9423677825927734,
1279
+ "grad_norm": 5.806890487670898,
1280
+ "learning_rate": 3.422105263157895e-06,
1281
+ "epoch": 1.7210606833248343,
1282
+ "step": 6750
1283
+ },
1284
+ {
1285
+ "loss": 0.9356842803955078,
1286
+ "grad_norm": 5.708329200744629,
1287
+ "learning_rate": 3.369473684210526e-06,
1288
+ "epoch": 1.7338092809790924,
1289
+ "step": 6800
1290
+ },
1291
+ {
1292
+ "loss": 0.9470310974121093,
1293
+ "grad_norm": 5.327975749969482,
1294
+ "learning_rate": 3.3168421052631582e-06,
1295
+ "epoch": 1.7465578786333502,
1296
+ "step": 6850
1297
+ },
1298
+ {
1299
+ "loss": 0.9314807891845703,
1300
+ "grad_norm": 5.52776575088501,
1301
+ "learning_rate": 3.26421052631579e-06,
1302
+ "epoch": 1.7593064762876085,
1303
+ "step": 6900
1304
+ },
1305
+ {
1306
+ "loss": 0.9288822937011719,
1307
+ "grad_norm": 6.206039905548096,
1308
+ "learning_rate": 3.211578947368421e-06,
1309
+ "epoch": 1.7720550739418663,
1310
+ "step": 6950
1311
+ },
1312
+ {
1313
+ "loss": 0.9206436920166016,
1314
+ "grad_norm": 5.821423530578613,
1315
+ "learning_rate": 3.158947368421053e-06,
1316
+ "epoch": 1.7848036715961244,
1317
+ "step": 7000
1318
+ },
1319
+ {
1320
+ "loss": 0.9493258666992187,
1321
+ "grad_norm": 5.648779392242432,
1322
+ "learning_rate": 3.1063157894736843e-06,
1323
+ "epoch": 1.7975522692503825,
1324
+ "step": 7050
1325
+ },
1326
+ {
1327
+ "loss": 0.953025131225586,
1328
+ "grad_norm": 6.218502998352051,
1329
+ "learning_rate": 3.053684210526316e-06,
1330
+ "epoch": 1.8103008669046405,
1331
+ "step": 7100
1332
+ },
1333
+ {
1334
+ "loss": 0.9378658294677734,
1335
+ "grad_norm": 6.421555042266846,
1336
+ "learning_rate": 3.001052631578948e-06,
1337
+ "epoch": 1.8230494645588986,
1338
+ "step": 7150
1339
+ },
1340
+ {
1341
+ "loss": 0.9349431610107422,
1342
+ "grad_norm": 5.718181610107422,
1343
+ "learning_rate": 2.9484210526315792e-06,
1344
+ "epoch": 1.8357980622131564,
1345
+ "step": 7200
1346
+ },
1347
+ {
1348
+ "loss": 0.9017268371582031,
1349
+ "grad_norm": 6.304521560668945,
1350
+ "learning_rate": 2.895789473684211e-06,
1351
+ "epoch": 1.8485466598674147,
1352
+ "step": 7250
1353
+ },
1354
+ {
1355
+ "loss": 0.9407036590576172,
1356
+ "grad_norm": 6.817096710205078,
1357
+ "learning_rate": 2.843157894736842e-06,
1358
+ "epoch": 1.8612952575216726,
1359
+ "step": 7300
1360
+ },
1361
+ {
1362
+ "loss": 0.9158139038085937,
1363
+ "grad_norm": 6.281953811645508,
1364
+ "learning_rate": 2.790526315789474e-06,
1365
+ "epoch": 1.8740438551759306,
1366
+ "step": 7350
1367
+ },
1368
+ {
1369
+ "loss": 0.941224594116211,
1370
+ "grad_norm": 5.203763008117676,
1371
+ "learning_rate": 2.7378947368421054e-06,
1372
+ "epoch": 1.8867924528301887,
1373
+ "step": 7400
1374
+ },
1375
+ {
1376
+ "loss": 0.923984375,
1377
+ "grad_norm": 4.658481597900391,
1378
+ "learning_rate": 2.685263157894737e-06,
1379
+ "epoch": 1.8995410504844465,
1380
+ "step": 7450
1381
+ },
1382
+ {
1383
+ "loss": 0.9257674407958985,
1384
+ "grad_norm": 4.8184075355529785,
1385
+ "learning_rate": 2.632631578947369e-06,
1386
+ "epoch": 1.9122896481387048,
1387
+ "step": 7500
1388
+ },
1389
+ {
1390
+ "loss": 0.9456035614013671,
1391
+ "grad_norm": 5.411379337310791,
1392
+ "learning_rate": 2.5800000000000003e-06,
1393
+ "epoch": 1.9250382457929627,
1394
+ "step": 7550
1395
+ },
1396
+ {
1397
+ "loss": 0.9428106689453125,
1398
+ "grad_norm": 5.237967014312744,
1399
+ "learning_rate": 2.5273684210526315e-06,
1400
+ "epoch": 1.937786843447221,
1401
+ "step": 7600
1402
+ },
1403
+ {
1404
+ "loss": 0.9630010986328125,
1405
+ "grad_norm": 5.7921857833862305,
1406
+ "learning_rate": 2.4747368421052636e-06,
1407
+ "epoch": 1.9505354411014788,
1408
+ "step": 7650
1409
+ },
1410
+ {
1411
+ "loss": 0.945229263305664,
1412
+ "grad_norm": 5.247952461242676,
1413
+ "learning_rate": 2.422105263157895e-06,
1414
+ "epoch": 1.9632840387557369,
1415
+ "step": 7700
1416
+ },
1417
+ {
1418
+ "loss": 0.9221783447265625,
1419
+ "grad_norm": 5.966209411621094,
1420
+ "learning_rate": 2.3694736842105264e-06,
1421
+ "epoch": 1.976032636409995,
1422
+ "step": 7750
1423
+ },
1424
+ {
1425
+ "loss": 0.9287884521484375,
1426
+ "grad_norm": 5.2588982582092285,
1427
+ "learning_rate": 2.316842105263158e-06,
1428
+ "epoch": 1.9887812340642528,
1429
+ "step": 7800
1430
+ },
1431
+ {
1432
+ "loss": 0.9349156188964843,
1433
+ "grad_norm": 5.415347576141357,
1434
+ "learning_rate": 2.2642105263157897e-06,
1435
+ "epoch": 2.001529831718511,
1436
+ "step": 7850
1437
+ },
1438
+ {
1439
+ "loss": 0.9354501342773438,
1440
+ "grad_norm": 5.250112056732178,
1441
+ "learning_rate": 2.2115789473684213e-06,
1442
+ "epoch": 2.014278429372769,
1443
+ "step": 7900
1444
+ },
1445
+ {
1446
+ "loss": 0.9040863800048828,
1447
+ "grad_norm": 5.810637474060059,
1448
+ "learning_rate": 2.1589473684210525e-06,
1449
+ "epoch": 2.027027027027027,
1450
+ "step": 7950
1451
+ },
1452
+ {
1453
+ "loss": 0.9092691802978515,
1454
+ "grad_norm": 5.8565473556518555,
1455
+ "learning_rate": 2.106315789473684e-06,
1456
+ "epoch": 2.039775624681285,
1457
+ "step": 8000
1458
+ },
1459
+ {
1460
+ "eval_loss": 0.6677519083023071,
1461
+ "eval_NanoMSMARCO_maxsim_accuracy@1": 0.14,
1462
+ "eval_NanoMSMARCO_maxsim_accuracy@3": 0.32,
1463
+ "eval_NanoMSMARCO_maxsim_accuracy@5": 0.42,
1464
+ "eval_NanoMSMARCO_maxsim_accuracy@10": 0.7,
1465
+ "eval_NanoMSMARCO_maxsim_precision@1": 0.14,
1466
+ "eval_NanoMSMARCO_maxsim_precision@3": 0.10666666666666666,
1467
+ "eval_NanoMSMARCO_maxsim_precision@5": 0.084,
1468
+ "eval_NanoMSMARCO_maxsim_precision@10": 0.07,
1469
+ "eval_NanoMSMARCO_maxsim_recall@1": 0.14,
1470
+ "eval_NanoMSMARCO_maxsim_recall@3": 0.32,
1471
+ "eval_NanoMSMARCO_maxsim_recall@5": 0.42,
1472
+ "eval_NanoMSMARCO_maxsim_recall@10": 0.7,
1473
+ "eval_NanoMSMARCO_maxsim_ndcg@10": 0.3805043522757907,
1474
+ "eval_NanoMSMARCO_maxsim_mrr@10": 0.28398412698412695,
1475
+ "eval_NanoMSMARCO_maxsim_map@100": 0.296316419715738,
1476
+ "eval_NanoNQ_maxsim_accuracy@1": 0.28,
1477
+ "eval_NanoNQ_maxsim_accuracy@3": 0.4,
1478
+ "eval_NanoNQ_maxsim_accuracy@5": 0.48,
1479
+ "eval_NanoNQ_maxsim_accuracy@10": 0.68,
1480
+ "eval_NanoNQ_maxsim_precision@1": 0.28,
1481
+ "eval_NanoNQ_maxsim_precision@3": 0.13333333333333333,
1482
+ "eval_NanoNQ_maxsim_precision@5": 0.09600000000000002,
1483
+ "eval_NanoNQ_maxsim_precision@10": 0.068,
1484
+ "eval_NanoNQ_maxsim_recall@1": 0.27,
1485
+ "eval_NanoNQ_maxsim_recall@3": 0.39,
1486
+ "eval_NanoNQ_maxsim_recall@5": 0.47,
1487
+ "eval_NanoNQ_maxsim_recall@10": 0.63,
1488
+ "eval_NanoNQ_maxsim_ndcg@10": 0.4370852546051692,
1489
+ "eval_NanoNQ_maxsim_mrr@10": 0.38649206349206344,
1490
+ "eval_NanoNQ_maxsim_map@100": 0.3855877539952645,
1491
+ "eval_NanoFiQA2018_maxsim_accuracy@1": 0.26,
1492
+ "eval_NanoFiQA2018_maxsim_accuracy@3": 0.4,
1493
+ "eval_NanoFiQA2018_maxsim_accuracy@5": 0.46,
1494
+ "eval_NanoFiQA2018_maxsim_accuracy@10": 0.56,
1495
+ "eval_NanoFiQA2018_maxsim_precision@1": 0.26,
1496
+ "eval_NanoFiQA2018_maxsim_precision@3": 0.18,
1497
+ "eval_NanoFiQA2018_maxsim_precision@5": 0.128,
1498
+ "eval_NanoFiQA2018_maxsim_precision@10": 0.08,
1499
+ "eval_NanoFiQA2018_maxsim_recall@1": 0.12285714285714285,
1500
+ "eval_NanoFiQA2018_maxsim_recall@3": 0.251047619047619,
1501
+ "eval_NanoFiQA2018_maxsim_recall@5": 0.3017142857142857,
1502
+ "eval_NanoFiQA2018_maxsim_recall@10": 0.381047619047619,
1503
+ "eval_NanoFiQA2018_maxsim_ndcg@10": 0.29954379794287644,
1504
+ "eval_NanoFiQA2018_maxsim_mrr@10": 0.3564126984126984,
1505
+ "eval_NanoFiQA2018_maxsim_map@100": 0.24427189079021108,
1506
+ "eval_NanoBEIR_mean_maxsim_accuracy@1": 0.22666666666666668,
1507
+ "eval_NanoBEIR_mean_maxsim_accuracy@3": 0.37333333333333335,
1508
+ "eval_NanoBEIR_mean_maxsim_accuracy@5": 0.4533333333333333,
1509
+ "eval_NanoBEIR_mean_maxsim_accuracy@10": 0.6466666666666666,
1510
+ "eval_NanoBEIR_mean_maxsim_precision@1": 0.22666666666666668,
1511
+ "eval_NanoBEIR_mean_maxsim_precision@3": 0.13999999999999999,
1512
+ "eval_NanoBEIR_mean_maxsim_precision@5": 0.10266666666666668,
1513
+ "eval_NanoBEIR_mean_maxsim_precision@10": 0.07266666666666667,
1514
+ "eval_NanoBEIR_mean_maxsim_recall@1": 0.17761904761904765,
1515
+ "eval_NanoBEIR_mean_maxsim_recall@3": 0.32034920634920633,
1516
+ "eval_NanoBEIR_mean_maxsim_recall@5": 0.3972380952380952,
1517
+ "eval_NanoBEIR_mean_maxsim_recall@10": 0.5703492063492064,
1518
+ "eval_NanoBEIR_mean_maxsim_ndcg@10": 0.3723778016079455,
1519
+ "eval_NanoBEIR_mean_maxsim_mrr@10": 0.34229629629629627,
1520
+ "eval_NanoBEIR_mean_maxsim_map@100": 0.30872535483373786,
1521
+ "eval_runtime": 3.9834,
1522
+ "eval_samples_per_second": 257.069,
1523
+ "eval_steps_per_second": 8.033,
1524
+ "epoch": 2.039775624681285,
1525
+ "step": 8000
1526
+ },
1527
+ {
1528
+ "loss": 0.9139494323730468,
1529
+ "grad_norm": 5.329063892364502,
1530
+ "learning_rate": 2.0536842105263162e-06,
1531
+ "epoch": 2.052524222335543,
1532
+ "step": 8050
1533
+ },
1534
+ {
1535
+ "loss": 0.9532962799072265,
1536
+ "grad_norm": 6.04278039932251,
1537
+ "learning_rate": 2.0010526315789474e-06,
1538
+ "epoch": 2.065272819989801,
1539
+ "step": 8100
1540
+ },
1541
+ {
1542
+ "loss": 0.9313519287109375,
1543
+ "grad_norm": 5.422224521636963,
1544
+ "learning_rate": 1.948421052631579e-06,
1545
+ "epoch": 2.078021417644059,
1546
+ "step": 8150
1547
+ },
1548
+ {
1549
+ "loss": 0.9206269836425781,
1550
+ "grad_norm": 5.794032573699951,
1551
+ "learning_rate": 1.8957894736842105e-06,
1552
+ "epoch": 2.0907700152983173,
1553
+ "step": 8200
1554
+ },
1555
+ {
1556
+ "loss": 0.9299026489257812,
1557
+ "grad_norm": 5.899726390838623,
1558
+ "learning_rate": 1.8431578947368424e-06,
1559
+ "epoch": 2.103518612952575,
1560
+ "step": 8250
1561
+ },
1562
+ {
1563
+ "loss": 0.91742431640625,
1564
+ "grad_norm": 5.519539833068848,
1565
+ "learning_rate": 1.790526315789474e-06,
1566
+ "epoch": 2.1162672106068334,
1567
+ "step": 8300
1568
+ },
1569
+ {
1570
+ "loss": 0.9038812255859375,
1571
+ "grad_norm": 5.41025447845459,
1572
+ "learning_rate": 1.7378947368421054e-06,
1573
+ "epoch": 2.1290158082610913,
1574
+ "step": 8350
1575
+ },
1576
+ {
1577
+ "loss": 0.9091542053222657,
1578
+ "grad_norm": 5.703662872314453,
1579
+ "learning_rate": 1.685263157894737e-06,
1580
+ "epoch": 2.141764405915349,
1581
+ "step": 8400
1582
+ },
1583
+ {
1584
+ "loss": 0.9085771942138672,
1585
+ "grad_norm": 6.512147426605225,
1586
+ "learning_rate": 1.6326315789473685e-06,
1587
+ "epoch": 2.1545130035696074,
1588
+ "step": 8450
1589
+ },
1590
+ {
1591
+ "loss": 0.9605217742919921,
1592
+ "grad_norm": 6.193598747253418,
1593
+ "learning_rate": 1.5800000000000001e-06,
1594
+ "epoch": 2.1672616012238652,
1595
+ "step": 8500
1596
+ },
1597
+ {
1598
+ "loss": 0.9296475219726562,
1599
+ "grad_norm": 5.669243812561035,
1600
+ "learning_rate": 1.5273684210526318e-06,
1601
+ "epoch": 2.1800101988781235,
1602
+ "step": 8550
1603
+ },
1604
+ {
1605
+ "loss": 0.9195587921142578,
1606
+ "grad_norm": 5.914945602416992,
1607
+ "learning_rate": 1.4747368421052632e-06,
1608
+ "epoch": 2.1927587965323814,
1609
+ "step": 8600
1610
+ },
1611
+ {
1612
+ "loss": 0.9145635223388672,
1613
+ "grad_norm": 5.360496997833252,
1614
+ "learning_rate": 1.422105263157895e-06,
1615
+ "epoch": 2.2055073941866397,
1616
+ "step": 8650
1617
+ },
1618
+ {
1619
+ "loss": 0.9129716491699219,
1620
+ "grad_norm": 5.839056015014648,
1621
+ "learning_rate": 1.3694736842105262e-06,
1622
+ "epoch": 2.2182559918408975,
1623
+ "step": 8700
1624
+ },
1625
+ {
1626
+ "loss": 0.9037160491943359,
1627
+ "grad_norm": 5.070700645446777,
1628
+ "learning_rate": 1.316842105263158e-06,
1629
+ "epoch": 2.2310045894951553,
1630
+ "step": 8750
1631
+ },
1632
+ {
1633
+ "loss": 0.93569580078125,
1634
+ "grad_norm": 6.184298992156982,
1635
+ "learning_rate": 1.2642105263157895e-06,
1636
+ "epoch": 2.2437531871494136,
1637
+ "step": 8800
1638
+ },
1639
+ {
1640
+ "loss": 0.9313115692138672,
1641
+ "grad_norm": 5.569518089294434,
1642
+ "learning_rate": 1.2115789473684212e-06,
1643
+ "epoch": 2.2565017848036715,
1644
+ "step": 8850
1645
+ },
1646
+ {
1647
+ "loss": 0.9179414367675781,
1648
+ "grad_norm": 5.4879608154296875,
1649
+ "learning_rate": 1.1589473684210526e-06,
1650
+ "epoch": 2.2692503824579298,
1651
+ "step": 8900
1652
+ },
1653
+ {
1654
+ "loss": 0.9597310638427734,
1655
+ "grad_norm": 5.215628147125244,
1656
+ "learning_rate": 1.1063157894736842e-06,
1657
+ "epoch": 2.2819989801121876,
1658
+ "step": 8950
1659
+ },
1660
+ {
1661
+ "loss": 0.9274011993408203,
1662
+ "grad_norm": 5.446476459503174,
1663
+ "learning_rate": 1.0536842105263159e-06,
1664
+ "epoch": 2.2947475777664454,
1665
+ "step": 9000
1666
+ },
1667
+ {
1668
+ "loss": 0.9092082977294922,
1669
+ "grad_norm": 6.1666412353515625,
1670
+ "learning_rate": 1.0010526315789475e-06,
1671
+ "epoch": 2.3074961754207037,
1672
+ "step": 9050
1673
+ },
1674
+ {
1675
+ "loss": 0.9072042083740235,
1676
+ "grad_norm": 5.2612996101379395,
1677
+ "learning_rate": 9.48421052631579e-07,
1678
+ "epoch": 2.3202447730749616,
1679
+ "step": 9100
1680
+ },
1681
+ {
1682
+ "loss": 0.9166269683837891,
1683
+ "grad_norm": 5.176909446716309,
1684
+ "learning_rate": 8.957894736842106e-07,
1685
+ "epoch": 2.33299337072922,
1686
+ "step": 9150
1687
+ },
1688
+ {
1689
+ "loss": 0.897437744140625,
1690
+ "grad_norm": 5.641931533813477,
1691
+ "learning_rate": 8.431578947368421e-07,
1692
+ "epoch": 2.3457419683834777,
1693
+ "step": 9200
1694
+ },
1695
+ {
1696
+ "loss": 0.916630859375,
1697
+ "grad_norm": 6.095193386077881,
1698
+ "learning_rate": 7.905263157894738e-07,
1699
+ "epoch": 2.358490566037736,
1700
+ "step": 9250
1701
+ },
1702
+ {
1703
+ "loss": 0.9280863952636719,
1704
+ "grad_norm": 6.095040321350098,
1705
+ "learning_rate": 7.378947368421054e-07,
1706
+ "epoch": 2.371239163691994,
1707
+ "step": 9300
1708
+ },
1709
+ {
1710
+ "loss": 0.9291934204101563,
1711
+ "grad_norm": 5.4914326667785645,
1712
+ "learning_rate": 6.852631578947369e-07,
1713
+ "epoch": 2.383987761346252,
1714
+ "step": 9350
1715
+ },
1716
+ {
1717
+ "loss": 0.9166167449951171,
1718
+ "grad_norm": 5.818508625030518,
1719
+ "learning_rate": 6.326315789473684e-07,
1720
+ "epoch": 2.39673635900051,
1721
+ "step": 9400
1722
+ },
1723
+ {
1724
+ "loss": 0.9265275573730469,
1725
+ "grad_norm": 5.380603790283203,
1726
+ "learning_rate": 5.800000000000001e-07,
1727
+ "epoch": 2.409484956654768,
1728
+ "step": 9450
1729
+ },
1730
+ {
1731
+ "loss": 0.9142233276367188,
1732
+ "grad_norm": 6.24432897567749,
1733
+ "learning_rate": 5.273684210526316e-07,
1734
+ "epoch": 2.422233554309026,
1735
+ "step": 9500
1736
+ },
1737
+ {
1738
+ "loss": 0.9131135559082031,
1739
+ "grad_norm": 6.197072505950928,
1740
+ "learning_rate": 4.747368421052632e-07,
1741
+ "epoch": 2.434982151963284,
1742
+ "step": 9550
1743
+ },
1744
+ {
1745
+ "loss": 0.8901510620117188,
1746
+ "grad_norm": 6.033638954162598,
1747
+ "learning_rate": 4.221052631578948e-07,
1748
+ "epoch": 2.4477307496175422,
1749
+ "step": 9600
1750
+ },
1751
+ {
1752
+ "loss": 0.9464206695556641,
1753
+ "grad_norm": 4.991445541381836,
1754
+ "learning_rate": 3.6947368421052634e-07,
1755
+ "epoch": 2.4604793472718,
1756
+ "step": 9650
1757
+ },
1758
+ {
1759
+ "loss": 0.9166670227050782,
1760
+ "grad_norm": 5.999915599822998,
1761
+ "learning_rate": 3.1684210526315787e-07,
1762
+ "epoch": 2.473227944926058,
1763
+ "step": 9700
1764
+ },
1765
+ {
1766
+ "loss": 0.9121665191650391,
1767
+ "grad_norm": 5.635122299194336,
1768
+ "learning_rate": 2.6421052631578946e-07,
1769
+ "epoch": 2.485976542580316,
1770
+ "step": 9750
1771
+ },
1772
+ {
1773
+ "loss": 0.9216783142089844,
1774
+ "grad_norm": 5.740274429321289,
1775
+ "learning_rate": 2.1157894736842107e-07,
1776
+ "epoch": 2.498725140234574,
1777
+ "step": 9800
1778
+ },
1779
+ {
1780
+ "loss": 0.9245352172851562,
1781
+ "grad_norm": 5.6520280838012695,
1782
+ "learning_rate": 1.5894736842105265e-07,
1783
+ "epoch": 2.5114737378888323,
1784
+ "step": 9850
1785
+ },
1786
+ {
1787
+ "loss": 0.8962958526611328,
1788
+ "grad_norm": 4.8676042556762695,
1789
+ "learning_rate": 1.0631578947368421e-07,
1790
+ "epoch": 2.52422233554309,
1791
+ "step": 9900
1792
+ },
1793
+ {
1794
+ "loss": 0.9255727386474609,
1795
+ "grad_norm": 5.440561294555664,
1796
+ "learning_rate": 5.3684210526315794e-08,
1797
+ "epoch": 2.5369709331973485,
1798
+ "step": 9950
1799
+ },
1800
+ {
1801
+ "loss": 0.9120568084716797,
1802
+ "grad_norm": 6.291442394256592,
1803
+ "learning_rate": 1.0526315789473685e-09,
1804
+ "epoch": 2.5497195308516063,
1805
+ "step": 10000
1806
+ },
1807
+ {
1808
+ "eval_loss": 0.6629586219787598,
1809
+ "eval_NanoMSMARCO_maxsim_accuracy@1": 0.12,
1810
+ "eval_NanoMSMARCO_maxsim_accuracy@3": 0.32,
1811
+ "eval_NanoMSMARCO_maxsim_accuracy@5": 0.44,
1812
+ "eval_NanoMSMARCO_maxsim_accuracy@10": 0.72,
1813
+ "eval_NanoMSMARCO_maxsim_precision@1": 0.12,
1814
+ "eval_NanoMSMARCO_maxsim_precision@3": 0.10666666666666666,
1815
+ "eval_NanoMSMARCO_maxsim_precision@5": 0.08800000000000001,
1816
+ "eval_NanoMSMARCO_maxsim_precision@10": 0.07200000000000001,
1817
+ "eval_NanoMSMARCO_maxsim_recall@1": 0.12,
1818
+ "eval_NanoMSMARCO_maxsim_recall@3": 0.32,
1819
+ "eval_NanoMSMARCO_maxsim_recall@5": 0.44,
1820
+ "eval_NanoMSMARCO_maxsim_recall@10": 0.72,
1821
+ "eval_NanoMSMARCO_maxsim_ndcg@10": 0.37958582778462513,
1822
+ "eval_NanoMSMARCO_maxsim_mrr@10": 0.27673015873015866,
1823
+ "eval_NanoMSMARCO_maxsim_map@100": 0.2873170010783349,
1824
+ "eval_NanoNQ_maxsim_accuracy@1": 0.28,
1825
+ "eval_NanoNQ_maxsim_accuracy@3": 0.4,
1826
+ "eval_NanoNQ_maxsim_accuracy@5": 0.48,
1827
+ "eval_NanoNQ_maxsim_accuracy@10": 0.68,
1828
+ "eval_NanoNQ_maxsim_precision@1": 0.28,
1829
+ "eval_NanoNQ_maxsim_precision@3": 0.13333333333333333,
1830
+ "eval_NanoNQ_maxsim_precision@5": 0.09600000000000002,
1831
+ "eval_NanoNQ_maxsim_precision@10": 0.068,
1832
+ "eval_NanoNQ_maxsim_recall@1": 0.27,
1833
+ "eval_NanoNQ_maxsim_recall@3": 0.39,
1834
+ "eval_NanoNQ_maxsim_recall@5": 0.47,
1835
+ "eval_NanoNQ_maxsim_recall@10": 0.63,
1836
+ "eval_NanoNQ_maxsim_ndcg@10": 0.4370217580491872,
1837
+ "eval_NanoNQ_maxsim_mrr@10": 0.3862142857142857,
1838
+ "eval_NanoNQ_maxsim_map@100": 0.38546703015776257,
1839
+ "eval_NanoFiQA2018_maxsim_accuracy@1": 0.24,
1840
+ "eval_NanoFiQA2018_maxsim_accuracy@3": 0.38,
1841
+ "eval_NanoFiQA2018_maxsim_accuracy@5": 0.48,
1842
+ "eval_NanoFiQA2018_maxsim_accuracy@10": 0.58,
1843
+ "eval_NanoFiQA2018_maxsim_precision@1": 0.24,
1844
+ "eval_NanoFiQA2018_maxsim_precision@3": 0.1733333333333333,
1845
+ "eval_NanoFiQA2018_maxsim_precision@5": 0.132,
1846
+ "eval_NanoFiQA2018_maxsim_precision@10": 0.084,
1847
+ "eval_NanoFiQA2018_maxsim_recall@1": 0.11285714285714285,
1848
+ "eval_NanoFiQA2018_maxsim_recall@3": 0.24104761904761904,
1849
+ "eval_NanoFiQA2018_maxsim_recall@5": 0.3117142857142857,
1850
+ "eval_NanoFiQA2018_maxsim_recall@10": 0.411047619047619,
1851
+ "eval_NanoFiQA2018_maxsim_ndcg@10": 0.30102991380311517,
1852
+ "eval_NanoFiQA2018_maxsim_mrr@10": 0.3407460317460317,
1853
+ "eval_NanoFiQA2018_maxsim_map@100": 0.23583308989441118,
1854
+ "eval_NanoBEIR_mean_maxsim_accuracy@1": 0.21333333333333335,
1855
+ "eval_NanoBEIR_mean_maxsim_accuracy@3": 0.3666666666666667,
1856
+ "eval_NanoBEIR_mean_maxsim_accuracy@5": 0.4666666666666666,
1857
+ "eval_NanoBEIR_mean_maxsim_accuracy@10": 0.66,
1858
+ "eval_NanoBEIR_mean_maxsim_precision@1": 0.21333333333333335,
1859
+ "eval_NanoBEIR_mean_maxsim_precision@3": 0.13777777777777778,
1860
+ "eval_NanoBEIR_mean_maxsim_precision@5": 0.10533333333333335,
1861
+ "eval_NanoBEIR_mean_maxsim_precision@10": 0.07466666666666667,
1862
+ "eval_NanoBEIR_mean_maxsim_recall@1": 0.16761904761904764,
1863
+ "eval_NanoBEIR_mean_maxsim_recall@3": 0.31701587301587303,
1864
+ "eval_NanoBEIR_mean_maxsim_recall@5": 0.4072380952380952,
1865
+ "eval_NanoBEIR_mean_maxsim_recall@10": 0.587015873015873,
1866
+ "eval_NanoBEIR_mean_maxsim_ndcg@10": 0.3725458332123092,
1867
+ "eval_NanoBEIR_mean_maxsim_mrr@10": 0.33456349206349206,
1868
+ "eval_NanoBEIR_mean_maxsim_map@100": 0.30287237371016956,
1869
+ "eval_runtime": 3.9106,
1870
+ "eval_samples_per_second": 261.85,
1871
+ "eval_steps_per_second": 8.183,
1872
+ "epoch": 2.5497195308516063,
1873
+ "step": 10000
1874
+ },
1875
+ {
1876
+ "train_runtime": 933.4176,
1877
+ "train_samples_per_second": 1371.305,
1878
+ "train_steps_per_second": 10.713,
1879
+ "total_flos": 0.0,
1880
+ "train_loss": 1.0466781759500503,
1881
+ "epoch": 2.5497195308516063,
1882
+ "step": 10000
1883
+ }
1884
+ ],
1885
+ "verdict": "WIN",
1886
+ "full_baseline": {
1887
+ "NanoClimateFEVER_maxsim_accuracy@1": 0.16,
1888
+ "NanoClimateFEVER_maxsim_accuracy@3": 0.22,
1889
+ "NanoClimateFEVER_maxsim_accuracy@5": 0.36,
1890
+ "NanoClimateFEVER_maxsim_accuracy@10": 0.44,
1891
+ "NanoClimateFEVER_maxsim_precision@1": 0.16,
1892
+ "NanoClimateFEVER_maxsim_precision@3": 0.07333333333333333,
1893
+ "NanoClimateFEVER_maxsim_precision@5": 0.07200000000000001,
1894
+ "NanoClimateFEVER_maxsim_precision@10": 0.04800000000000001,
1895
+ "NanoClimateFEVER_maxsim_recall@1": 0.05166666666666666,
1896
+ "NanoClimateFEVER_maxsim_recall@3": 0.06833333333333332,
1897
+ "NanoClimateFEVER_maxsim_recall@5": 0.14,
1898
+ "NanoClimateFEVER_maxsim_recall@10": 0.19166666666666668,
1899
+ "NanoClimateFEVER_maxsim_ndcg@10": 0.1430503253366106,
1900
+ "NanoClimateFEVER_maxsim_mrr@10": 0.22544444444444436,
1901
+ "NanoClimateFEVER_maxsim_map@100": 0.10079213378441537,
1902
+ "NanoDBPedia_maxsim_accuracy@1": 0.5,
1903
+ "NanoDBPedia_maxsim_accuracy@3": 0.66,
1904
+ "NanoDBPedia_maxsim_accuracy@5": 0.72,
1905
+ "NanoDBPedia_maxsim_accuracy@10": 0.84,
1906
+ "NanoDBPedia_maxsim_precision@1": 0.5,
1907
+ "NanoDBPedia_maxsim_precision@3": 0.41333333333333333,
1908
+ "NanoDBPedia_maxsim_precision@5": 0.36800000000000005,
1909
+ "NanoDBPedia_maxsim_precision@10": 0.33399999999999996,
1910
+ "NanoDBPedia_maxsim_recall@1": 0.04408091090369132,
1911
+ "NanoDBPedia_maxsim_recall@3": 0.08487585048407303,
1912
+ "NanoDBPedia_maxsim_recall@5": 0.1194462940735006,
1913
+ "NanoDBPedia_maxsim_recall@10": 0.19190492577336216,
1914
+ "NanoDBPedia_maxsim_ndcg@10": 0.39135002435773386,
1915
+ "NanoDBPedia_maxsim_mrr@10": 0.5964126984126984,
1916
+ "NanoDBPedia_maxsim_map@100": 0.27001428971109065,
1917
+ "NanoFEVER_maxsim_accuracy@1": 0.48,
1918
+ "NanoFEVER_maxsim_accuracy@3": 0.62,
1919
+ "NanoFEVER_maxsim_accuracy@5": 0.76,
1920
+ "NanoFEVER_maxsim_accuracy@10": 0.82,
1921
+ "NanoFEVER_maxsim_precision@1": 0.48,
1922
+ "NanoFEVER_maxsim_precision@3": 0.20666666666666664,
1923
+ "NanoFEVER_maxsim_precision@5": 0.16,
1924
+ "NanoFEVER_maxsim_precision@10": 0.088,
1925
+ "NanoFEVER_maxsim_recall@1": 0.46,
1926
+ "NanoFEVER_maxsim_recall@3": 0.59,
1927
+ "NanoFEVER_maxsim_recall@5": 0.74,
1928
+ "NanoFEVER_maxsim_recall@10": 0.81,
1929
+ "NanoFEVER_maxsim_ndcg@10": 0.6309993851065319,
1930
+ "NanoFEVER_maxsim_mrr@10": 0.5839126984126983,
1931
+ "NanoFEVER_maxsim_map@100": 0.5734082716919167,
1932
+ "NanoFiQA2018_maxsim_accuracy@1": 0.22,
1933
+ "NanoFiQA2018_maxsim_accuracy@3": 0.4,
1934
+ "NanoFiQA2018_maxsim_accuracy@5": 0.46,
1935
+ "NanoFiQA2018_maxsim_accuracy@10": 0.54,
1936
+ "NanoFiQA2018_maxsim_precision@1": 0.22,
1937
+ "NanoFiQA2018_maxsim_precision@3": 0.1533333333333333,
1938
+ "NanoFiQA2018_maxsim_precision@5": 0.11600000000000002,
1939
+ "NanoFiQA2018_maxsim_precision@10": 0.07400000000000001,
1940
+ "NanoFiQA2018_maxsim_recall@1": 0.09552380952380952,
1941
+ "NanoFiQA2018_maxsim_recall@3": 0.22885714285714287,
1942
+ "NanoFiQA2018_maxsim_recall@5": 0.2935238095238095,
1943
+ "NanoFiQA2018_maxsim_recall@10": 0.35904761904761906,
1944
+ "NanoFiQA2018_maxsim_ndcg@10": 0.2657905839072199,
1945
+ "NanoFiQA2018_maxsim_mrr@10": 0.3248888888888888,
1946
+ "NanoFiQA2018_maxsim_map@100": 0.19617277017199142,
1947
+ "NanoHotpotQA_maxsim_accuracy@1": 0.66,
1948
+ "NanoHotpotQA_maxsim_accuracy@3": 0.78,
1949
+ "NanoHotpotQA_maxsim_accuracy@5": 0.78,
1950
+ "NanoHotpotQA_maxsim_accuracy@10": 0.88,
1951
+ "NanoHotpotQA_maxsim_precision@1": 0.66,
1952
+ "NanoHotpotQA_maxsim_precision@3": 0.33333333333333326,
1953
+ "NanoHotpotQA_maxsim_precision@5": 0.20800000000000002,
1954
+ "NanoHotpotQA_maxsim_precision@10": 0.11999999999999998,
1955
+ "NanoHotpotQA_maxsim_recall@1": 0.33,
1956
+ "NanoHotpotQA_maxsim_recall@3": 0.5,
1957
+ "NanoHotpotQA_maxsim_recall@5": 0.52,
1958
+ "NanoHotpotQA_maxsim_recall@10": 0.6,
1959
+ "NanoHotpotQA_maxsim_ndcg@10": 0.5766198628107968,
1960
+ "NanoHotpotQA_maxsim_mrr@10": 0.7281904761904763,
1961
+ "NanoHotpotQA_maxsim_map@100": 0.50870551366162,
1962
+ "NanoMSMARCO_maxsim_accuracy@1": 0.26,
1963
+ "NanoMSMARCO_maxsim_accuracy@3": 0.42,
1964
+ "NanoMSMARCO_maxsim_accuracy@5": 0.48,
1965
+ "NanoMSMARCO_maxsim_accuracy@10": 0.66,
1966
+ "NanoMSMARCO_maxsim_precision@1": 0.26,
1967
+ "NanoMSMARCO_maxsim_precision@3": 0.13999999999999999,
1968
+ "NanoMSMARCO_maxsim_precision@5": 0.09600000000000002,
1969
+ "NanoMSMARCO_maxsim_precision@10": 0.066,
1970
+ "NanoMSMARCO_maxsim_recall@1": 0.26,
1971
+ "NanoMSMARCO_maxsim_recall@3": 0.42,
1972
+ "NanoMSMARCO_maxsim_recall@5": 0.48,
1973
+ "NanoMSMARCO_maxsim_recall@10": 0.66,
1974
+ "NanoMSMARCO_maxsim_ndcg@10": 0.4328151442419565,
1975
+ "NanoMSMARCO_maxsim_mrr@10": 0.36385714285714277,
1976
+ "NanoMSMARCO_maxsim_map@100": 0.37752742505056863,
1977
+ "NanoNFCorpus_maxsim_accuracy@1": 0.32,
1978
+ "NanoNFCorpus_maxsim_accuracy@3": 0.46,
1979
+ "NanoNFCorpus_maxsim_accuracy@5": 0.5,
1980
+ "NanoNFCorpus_maxsim_accuracy@10": 0.62,
1981
+ "NanoNFCorpus_maxsim_precision@1": 0.32,
1982
+ "NanoNFCorpus_maxsim_precision@3": 0.29333333333333333,
1983
+ "NanoNFCorpus_maxsim_precision@5": 0.26,
1984
+ "NanoNFCorpus_maxsim_precision@10": 0.214,
1985
+ "NanoNFCorpus_maxsim_recall@1": 0.019117208735943185,
1986
+ "NanoNFCorpus_maxsim_recall@3": 0.04835066438112243,
1987
+ "NanoNFCorpus_maxsim_recall@5": 0.0791661183332248,
1988
+ "NanoNFCorpus_maxsim_recall@10": 0.10063747732755461,
1989
+ "NanoNFCorpus_maxsim_ndcg@10": 0.25687747773042474,
1990
+ "NanoNFCorpus_maxsim_mrr@10": 0.40573809523809523,
1991
+ "NanoNFCorpus_maxsim_map@100": 0.0960046491785074,
1992
+ "NanoNQ_maxsim_accuracy@1": 0.16,
1993
+ "NanoNQ_maxsim_accuracy@3": 0.3,
1994
+ "NanoNQ_maxsim_accuracy@5": 0.36,
1995
+ "NanoNQ_maxsim_accuracy@10": 0.52,
1996
+ "NanoNQ_maxsim_precision@1": 0.16,
1997
+ "NanoNQ_maxsim_precision@3": 0.1,
1998
+ "NanoNQ_maxsim_precision@5": 0.076,
1999
+ "NanoNQ_maxsim_precision@10": 0.054000000000000006,
2000
+ "NanoNQ_maxsim_recall@1": 0.15,
2001
+ "NanoNQ_maxsim_recall@3": 0.28,
2002
+ "NanoNQ_maxsim_recall@5": 0.35,
2003
+ "NanoNQ_maxsim_recall@10": 0.49,
2004
+ "NanoNQ_maxsim_ndcg@10": 0.29921814444748196,
2005
+ "NanoNQ_maxsim_mrr@10": 0.2469365079365079,
2006
+ "NanoNQ_maxsim_map@100": 0.24908347359322217,
2007
+ "NanoQuoraRetrieval_maxsim_accuracy@1": 0.72,
2008
+ "NanoQuoraRetrieval_maxsim_accuracy@3": 0.86,
2009
+ "NanoQuoraRetrieval_maxsim_accuracy@5": 0.9,
2010
+ "NanoQuoraRetrieval_maxsim_accuracy@10": 0.9,
2011
+ "NanoQuoraRetrieval_maxsim_precision@1": 0.72,
2012
+ "NanoQuoraRetrieval_maxsim_precision@3": 0.32666666666666666,
2013
+ "NanoQuoraRetrieval_maxsim_precision@5": 0.21199999999999997,
2014
+ "NanoQuoraRetrieval_maxsim_precision@10": 0.11199999999999997,
2015
+ "NanoQuoraRetrieval_maxsim_recall@1": 0.634,
2016
+ "NanoQuoraRetrieval_maxsim_recall@3": 0.8106666666666666,
2017
+ "NanoQuoraRetrieval_maxsim_recall@5": 0.8640000000000001,
2018
+ "NanoQuoraRetrieval_maxsim_recall@10": 0.882,
2019
+ "NanoQuoraRetrieval_maxsim_ndcg@10": 0.7976010412361851,
2020
+ "NanoQuoraRetrieval_maxsim_mrr@10": 0.79,
2021
+ "NanoQuoraRetrieval_maxsim_map@100": 0.767328447138698,
2022
+ "NanoSCIDOCS_maxsim_accuracy@1": 0.32,
2023
+ "NanoSCIDOCS_maxsim_accuracy@3": 0.44,
2024
+ "NanoSCIDOCS_maxsim_accuracy@5": 0.48,
2025
+ "NanoSCIDOCS_maxsim_accuracy@10": 0.52,
2026
+ "NanoSCIDOCS_maxsim_precision@1": 0.32,
2027
+ "NanoSCIDOCS_maxsim_precision@3": 0.1733333333333333,
2028
+ "NanoSCIDOCS_maxsim_precision@5": 0.14400000000000002,
2029
+ "NanoSCIDOCS_maxsim_precision@10": 0.094,
2030
+ "NanoSCIDOCS_maxsim_recall@1": 0.065,
2031
+ "NanoSCIDOCS_maxsim_recall@3": 0.10600000000000001,
2032
+ "NanoSCIDOCS_maxsim_recall@5": 0.146,
2033
+ "NanoSCIDOCS_maxsim_recall@10": 0.192,
2034
+ "NanoSCIDOCS_maxsim_ndcg@10": 0.20285224967508114,
2035
+ "NanoSCIDOCS_maxsim_mrr@10": 0.3920238095238095,
2036
+ "NanoSCIDOCS_maxsim_map@100": 0.14807508167223443,
2037
+ "NanoArguAna_maxsim_accuracy@1": 0.16,
2038
+ "NanoArguAna_maxsim_accuracy@3": 0.3,
2039
+ "NanoArguAna_maxsim_accuracy@5": 0.34,
2040
+ "NanoArguAna_maxsim_accuracy@10": 0.44,
2041
+ "NanoArguAna_maxsim_precision@1": 0.16,
2042
+ "NanoArguAna_maxsim_precision@3": 0.09999999999999998,
2043
+ "NanoArguAna_maxsim_precision@5": 0.068,
2044
+ "NanoArguAna_maxsim_precision@10": 0.044000000000000004,
2045
+ "NanoArguAna_maxsim_recall@1": 0.16,
2046
+ "NanoArguAna_maxsim_recall@3": 0.3,
2047
+ "NanoArguAna_maxsim_recall@5": 0.34,
2048
+ "NanoArguAna_maxsim_recall@10": 0.44,
2049
+ "NanoArguAna_maxsim_ndcg@10": 0.2863387851680152,
2050
+ "NanoArguAna_maxsim_mrr@10": 0.23894444444444446,
2051
+ "NanoArguAna_maxsim_map@100": 0.2544415223253022,
2052
+ "NanoSciFact_maxsim_accuracy@1": 0.38,
2053
+ "NanoSciFact_maxsim_accuracy@3": 0.58,
2054
+ "NanoSciFact_maxsim_accuracy@5": 0.6,
2055
+ "NanoSciFact_maxsim_accuracy@10": 0.64,
2056
+ "NanoSciFact_maxsim_precision@1": 0.38,
2057
+ "NanoSciFact_maxsim_precision@3": 0.20666666666666664,
2058
+ "NanoSciFact_maxsim_precision@5": 0.13999999999999999,
2059
+ "NanoSciFact_maxsim_precision@10": 0.07400000000000001,
2060
+ "NanoSciFact_maxsim_recall@1": 0.345,
2061
+ "NanoSciFact_maxsim_recall@3": 0.56,
2062
+ "NanoSciFact_maxsim_recall@5": 0.6,
2063
+ "NanoSciFact_maxsim_recall@10": 0.64,
2064
+ "NanoSciFact_maxsim_ndcg@10": 0.5059540169647135,
2065
+ "NanoSciFact_maxsim_mrr@10": 0.469079365079365,
2066
+ "NanoSciFact_maxsim_map@100": 0.46646371070156983,
2067
+ "NanoTouche2020_maxsim_accuracy@1": 0.46938775510204084,
2068
+ "NanoTouche2020_maxsim_accuracy@3": 0.8367346938775511,
2069
+ "NanoTouche2020_maxsim_accuracy@5": 0.9387755102040817,
2070
+ "NanoTouche2020_maxsim_accuracy@10": 0.9795918367346939,
2071
+ "NanoTouche2020_maxsim_precision@1": 0.46938775510204084,
2072
+ "NanoTouche2020_maxsim_precision@3": 0.4693877551020409,
2073
+ "NanoTouche2020_maxsim_precision@5": 0.42040816326530617,
2074
+ "NanoTouche2020_maxsim_precision@10": 0.3734693877551021,
2075
+ "NanoTouche2020_maxsim_recall@1": 0.03474212930386132,
2076
+ "NanoTouche2020_maxsim_recall@3": 0.09924425724264538,
2077
+ "NanoTouche2020_maxsim_recall@5": 0.14517246914627877,
2078
+ "NanoTouche2020_maxsim_recall@10": 0.2381279528342168,
2079
+ "NanoTouche2020_maxsim_ndcg@10": 0.40956626471932184,
2080
+ "NanoTouche2020_maxsim_mrr@10": 0.6619047619047617,
2081
+ "NanoTouche2020_maxsim_map@100": 0.27301109115158484,
2082
+ "NanoBEIR_mean_maxsim_accuracy@1": 0.3699529042386185,
2083
+ "NanoBEIR_mean_maxsim_accuracy@3": 0.5289795918367347,
2084
+ "NanoBEIR_mean_maxsim_accuracy@5": 0.5906750392464678,
2085
+ "NanoBEIR_mean_maxsim_accuracy@10": 0.6768916797488226,
2086
+ "NanoBEIR_mean_maxsim_precision@1": 0.3699529042386185,
2087
+ "NanoBEIR_mean_maxsim_precision@3": 0.22995290423861856,
2088
+ "NanoBEIR_mean_maxsim_precision@5": 0.18003139717425432,
2089
+ "NanoBEIR_mean_maxsim_precision@10": 0.13042072213500785,
2090
+ "NanoBEIR_mean_maxsim_recall@1": 0.20377928654876706,
2091
+ "NanoBEIR_mean_maxsim_recall@3": 0.31510214730499875,
2092
+ "NanoBEIR_mean_maxsim_recall@5": 0.3705622070059087,
2093
+ "NanoBEIR_mean_maxsim_recall@10": 0.44579881858841686,
2094
+ "NanoBEIR_mean_maxsim_ndcg@10": 0.3999256389001595,
2095
+ "NanoBEIR_mean_maxsim_mrr@10": 0.4636410256410256,
2096
+ "NanoBEIR_mean_maxsim_map@100": 0.3293098753717478
2097
+ },
2098
+ "full_final": {
2099
+ "NanoClimateFEVER_maxsim_accuracy@1": 0.18,
2100
+ "NanoClimateFEVER_maxsim_accuracy@3": 0.3,
2101
+ "NanoClimateFEVER_maxsim_accuracy@5": 0.34,
2102
+ "NanoClimateFEVER_maxsim_accuracy@10": 0.5,
2103
+ "NanoClimateFEVER_maxsim_precision@1": 0.18,
2104
+ "NanoClimateFEVER_maxsim_precision@3": 0.1,
2105
+ "NanoClimateFEVER_maxsim_precision@5": 0.07600000000000001,
2106
+ "NanoClimateFEVER_maxsim_precision@10": 0.05800000000000001,
2107
+ "NanoClimateFEVER_maxsim_recall@1": 0.09166666666666667,
2108
+ "NanoClimateFEVER_maxsim_recall@3": 0.12999999999999998,
2109
+ "NanoClimateFEVER_maxsim_recall@5": 0.16,
2110
+ "NanoClimateFEVER_maxsim_recall@10": 0.2383333333333333,
2111
+ "NanoClimateFEVER_maxsim_ndcg@10": 0.19191034232336726,
2112
+ "NanoClimateFEVER_maxsim_mrr@10": 0.2662698412698412,
2113
+ "NanoClimateFEVER_maxsim_map@100": 0.14946591363353165,
2114
+ "NanoDBPedia_maxsim_accuracy@1": 0.62,
2115
+ "NanoDBPedia_maxsim_accuracy@3": 0.78,
2116
+ "NanoDBPedia_maxsim_accuracy@5": 0.84,
2117
+ "NanoDBPedia_maxsim_accuracy@10": 0.94,
2118
+ "NanoDBPedia_maxsim_precision@1": 0.62,
2119
+ "NanoDBPedia_maxsim_precision@3": 0.4733333333333333,
2120
+ "NanoDBPedia_maxsim_precision@5": 0.44800000000000006,
2121
+ "NanoDBPedia_maxsim_precision@10": 0.39199999999999996,
2122
+ "NanoDBPedia_maxsim_recall@1": 0.052136771709552124,
2123
+ "NanoDBPedia_maxsim_recall@3": 0.11703776658357738,
2124
+ "NanoDBPedia_maxsim_recall@5": 0.16684861747261473,
2125
+ "NanoDBPedia_maxsim_recall@10": 0.2876252581653851,
2126
+ "NanoDBPedia_maxsim_ndcg@10": 0.4820307033364284,
2127
+ "NanoDBPedia_maxsim_mrr@10": 0.7217380952380953,
2128
+ "NanoDBPedia_maxsim_map@100": 0.3543191098253603,
2129
+ "NanoFEVER_maxsim_accuracy@1": 0.56,
2130
+ "NanoFEVER_maxsim_accuracy@3": 0.72,
2131
+ "NanoFEVER_maxsim_accuracy@5": 0.82,
2132
+ "NanoFEVER_maxsim_accuracy@10": 0.86,
2133
+ "NanoFEVER_maxsim_precision@1": 0.56,
2134
+ "NanoFEVER_maxsim_precision@3": 0.24666666666666665,
2135
+ "NanoFEVER_maxsim_precision@5": 0.172,
2136
+ "NanoFEVER_maxsim_precision@10": 0.092,
2137
+ "NanoFEVER_maxsim_recall@1": 0.5266666666666667,
2138
+ "NanoFEVER_maxsim_recall@3": 0.6766666666666667,
2139
+ "NanoFEVER_maxsim_recall@5": 0.7833333333333333,
2140
+ "NanoFEVER_maxsim_recall@10": 0.8233333333333333,
2141
+ "NanoFEVER_maxsim_ndcg@10": 0.6792624024637341,
2142
+ "NanoFEVER_maxsim_mrr@10": 0.6485555555555556,
2143
+ "NanoFEVER_maxsim_map@100": 0.6348848591340011,
2144
+ "NanoFiQA2018_maxsim_accuracy@1": 0.26,
2145
+ "NanoFiQA2018_maxsim_accuracy@3": 0.4,
2146
+ "NanoFiQA2018_maxsim_accuracy@5": 0.48,
2147
+ "NanoFiQA2018_maxsim_accuracy@10": 0.58,
2148
+ "NanoFiQA2018_maxsim_precision@1": 0.26,
2149
+ "NanoFiQA2018_maxsim_precision@3": 0.18,
2150
+ "NanoFiQA2018_maxsim_precision@5": 0.132,
2151
+ "NanoFiQA2018_maxsim_precision@10": 0.08,
2152
+ "NanoFiQA2018_maxsim_recall@1": 0.12285714285714285,
2153
+ "NanoFiQA2018_maxsim_recall@3": 0.251047619047619,
2154
+ "NanoFiQA2018_maxsim_recall@5": 0.3117142857142857,
2155
+ "NanoFiQA2018_maxsim_recall@10": 0.38704761904761903,
2156
+ "NanoFiQA2018_maxsim_ndcg@10": 0.3027040168258662,
2157
+ "NanoFiQA2018_maxsim_mrr@10": 0.36041269841269835,
2158
+ "NanoFiQA2018_maxsim_map@100": 0.24465363135397997,
2159
+ "NanoHotpotQA_maxsim_accuracy@1": 0.72,
2160
+ "NanoHotpotQA_maxsim_accuracy@3": 0.9,
2161
+ "NanoHotpotQA_maxsim_accuracy@5": 0.92,
2162
+ "NanoHotpotQA_maxsim_accuracy@10": 0.96,
2163
+ "NanoHotpotQA_maxsim_precision@1": 0.72,
2164
+ "NanoHotpotQA_maxsim_precision@3": 0.41333333333333333,
2165
+ "NanoHotpotQA_maxsim_precision@5": 0.268,
2166
+ "NanoHotpotQA_maxsim_precision@10": 0.148,
2167
+ "NanoHotpotQA_maxsim_recall@1": 0.36,
2168
+ "NanoHotpotQA_maxsim_recall@3": 0.62,
2169
+ "NanoHotpotQA_maxsim_recall@5": 0.67,
2170
+ "NanoHotpotQA_maxsim_recall@10": 0.74,
2171
+ "NanoHotpotQA_maxsim_ndcg@10": 0.6839586445702376,
2172
+ "NanoHotpotQA_maxsim_mrr@10": 0.8070238095238095,
2173
+ "NanoHotpotQA_maxsim_map@100": 0.6020088740548019,
2174
+ "NanoMSMARCO_maxsim_accuracy@1": 0.16,
2175
+ "NanoMSMARCO_maxsim_accuracy@3": 0.32,
2176
+ "NanoMSMARCO_maxsim_accuracy@5": 0.42,
2177
+ "NanoMSMARCO_maxsim_accuracy@10": 0.7,
2178
+ "NanoMSMARCO_maxsim_precision@1": 0.16,
2179
+ "NanoMSMARCO_maxsim_precision@3": 0.10666666666666666,
2180
+ "NanoMSMARCO_maxsim_precision@5": 0.084,
2181
+ "NanoMSMARCO_maxsim_precision@10": 0.07,
2182
+ "NanoMSMARCO_maxsim_recall@1": 0.16,
2183
+ "NanoMSMARCO_maxsim_recall@3": 0.32,
2184
+ "NanoMSMARCO_maxsim_recall@5": 0.42,
2185
+ "NanoMSMARCO_maxsim_recall@10": 0.7,
2186
+ "NanoMSMARCO_maxsim_ndcg@10": 0.3858968432351718,
2187
+ "NanoMSMARCO_maxsim_mrr@10": 0.292015873015873,
2188
+ "NanoMSMARCO_maxsim_map@100": 0.3048587220806822,
2189
+ "NanoNFCorpus_maxsim_accuracy@1": 0.42,
2190
+ "NanoNFCorpus_maxsim_accuracy@3": 0.52,
2191
+ "NanoNFCorpus_maxsim_accuracy@5": 0.52,
2192
+ "NanoNFCorpus_maxsim_accuracy@10": 0.6,
2193
+ "NanoNFCorpus_maxsim_precision@1": 0.42,
2194
+ "NanoNFCorpus_maxsim_precision@3": 0.32666666666666666,
2195
+ "NanoNFCorpus_maxsim_precision@5": 0.26799999999999996,
2196
+ "NanoNFCorpus_maxsim_precision@10": 0.22399999999999998,
2197
+ "NanoNFCorpus_maxsim_recall@1": 0.044696247294946014,
2198
+ "NanoNFCorpus_maxsim_recall@3": 0.07181046561572595,
2199
+ "NanoNFCorpus_maxsim_recall@5": 0.0834824676415163,
2200
+ "NanoNFCorpus_maxsim_recall@10": 0.10608315478868971,
2201
+ "NanoNFCorpus_maxsim_ndcg@10": 0.28518605272369923,
2202
+ "NanoNFCorpus_maxsim_mrr@10": 0.4765238095238095,
2203
+ "NanoNFCorpus_maxsim_map@100": 0.12589300813746968,
2204
+ "NanoNQ_maxsim_accuracy@1": 0.28,
2205
+ "NanoNQ_maxsim_accuracy@3": 0.4,
2206
+ "NanoNQ_maxsim_accuracy@5": 0.48,
2207
+ "NanoNQ_maxsim_accuracy@10": 0.66,
2208
+ "NanoNQ_maxsim_precision@1": 0.28,
2209
+ "NanoNQ_maxsim_precision@3": 0.13333333333333333,
2210
+ "NanoNQ_maxsim_precision@5": 0.09600000000000002,
2211
+ "NanoNQ_maxsim_precision@10": 0.066,
2212
+ "NanoNQ_maxsim_recall@1": 0.27,
2213
+ "NanoNQ_maxsim_recall@3": 0.39,
2214
+ "NanoNQ_maxsim_recall@5": 0.46,
2215
+ "NanoNQ_maxsim_recall@10": 0.61,
2216
+ "NanoNQ_maxsim_ndcg@10": 0.42995107279160477,
2217
+ "NanoNQ_maxsim_mrr@10": 0.3832698412698412,
2218
+ "NanoNQ_maxsim_map@100": 0.3841358170885305,
2219
+ "NanoQuoraRetrieval_maxsim_accuracy@1": 0.74,
2220
+ "NanoQuoraRetrieval_maxsim_accuracy@3": 0.88,
2221
+ "NanoQuoraRetrieval_maxsim_accuracy@5": 0.9,
2222
+ "NanoQuoraRetrieval_maxsim_accuracy@10": 0.92,
2223
+ "NanoQuoraRetrieval_maxsim_precision@1": 0.74,
2224
+ "NanoQuoraRetrieval_maxsim_precision@3": 0.35999999999999993,
2225
+ "NanoQuoraRetrieval_maxsim_precision@5": 0.22799999999999998,
2226
+ "NanoQuoraRetrieval_maxsim_precision@10": 0.12,
2227
+ "NanoQuoraRetrieval_maxsim_recall@1": 0.654,
2228
+ "NanoQuoraRetrieval_maxsim_recall@3": 0.8586666666666667,
2229
+ "NanoQuoraRetrieval_maxsim_recall@5": 0.8859999999999999,
2230
+ "NanoQuoraRetrieval_maxsim_recall@10": 0.9126666666666666,
2231
+ "NanoQuoraRetrieval_maxsim_ndcg@10": 0.8354929187617376,
2232
+ "NanoQuoraRetrieval_maxsim_mrr@10": 0.8162222222222222,
2233
+ "NanoQuoraRetrieval_maxsim_map@100": 0.8099312372179969,
2234
+ "NanoSCIDOCS_maxsim_accuracy@1": 0.26,
2235
+ "NanoSCIDOCS_maxsim_accuracy@3": 0.42,
2236
+ "NanoSCIDOCS_maxsim_accuracy@5": 0.52,
2237
+ "NanoSCIDOCS_maxsim_accuracy@10": 0.74,
2238
+ "NanoSCIDOCS_maxsim_precision@1": 0.26,
2239
+ "NanoSCIDOCS_maxsim_precision@3": 0.18,
2240
+ "NanoSCIDOCS_maxsim_precision@5": 0.15200000000000002,
2241
+ "NanoSCIDOCS_maxsim_precision@10": 0.11800000000000001,
2242
+ "NanoSCIDOCS_maxsim_recall@1": 0.054000000000000006,
2243
+ "NanoSCIDOCS_maxsim_recall@3": 0.11000000000000001,
2244
+ "NanoSCIDOCS_maxsim_recall@5": 0.15400000000000003,
2245
+ "NanoSCIDOCS_maxsim_recall@10": 0.23999999999999996,
2246
+ "NanoSCIDOCS_maxsim_ndcg@10": 0.22168572688545177,
2247
+ "NanoSCIDOCS_maxsim_mrr@10": 0.38710317460317456,
2248
+ "NanoSCIDOCS_maxsim_map@100": 0.15605007993528686,
2249
+ "NanoArguAna_maxsim_accuracy@1": 0.18,
2250
+ "NanoArguAna_maxsim_accuracy@3": 0.38,
2251
+ "NanoArguAna_maxsim_accuracy@5": 0.42,
2252
+ "NanoArguAna_maxsim_accuracy@10": 0.56,
2253
+ "NanoArguAna_maxsim_precision@1": 0.18,
2254
+ "NanoArguAna_maxsim_precision@3": 0.12666666666666665,
2255
+ "NanoArguAna_maxsim_precision@5": 0.084,
2256
+ "NanoArguAna_maxsim_precision@10": 0.05600000000000001,
2257
+ "NanoArguAna_maxsim_recall@1": 0.18,
2258
+ "NanoArguAna_maxsim_recall@3": 0.38,
2259
+ "NanoArguAna_maxsim_recall@5": 0.42,
2260
+ "NanoArguAna_maxsim_recall@10": 0.56,
2261
+ "NanoArguAna_maxsim_ndcg@10": 0.3539147678833996,
2262
+ "NanoArguAna_maxsim_mrr@10": 0.29019047619047617,
2263
+ "NanoArguAna_maxsim_map@100": 0.3044243083606632,
2264
+ "NanoSciFact_maxsim_accuracy@1": 0.48,
2265
+ "NanoSciFact_maxsim_accuracy@3": 0.58,
2266
+ "NanoSciFact_maxsim_accuracy@5": 0.6,
2267
+ "NanoSciFact_maxsim_accuracy@10": 0.68,
2268
+ "NanoSciFact_maxsim_precision@1": 0.48,
2269
+ "NanoSciFact_maxsim_precision@3": 0.21333333333333332,
2270
+ "NanoSciFact_maxsim_precision@5": 0.136,
2271
+ "NanoSciFact_maxsim_precision@10": 0.078,
2272
+ "NanoSciFact_maxsim_recall@1": 0.445,
2273
+ "NanoSciFact_maxsim_recall@3": 0.565,
2274
+ "NanoSciFact_maxsim_recall@5": 0.59,
2275
+ "NanoSciFact_maxsim_recall@10": 0.67,
2276
+ "NanoSciFact_maxsim_ndcg@10": 0.564998292690912,
2277
+ "NanoSciFact_maxsim_mrr@10": 0.5397142857142857,
2278
+ "NanoSciFact_maxsim_map@100": 0.5383929411495326,
2279
+ "NanoTouche2020_maxsim_accuracy@1": 0.3877551020408163,
2280
+ "NanoTouche2020_maxsim_accuracy@3": 0.7755102040816326,
2281
+ "NanoTouche2020_maxsim_accuracy@5": 0.8775510204081632,
2282
+ "NanoTouche2020_maxsim_accuracy@10": 0.9795918367346939,
2283
+ "NanoTouche2020_maxsim_precision@1": 0.3877551020408163,
2284
+ "NanoTouche2020_maxsim_precision@3": 0.44897959183673464,
2285
+ "NanoTouche2020_maxsim_precision@5": 0.4408163265306122,
2286
+ "NanoTouche2020_maxsim_precision@10": 0.35510204081632657,
2287
+ "NanoTouche2020_maxsim_recall@1": 0.024481017655879133,
2288
+ "NanoTouche2020_maxsim_recall@3": 0.09602376403984346,
2289
+ "NanoTouche2020_maxsim_recall@5": 0.15246910845015463,
2290
+ "NanoTouche2020_maxsim_recall@10": 0.24076032744792322,
2291
+ "NanoTouche2020_maxsim_ndcg@10": 0.39199232237420195,
2292
+ "NanoTouche2020_maxsim_mrr@10": 0.5977324263038547,
2293
+ "NanoTouche2020_maxsim_map@100": 0.2962394648624034,
2294
+ "NanoBEIR_mean_maxsim_accuracy@1": 0.40367346938775506,
2295
+ "NanoBEIR_mean_maxsim_accuracy@3": 0.5673469387755101,
2296
+ "NanoBEIR_mean_maxsim_accuracy@5": 0.6259654631083202,
2297
+ "NanoBEIR_mean_maxsim_accuracy@10": 0.7445839874411302,
2298
+ "NanoBEIR_mean_maxsim_precision@1": 0.40367346938775506,
2299
+ "NanoBEIR_mean_maxsim_precision@3": 0.2545368916797488,
2300
+ "NanoBEIR_mean_maxsim_precision@5": 0.1988320251177394,
2301
+ "NanoBEIR_mean_maxsim_precision@10": 0.14285400313971744,
2302
+ "NanoBEIR_mean_maxsim_recall@1": 0.2296541932962195,
2303
+ "NanoBEIR_mean_maxsim_recall@3": 0.3527886883553923,
2304
+ "NanoBEIR_mean_maxsim_recall@5": 0.4044498317393773,
2305
+ "NanoBEIR_mean_maxsim_recall@10": 0.5012192071371501,
2306
+ "NanoBEIR_mean_maxsim_ndcg@10": 0.44684493129737013,
2307
+ "NanoBEIR_mean_maxsim_mrr@10": 0.506674777603349,
2308
+ "NanoBEIR_mean_maxsim_map@100": 0.37732753591032614
2309
+ },
2310
+ "configuration": {
2311
+ "smoke_test": false,
2312
+ "push_to_hub": false,
2313
+ "long_run": true
2314
+ },
2315
+ "training_args": {
2316
+ "output_dir": "models\\bert-tiny-msmarco-long",
2317
+ "per_device_train_batch_size": 128,
2318
+ "num_train_epochs": 3.0,
2319
+ "max_steps": 10000,
2320
+ "learning_rate": 1e-05,
2321
+ "lr_scheduler_type": "linear",
2322
+ "lr_scheduler_kwargs": null,
2323
+ "warmup_steps": 0.05,
2324
+ "optim": "adamw_torch_fused",
2325
+ "optim_args": null,
2326
+ "weight_decay": 0.01,
2327
+ "adam_beta1": 0.9,
2328
+ "adam_beta2": 0.999,
2329
+ "adam_epsilon": 1e-08,
2330
+ "optim_target_modules": null,
2331
+ "gradient_accumulation_steps": 1,
2332
+ "average_tokens_across_devices": true,
2333
+ "max_grad_norm": 1.0,
2334
+ "label_smoothing_factor": 0.0,
2335
+ "bf16": true,
2336
+ "fp16": false,
2337
+ "bf16_full_eval": false,
2338
+ "fp16_full_eval": false,
2339
+ "tf32": null,
2340
+ "gradient_checkpointing": false,
2341
+ "gradient_checkpointing_kwargs": null,
2342
+ "torch_compile": false,
2343
+ "torch_compile_backend": null,
2344
+ "torch_compile_mode": null,
2345
+ "use_liger_kernel": false,
2346
+ "liger_kernel_config": null,
2347
+ "use_cache": false,
2348
+ "neftune_noise_alpha": null,
2349
+ "torch_empty_cache_steps": null,
2350
+ "auto_find_batch_size": false,
2351
+ "logging_strategy": "steps",
2352
+ "logging_steps": 0.005,
2353
+ "logging_first_step": true,
2354
+ "log_on_each_node": true,
2355
+ "logging_nan_inf_filter": true,
2356
+ "include_num_input_tokens_seen": "no",
2357
+ "log_level": "passive",
2358
+ "log_level_replica": "warning",
2359
+ "disable_tqdm": true,
2360
+ "report_to": [],
2361
+ "run_name": "bert-tiny-msmarco-long",
2362
+ "project": "huggingface",
2363
+ "trackio_space_id": null,
2364
+ "trackio_bucket_id": null,
2365
+ "trackio_static_space_id": null,
2366
+ "eval_strategy": "steps",
2367
+ "eval_steps": 0.2,
2368
+ "eval_delay": 0,
2369
+ "per_device_eval_batch_size": 32,
2370
+ "prediction_loss_only": true,
2371
+ "eval_on_start": false,
2372
+ "eval_do_concat_batches": true,
2373
+ "eval_use_gather_object": false,
2374
+ "eval_accumulation_steps": null,
2375
+ "include_for_metrics": [],
2376
+ "batch_eval_metrics": false,
2377
+ "save_only_model": false,
2378
+ "save_strategy": "steps",
2379
+ "save_steps": 0.2,
2380
+ "save_on_each_node": false,
2381
+ "save_total_limit": 2,
2382
+ "enable_jit_checkpoint": false,
2383
+ "push_to_hub": false,
2384
+ "hub_token": "<HUB_TOKEN>",
2385
+ "hub_private_repo": null,
2386
+ "hub_model_id": null,
2387
+ "hub_strategy": "every_save",
2388
+ "hub_always_push": false,
2389
+ "hub_revision": null,
2390
+ "load_best_model_at_end": true,
2391
+ "metric_for_best_model": "eval_NanoBEIR_mean_maxsim_ndcg@10",
2392
+ "greater_is_better": true,
2393
+ "ignore_data_skip": false,
2394
+ "restore_callback_states_from_checkpoint": false,
2395
+ "full_determinism": false,
2396
+ "seed": 12,
2397
+ "data_seed": null,
2398
+ "use_cpu": false,
2399
+ "accelerator_config": {
2400
+ "split_batches": false,
2401
+ "dispatch_batches": null,
2402
+ "even_batches": true,
2403
+ "use_seedable_sampler": true,
2404
+ "non_blocking": false,
2405
+ "gradient_accumulation_kwargs": null
2406
+ },
2407
+ "parallelism_config": null,
2408
+ "dataloader_drop_last": false,
2409
+ "dataloader_num_workers": 0,
2410
+ "dataloader_pin_memory": true,
2411
+ "dataloader_persistent_workers": false,
2412
+ "dataloader_prefetch_factor": null,
2413
+ "dataloader_multiprocessing_context": null,
2414
+ "dataloader_in_order": true,
2415
+ "remove_unused_columns": true,
2416
+ "label_names": null,
2417
+ "train_sampling_strategy": "random",
2418
+ "length_column_name": "length",
2419
+ "ddp_find_unused_parameters": null,
2420
+ "ddp_bucket_cap_mb": null,
2421
+ "ddp_broadcast_buffers": false,
2422
+ "ddp_static_graph": null,
2423
+ "ddp_backend": null,
2424
+ "ddp_timeout": 1800,
2425
+ "fsdp": null,
2426
+ "fsdp_config": null,
2427
+ "deepspeed": null,
2428
+ "debug": [],
2429
+ "skip_memory_metrics": true,
2430
+ "do_train": false,
2431
+ "do_eval": true,
2432
+ "do_predict": false,
2433
+ "resume_from_checkpoint": null,
2434
+ "local_rank": -1,
2435
+ "prompts": null,
2436
+ "batch_sampler": "no_duplicates",
2437
+ "multi_dataset_batch_sampler": "proportional",
2438
+ "router_mapping": {},
2439
+ "learning_rate_mapping": {},
2440
+ "warmup_ratio": null,
2441
+ "max_length": null
2442
+ }
2443
+ }
sentence_bert_config.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "transformer_task": "feature-extraction",
3
+ "modality_config": {
4
+ "text": {
5
+ "method": "forward",
6
+ "method_output_name": "last_hidden_state"
7
+ }
8
+ },
9
+ "module_output_name": "token_embeddings",
10
+ "query_length": 32,
11
+ "document_length": 256,
12
+ "query_expansion": {
13
+ "strategy": "min",
14
+ "attend": false,
15
+ "token": null,
16
+ "length": 32
17
+ }
18
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "cls_token": "[CLS]",
4
+ "do_lower_case": true,
5
+ "is_local": false,
6
+ "local_files_only": false,
7
+ "mask_token": "[MASK]",
8
+ "max_length": 256,
9
+ "model_max_length": 512,
10
+ "pad_to_multiple_of": null,
11
+ "pad_token": "[PAD]",
12
+ "pad_token_type_id": 0,
13
+ "padding_side": "right",
14
+ "sep_token": "[SEP]",
15
+ "stride": 0,
16
+ "strip_accents": null,
17
+ "tokenize_chinese_chars": true,
18
+ "tokenizer_class": "BertTokenizer",
19
+ "truncation_side": "right",
20
+ "truncation_strategy": "longest_first",
21
+ "unk_token": "[UNK]"
22
+ }
train.py ADDED
@@ -0,0 +1,215 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Train BERT tiny, evaluating three NanoBEIR datasets every 20%.
2
+
3
+ Adapted from the multi-vector training skill template and training_contrastive.py.
4
+ Run from the repository root with Python and the training dependencies installed.
5
+ Use --smoke-test for one step, or --push-to-hub to upload the best checkpoint.
6
+ Use --long-run to continue the initial model for 10,000 steps on the full dataset.
7
+ """
8
+
9
+ import argparse
10
+ import json
11
+ import logging
12
+ import shutil
13
+ from contextlib import nullcontext
14
+ from pathlib import Path
15
+
16
+ import torch
17
+ from datasets import load_dataset
18
+ from transformers import BertConfig, BertModel, BertTokenizer, TrainerCallback, set_seed
19
+
20
+ from sentence_transformers import (
21
+ MultiVectorEncoder,
22
+ MultiVectorEncoderModelCardData,
23
+ MultiVectorEncoderTrainer,
24
+ MultiVectorEncoderTrainingArguments,
25
+ )
26
+ from sentence_transformers.base.modules import Dense, Normalize, Transformer
27
+ from sentence_transformers.base.sampler import BatchSamplers
28
+ from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorNanoBEIREvaluator
29
+ from sentence_transformers.multi_vector_encoder.losses import MultiVectorMultipleNegativesRankingLoss
30
+ from sentence_transformers.multi_vector_encoder.modules import MultiVectorMask
31
+
32
+ RUN_NAME = "bert-tiny-msmarco"
33
+ REPO_ID = f"multi-vector-encoder-testing/{RUN_NAME}"
34
+ INITIAL_REVISION = "81c5b4e78ac3bdbb01606e60e82bc34d86ed897b"
35
+
36
+
37
+ class LogProgress(TrainerCallback):
38
+ def on_log(self, args, state, control, logs=None, **kwargs):
39
+ values = {
40
+ key: value
41
+ for key, value in (logs or {}).items()
42
+ if key in ("loss", "learning_rate", "eval_loss", "eval_NanoBEIR_mean_maxsim_ndcg@10")
43
+ }
44
+ if values:
45
+ logging.info("Step %s/%s: %s", state.global_step, state.max_steps, values)
46
+
47
+
48
+ def autocast_ctx():
49
+ if not torch.cuda.is_available():
50
+ return nullcontext()
51
+ dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
52
+ return torch.autocast("cuda", dtype=dtype)
53
+
54
+
55
+ def main():
56
+ parser = argparse.ArgumentParser(description=__doc__)
57
+ parser.add_argument("--smoke-test", action="store_true")
58
+ parser.add_argument("--push-to-hub", action="store_true")
59
+ parser.add_argument("--long-run", action="store_true")
60
+ cli = parser.parse_args()
61
+ repo_id = REPO_ID + ("-long" if cli.long_run else "")
62
+ run_name = repo_id.split("/")[-1] + ("-smoke" if cli.smoke_test else "")
63
+ output_dir = Path("models") / run_name
64
+ output_dir.mkdir(parents=True, exist_ok=True)
65
+ Path("logs").mkdir(exist_ok=True)
66
+ logging.basicConfig(
67
+ format="%(asctime)s - %(message)s",
68
+ level=logging.INFO,
69
+ handlers=[logging.StreamHandler(), logging.FileHandler(f"logs/{run_name}.log", mode="w")],
70
+ force=True,
71
+ )
72
+ for noisy in ("httpx", "httpcore", "huggingface_hub", "urllib3", "filelock", "fsspec"):
73
+ logging.getLogger(noisy).setLevel(logging.WARNING)
74
+ set_seed(12)
75
+ if torch.cuda.is_available():
76
+ torch.set_float32_matmul_precision("high")
77
+
78
+ card = MultiVectorEncoderModelCardData(
79
+ language="en",
80
+ license="mit",
81
+ model_name="BERT tiny multi-vector encoder trained on MS MARCO",
82
+ model_id=repo_id,
83
+ )
84
+ if cli.long_run:
85
+ model = MultiVectorEncoder(REPO_ID, revision=INITIAL_REVISION, model_card_data=card)
86
+ else:
87
+ # The original checkpoint lacks model_type, which recent AutoConfig versions require.
88
+ base_dir = output_dir / "base"
89
+ base_model = BertModel.from_pretrained(
90
+ "prajjwal1/bert-tiny", config=BertConfig.from_pretrained("prajjwal1/bert-tiny")
91
+ )
92
+ base_model.save_pretrained(base_dir)
93
+ BertTokenizer.from_pretrained("prajjwal1/bert-tiny").save_pretrained(base_dir)
94
+ del base_model
95
+ transformer = Transformer(
96
+ str(base_dir),
97
+ query_length=32,
98
+ document_length=256,
99
+ query_expansion={"strategy": "min", "length": 32},
100
+ )
101
+ model = MultiVectorEncoder(
102
+ modules=[
103
+ transformer,
104
+ Dense(128, 128, bias=False, activation_function=None, module_input_name="token_embeddings"),
105
+ MultiVectorMask(),
106
+ Normalize(module_input_name="token_embeddings"),
107
+ ],
108
+ model_card_data=card,
109
+ )
110
+ model.model_card_data.set_base_model("prajjwal1/bert-tiny")
111
+ batch_size = 128 if cli.long_run else 32
112
+ train_size, eval_size = (batch_size * 2, 32) if cli.smoke_test else (16_000, 128)
113
+ split = "train" if cli.long_run and not cli.smoke_test else f"train[:{train_size + eval_size}]"
114
+ dataset = load_dataset("sentence-transformers/msmarco-bm25", "triplet", split=split).select_columns(
115
+ ["query", "positive", "negative"]
116
+ )
117
+ if cli.long_run and not cli.smoke_test:
118
+ eval_size = 1024
119
+ dataset = dataset.train_test_split(test_size=eval_size, seed=12)
120
+ evaluator = MultiVectorNanoBEIREvaluator(dataset_names=["msmarco", "nq", "fiqa2018"], batch_size=64)
121
+ logging.info("Baseline evaluation on three NanoBEIR datasets")
122
+ with autocast_ctx():
123
+ baseline_metrics = evaluator(model, output_path=str(output_dir), steps=0)
124
+ baseline_eval = baseline_metrics[evaluator.primary_metric]
125
+ full_evaluator = None
126
+ full_baseline = None
127
+ if cli.long_run and not cli.smoke_test:
128
+ full_evaluator = MultiVectorNanoBEIREvaluator(batch_size=64)
129
+ full_output = output_dir / "full_eval"
130
+ full_output.mkdir(exist_ok=True)
131
+ logging.info("Baseline evaluation on all 13 NanoBEIR datasets")
132
+ with autocast_ctx():
133
+ full_baseline = full_evaluator(model, output_path=str(full_output), steps=0)
134
+ (output_dir / "baseline.json").write_text(
135
+ json.dumps({"selection": baseline_metrics, "full": full_baseline}, indent=2), encoding="utf-8"
136
+ )
137
+
138
+ args = MultiVectorEncoderTrainingArguments(
139
+ output_dir=str(output_dir),
140
+ max_steps=1 if cli.smoke_test else 10_000 if cli.long_run else 500,
141
+ per_device_train_batch_size=batch_size,
142
+ per_device_eval_batch_size=32,
143
+ learning_rate=1e-5 if cli.long_run else 3e-5,
144
+ weight_decay=0.01,
145
+ warmup_steps=0.05,
146
+ bf16=torch.cuda.is_available() and torch.cuda.is_bf16_supported(),
147
+ fp16=torch.cuda.is_available() and not torch.cuda.is_bf16_supported(),
148
+ batch_sampler=BatchSamplers.NO_DUPLICATES,
149
+ eval_strategy="steps",
150
+ eval_steps=0.2,
151
+ save_strategy="steps",
152
+ save_steps=0.2,
153
+ save_total_limit=2,
154
+ logging_steps=0.005 if cli.long_run else 0.02,
155
+ logging_first_step=True,
156
+ disable_tqdm=True,
157
+ load_best_model_at_end=True,
158
+ metric_for_best_model=f"eval_{evaluator.primary_metric}",
159
+ greater_is_better=True,
160
+ report_to="none",
161
+ run_name=run_name,
162
+ seed=12,
163
+ )
164
+ trainer = MultiVectorEncoderTrainer(
165
+ model=model,
166
+ args=args,
167
+ train_dataset=dataset["train"],
168
+ eval_dataset=dataset["test"],
169
+ loss=MultiVectorMultipleNegativesRankingLoss(model, scale=1.0),
170
+ evaluator=evaluator,
171
+ callbacks=[LogProgress()],
172
+ )
173
+ logging.info("Training configuration: %s", args.to_dict())
174
+ trainer.train()
175
+ logging.info("Evaluating the best checkpoint on the same three datasets")
176
+ with autocast_ctx():
177
+ final_metrics = evaluator(model, output_path=str(output_dir / "eval"))
178
+ score = final_metrics[evaluator.primary_metric]
179
+ full_final = None
180
+ if full_evaluator is not None:
181
+ logging.info("Evaluating the best checkpoint on all 13 NanoBEIR datasets")
182
+ with autocast_ctx():
183
+ full_final = full_evaluator(model, output_path=str(full_output))
184
+ baseline_eval = full_baseline[full_evaluator.primary_metric]
185
+ score = full_final[full_evaluator.primary_metric]
186
+ delta = score - baseline_eval
187
+ verdict = "WIN" if delta >= 0.005 else "MARGINAL" if delta >= 0 else "REGRESSION"
188
+ logging.info("VERDICT: %s | score=%.4f | baseline=%.4f | delta=%+.4f", verdict, score, baseline_eval, delta)
189
+
190
+ final_dir = output_dir / "final"
191
+ model.save_pretrained(str(final_dir))
192
+ shutil.copy2(__file__, final_dir / "train.py")
193
+ results = {
194
+ "baseline": baseline_metrics,
195
+ "final": final_metrics,
196
+ "best_checkpoint": trainer.state.best_model_checkpoint,
197
+ "history": trainer.state.log_history,
198
+ "verdict": verdict,
199
+ "full_baseline": full_baseline,
200
+ "full_final": full_final,
201
+ "configuration": vars(cli),
202
+ "training_args": args.to_dict(),
203
+ }
204
+ (final_dir / "results.json").write_text(json.dumps(results, indent=2), encoding="utf-8")
205
+ logging.info("Saved model, training script, and metrics to %s", final_dir)
206
+ if cli.push_to_hub and not cli.smoke_test:
207
+ try:
208
+ url = model.push_to_hub(repo_id, local_model_path=str(final_dir))
209
+ logging.info("Uploaded to %s", url)
210
+ except Exception:
211
+ logging.exception("Hub upload failed. The model is saved at %s", final_dir)
212
+
213
+
214
+ if __name__ == "__main__":
215
+ main()