Upload L4_uniform_distilled (distilled) from MiniLM-L12
Browse files
README.md
CHANGED
|
@@ -104,6 +104,86 @@ embeddings = model.encode(sentences)
|
|
| 104 |
print(embeddings.shape) # (3, 384)
|
| 105 |
```
|
| 106 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 107 |
|
| 108 |
|
| 109 |
## Training
|
|
|
|
| 104 |
print(embeddings.shape) # (3, 384)
|
| 105 |
```
|
| 106 |
|
| 107 |
+
## MTEB Evaluation Results
|
| 108 |
+
|
| 109 |
+
**Overall Average: 54.62%**
|
| 110 |
+
|
| 111 |
+
| Task Group | Average |
|
| 112 |
+
|---|---|
|
| 113 |
+
| Classification | 58.95% |
|
| 114 |
+
| Clustering | 36.16% |
|
| 115 |
+
| STS | 67.19% |
|
| 116 |
+
|
| 117 |
+
### Classification
|
| 118 |
+
|
| 119 |
+
| Task | Average | Details |
|
| 120 |
+
|---|---|---|
|
| 121 |
+
| AmazonCounterfactualClassification | 65.7% | en: 68.34%, de: 67.09%, en-ext: 66.57% |
|
| 122 |
+
| Banking77Classification | 78.43% | default: 78.43% |
|
| 123 |
+
| ImdbClassification | 60.43% | default: 60.43% |
|
| 124 |
+
| MTOPDomainClassification | 74.26% | en: 83.96%, es: 78.09%, fr: 76.42% |
|
| 125 |
+
| MassiveIntentClassification | 34.72% | en: 65.57%, zh-CN: 62.56%, ja: 62.19% |
|
| 126 |
+
| MassiveScenarioClassification | 40.37% | en: 71.61%, zh-CN: 69.22%, ja: 67.48% |
|
| 127 |
+
| ToxicConversationsClassification | 61.37% | default: 61.37% |
|
| 128 |
+
| TweetSentimentExtractionClassification | 56.29% | default: 56.29% |
|
| 129 |
+
|
| 130 |
+
### Clustering
|
| 131 |
+
|
| 132 |
+
| Task | Average | Details |
|
| 133 |
+
|---|---|---|
|
| 134 |
+
| ArXivHierarchicalClusteringP2P | 51.36% | default: 51.36% |
|
| 135 |
+
| ArXivHierarchicalClusteringS2S | 46.3% | default: 46.3% |
|
| 136 |
+
| BiorxivClusteringP2P.v2 | 26.63% | default: 26.63% |
|
| 137 |
+
| MedrxivClusteringP2P.v2 | 30.15% | default: 30.15% |
|
| 138 |
+
| MedrxivClusteringS2S.v2 | 25.7% | default: 25.7% |
|
| 139 |
+
| StackExchangeClustering.v2 | 46.56% | default: 46.56% |
|
| 140 |
+
| StackExchangeClusteringP2P.v2 | 35.85% | default: 35.85% |
|
| 141 |
+
| TwentyNewsgroupsClustering.v2 | 26.74% | default: 26.74% |
|
| 142 |
+
|
| 143 |
+
### STS
|
| 144 |
+
|
| 145 |
+
| Task | Average | Details |
|
| 146 |
+
|---|---|---|
|
| 147 |
+
| BIOSSES | 53.2% | default: 53.2% |
|
| 148 |
+
| SICK-R | 73.61% | default: 73.61% |
|
| 149 |
+
| STS12 | 72.11% | default: 72.11% |
|
| 150 |
+
| STS13 | 75.62% | default: 75.62% |
|
| 151 |
+
| STS14 | 72.2% | default: 72.2% |
|
| 152 |
+
| STS15 | 80.86% | default: 80.86% |
|
| 153 |
+
| STS17 | 48.49% | en-en: 81.73%, es-es: 75.65%, ar-ar: 57.76% |
|
| 154 |
+
| STS22.v2 | 49.17% | fr: 69.99%, es-en: 65.46%, zh: 63.87% |
|
| 155 |
+
| STSBenchmark | 79.48% | default: 79.48% |
|
| 156 |
+
|
| 157 |
+
|
| 158 |
+
## Distillation Impact
|
| 159 |
+
|
| 160 |
+
| Task | Before | After | Delta |
|
| 161 |
+
|---|---|---|---|
|
| 162 |
+
| AmazonCounterfactualClassification | 67.02% | 65.7% | -1.32%p |
|
| 163 |
+
| ArXivHierarchicalClusteringP2P | 49.93% | 51.36% | +1.43%p |
|
| 164 |
+
| ArXivHierarchicalClusteringS2S | 46.08% | 46.3% | +0.22%p |
|
| 165 |
+
| Banking77Classification | 69.18% | 78.43% | +9.25%p |
|
| 166 |
+
| BiorxivClusteringP2P.v2 | 21.47% | 26.63% | +5.16%p |
|
| 167 |
+
| BIOSSES | 45.64% | 53.2% | +7.56%p |
|
| 168 |
+
| ImdbClassification | 59.38% | 60.43% | +1.05%p |
|
| 169 |
+
| MassiveIntentClassification | 36.9% | 34.72% | -2.18%p |
|
| 170 |
+
| MassiveScenarioClassification | 39.51% | 40.37% | +0.86%p |
|
| 171 |
+
| MedrxivClusteringP2P.v2 | 26.05% | 30.15% | +4.1%p |
|
| 172 |
+
| MedrxivClusteringS2S.v2 | 22.94% | 25.7% | +2.76%p |
|
| 173 |
+
| MTOPDomainClassification | 71.48% | 74.26% | +2.78%p |
|
| 174 |
+
| SICK-R | 62.01% | 73.61% | +11.6%p |
|
| 175 |
+
| StackExchangeClustering.v2 | 41.23% | 46.56% | +5.33%p |
|
| 176 |
+
| StackExchangeClusteringP2P.v2 | 32.19% | 35.85% | +3.66%p |
|
| 177 |
+
| STS12 | 57.85% | 72.11% | +14.26%p |
|
| 178 |
+
| STS13 | 65.48% | 75.62% | +10.14%p |
|
| 179 |
+
| STS14 | 60.39% | 72.2% | +11.81%p |
|
| 180 |
+
| STS15 | 73.93% | 80.86% | +6.93%p |
|
| 181 |
+
| STS17 | 46.29% | 48.49% | +2.2%p |
|
| 182 |
+
| STS22.v2 | 37.34% | 49.17% | +11.83%p |
|
| 183 |
+
| STSBenchmark | 65.38% | 79.48% | +14.1%p |
|
| 184 |
+
| ToxicConversationsClassification | 62.02% | 61.37% | -0.65%p |
|
| 185 |
+
| TweetSentimentExtractionClassification | 49.43% | 56.29% | +6.86%p |
|
| 186 |
+
| TwentyNewsgroupsClustering.v2 | 16.43% | 26.74% | +10.31%p |
|
| 187 |
|
| 188 |
|
| 189 |
## Training
|