Upload me5s_compressed_distilled (distilled) from mE5-small

README.md

@@ -24,12 +24,12 @@ Compact multilingual sentence encoder compressed from `intfloat/multilingual-e5-
 | Base model | `intfloat/multilingual-e5-small` |
 | Architecture | bert (encoder) |
 | Hidden dim | 384 (from 384) |
-| Layers | 5 (from 12) |
+| Layers | 4 (from 12) |
 | Intermediate | 1536 |
 | Attention heads | 12 |
-| Vocab size | 8,675 (from 250,037) |
-| Parameters | ~12.4M |
-| Model size (FP32) | 47.9MB |
+| Vocab size | 15,168 (from 250,037) |
+| Parameters | ~13.1M |
+| Model size (FP32) | 50.6MB |
 | Compression | 9x |
 | Distilled | Yes |
 
@@ -53,84 +53,84 @@ print(embeddings.shape)  # (4, 384)
 
 ## MTEB Evaluation Results
 
-**Overall Average: 46.91%**
+**Overall Average: 50.81%**
 
 | Task Group | Average |
 |---|---|
-| Classification | 51.82% |
-| Clustering | 28.72% |
-| STS | 58.72% |
+| Classification | 56.17% |
+| Clustering | 29.64% |
+| STS | 64.86% |
 
 ### Classification
 
 | Task | Average | Details |
 |---|---|---|
-| AmazonCounterfactualClassification | 62.93% | en: 69.51%, en-ext: 65.08%, ja: 59.78%, de: 57.34% |
-| Banking77Classification | 62.72% | default: 62.72% |
-| ImdbClassification | 50.74% | default: 50.74% |
-| MTOPDomainClassification | 68.85% | en: 79.79%, es: 71.9%, fr: 69.12%, th: 67.41%, hi: 64.79% |
-| MassiveIntentClassification | 26.86% | en: 57.38%, zh-CN: 48.33%, ja: 45.3%, de: 45.04%, fr: 44.92% |
-| MassiveScenarioClassification | 34.66% | en: 66.9%, de: 59.71%, zh-CN: 58.67%, nl: 56.51%, fr: 55.21% |
-| ToxicConversationsClassification | 54.97% | default: 54.97% |
-| TweetSentimentExtractionClassification | 52.8% | default: 52.8% |
+| AmazonCounterfactualClassification | 67.34% | de: 71.67%, en: 71.63%, en-ext: 69.81%, ja: 56.27% |
+| Banking77Classification | 67.81% | default: 67.81% |
+| ImdbClassification | 54.45% | default: 54.45% |
+| MTOPDomainClassification | 74.88% | en: 84.01%, es: 78.04%, fr: 75.53%, hi: 72.32%, th: 72.24% |
+| MassiveIntentClassification | 30.9% | en: 61.8%, ja: 56.38%, zh-CN: 56.37%, ko: 54.29%, es: 54.0% |
+| MassiveScenarioClassification | 38.29% | en: 68.19%, zh-CN: 67.13%, ja: 64.95%, de: 63.29%, ko: 63.23% |
+| ToxicConversationsClassification | 55.65% | default: 55.65% |
+| TweetSentimentExtractionClassification | 60.03% | default: 60.03% |
 
 ### Clustering
 
 | Task | Average | Details |
 |---|---|---|
-| ArXivHierarchicalClusteringP2P | 47.82% | default: 47.82% |
-| ArXivHierarchicalClusteringS2S | 43.73% | default: 43.73% |
-| BiorxivClusteringP2P.v2 | 13.34% | default: 13.34% |
-| MedrxivClusteringP2P.v2 | 20.89% | default: 20.89% |
-| MedrxivClusteringS2S.v2 | 20.47% | default: 20.47% |
-| StackExchangeClustering.v2 | 39.47% | default: 39.47% |
-| StackExchangeClusteringP2P.v2 | 31.1% | default: 31.1% |
-| TwentyNewsgroupsClustering.v2 | 12.96% | default: 12.96% |
+| ArXivHierarchicalClusteringP2P | 49.54% | default: 49.54% |
+| ArXivHierarchicalClusteringS2S | 46.49% | default: 46.49% |
+| BiorxivClusteringP2P.v2 | 13.53% | default: 13.53% |
+| MedrxivClusteringP2P.v2 | 22.28% | default: 22.28% |
+| MedrxivClusteringS2S.v2 | 21.37% | default: 21.37% |
+| StackExchangeClustering.v2 | 38.82% | default: 38.82% |
+| StackExchangeClusteringP2P.v2 | 30.07% | default: 30.07% |
+| TwentyNewsgroupsClustering.v2 | 15.01% | default: 15.01% |
 
 ### STS
 
 | Task | Average | Details |
 |---|---|---|
-| BIOSSES | 44.17% | default: 44.17% |
-| SICK-R | 69.93% | default: 69.93% |
-| STS12 | 71.78% | default: 71.78% |
-| STS13 | 61.49% | default: 61.49% |
-| STS14 | 65.8% | default: 65.8% |
-| STS15 | 78.31% | default: 78.31% |
-| STS17 | 34.39% | en-en: 78.79%, es-es: 71.36%, ar-ar: 55.54%, ko-ko: 50.35%, nl-en: 28.55% |
-| STS22.v2 | 32.35% | fr-pl: 61.98%, zh: 52.61%, es: 52.17%, it: 48.4%, fr: 45.5% |
-| STSBenchmark | 70.22% | default: 70.22% |
+| BIOSSES | 63.23% | default: 63.23% |
+| SICK-R | 74.84% | default: 74.84% |
+| STS12 | 74.24% | default: 74.24% |
+| STS13 | 69.75% | default: 69.75% |
+| STS14 | 70.86% | default: 70.86% |
+| STS15 | 82.14% | default: 82.14% |
+| STS17 | 40.06% | en-en: 82.62%, es-es: 76.34%, ar-ar: 60.47%, ko-ko: 60.46%, nl-en: 40.14% |
+| STS22.v2 | 31.5% | zh: 60.47%, es: 56.06%, fr: 54.8%, it: 40.81%, en: 40.52% |
+| STSBenchmark | 77.11% | default: 77.11% |
 
 
 ## Distillation Impact
 
 | Task | Before | After | Delta |
 |---|---|---|---|
-| AmazonCounterfactualClassification | 65.35% | 62.93% | -2.42%p |
-| ArXivHierarchicalClusteringP2P | 47.9% | 47.82% | -0.08%p |
-| ArXivHierarchicalClusteringS2S | 47.79% | 43.73% | -4.06%p |
-| BIOSSES | 56.57% | 44.17% | -12.4%p |
-| Banking77Classification | 59.84% | 62.72% | +2.88%p |
-| BiorxivClusteringP2P.v2 | 16.17% | 13.34% | -2.83%p |
-| ImdbClassification | 54.77% | 50.74% | -4.03%p |
-| MTOPDomainClassification | 63.66% | 68.85% | +5.19%p |
-| MassiveIntentClassification | 29.77% | 26.86% | -2.91%p |
-| MassiveScenarioClassification | 33.24% | 34.66% | +1.42%p |
-| MedrxivClusteringP2P.v2 | 22.33% | 20.89% | -1.44%p |
-| MedrxivClusteringS2S.v2 | 20.56% | 20.47% | -0.09%p |
-| SICK-R | 58.16% | 69.93% | +11.77%p |
-| STS12 | 53.44% | 71.78% | +18.34%p |
-| STS13 | 61.88% | 61.49% | -0.39%p |
-| STS14 | 58.82% | 65.8% | +6.98%p |
-| STS15 | 73.5% | 78.31% | +4.81%p |
-| STS17 | 36.19% | 34.39% | -1.8%p |
-| STS22.v2 | 29.53% | 32.35% | +2.82%p |
-| STSBenchmark | 59.49% | 70.22% | +10.73%p |
-| StackExchangeClustering.v2 | 39.48% | 39.47% | -0.01%p |
-| StackExchangeClusteringP2P.v2 | 30.91% | 31.1% | +0.19%p |
-| ToxicConversationsClassification | 55.55% | 54.97% | -0.58%p |
-| TweetSentimentExtractionClassification | 42.38% | 52.8% | +10.42%p |
-| TwentyNewsgroupsClustering.v2 | 12.68% | 12.96% | +0.28%p |
+| AmazonCounterfactualClassification | 67.37% | 67.34% | -0.03%p |
+| ArXivHierarchicalClusteringP2P | 47.08% | 49.54% | +2.46%p |
+| ArXivHierarchicalClusteringS2S | 48.29% | 46.49% | -1.8%p |
+| BIOSSES | 56.68% | 63.23% | +6.55%p |
+| Banking77Classification | 58.7% | 67.81% | +9.11%p |
+| BiorxivClusteringP2P.v2 | 17.24% | 13.53% | -3.71%p |
+| ImdbClassification | 57.14% | 54.45% | -2.69%p |
+| MTOPDomainClassification | 66.84% | 74.88% | +8.04%p |
+| MassiveIntentClassification | 31.12% | 30.9% | -0.22%p |
+| MassiveScenarioClassification | 34.85% | 38.29% | +3.44%p |
+| MedrxivClusteringP2P.v2 | 24.42% | 22.28% | -2.14%p |
+| MedrxivClusteringS2S.v2 | 21.55% | 21.37% | -0.18%p |
+| SICK-R | 59.22% | 74.84% | +15.62%p |
+| STS12 | 52.11% | 74.24% | +22.13%p |
+| STS13 | 64.25% | 69.75% | +5.5%p |
+| STS14 | 60.12% | 70.86% | +10.74%p |
+| STS15 | 74.19% | 82.14% | +7.95%p |
+| STS17 | 38.71% | 40.06% | +1.35%p |
+| STS22.v2 | 27.7% | 31.5% | +3.8%p |
+| STSBenchmark | 60.91% | 77.11% | +16.2%p |
+| StackExchangeClustering.v2 | 39.42% | 38.82% | -0.6%p |
+| StackExchangeClusteringP2P.v2 | 31.85% | 30.07% | -1.78%p |
+| ToxicConversationsClassification | 55.82% | 55.65% | -0.17%p |
+| TweetSentimentExtractionClassification | 45.74% | 60.03% | +14.29%p |
+| TwentyNewsgroupsClustering.v2 | 13.35% | 15.01% | +1.66%p |
 
 
 ## Training
@@ -138,7 +138,7 @@ print(embeddings.shape)  # (4, 384)
 ### Stage 1: Model Compression
 - **Teacher**: `intfloat/multilingual-e5-small` (12L, 384d)
 - **Compression**: Layer pruning + Vocab pruning
-- **Result**: 5L / 384d / 8,675 vocab
+- **Result**: 4L / 384d / 15,168 vocab
 
 ### Stage 2: Knowledge Distillation
 - **Method**: MSE + Cosine Similarity loss

config.json

@@ -14,12 +14,12 @@
   "max_position_embeddings": 512,
   "model_type": "bert",
   "num_attention_heads": 12,
-  "num_hidden_layers": 5,
+  "num_hidden_layers": 4,
   "pad_token_id": 0,
   "position_embedding_type": "absolute",
   "tokenizer_class": "XLMRobertaTokenizer",
   "transformers_version": "4.56.2",
   "type_vocab_size": 2,
   "use_cache": true,
-  "vocab_size": 8675
+  "vocab_size": 15168
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:504814e0b78cfde32f085153eda926e8e0b395756de301be97c064b26bf0c74e
-size 50207568
+oid sha256:248b7dfb9bb4c1e024b364077af53bc31b35b853f6050510d4a89dd74d0803de
+size 53081192

tokenizer_config.json

@@ -32,7 +32,7 @@
       "single_word": false,
       "special": true
     },
-    "8674": {
+    "15167": {
       "content": "<mask>",
       "lstrip": false,
       "normalized": false,