Upload me5s_compressed_distilled (distilled) from mE5-small

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+    "word_embedding_dimension": 384,
+    "pooling_mode_cls_token": false,
+    "pooling_mode_mean_tokens": true,
+    "pooling_mode_max_tokens": false,
+    "pooling_mode_mean_sqrt_len_tokens": false,
+    "pooling_mode_weightedmean_tokens": false,
+    "pooling_mode_lasttoken": false,
+    "include_prompt": true
+}

README.md

@@ -0,0 +1,152 @@
+---
+language: ["ko", "en", "ja", "zh", "es", "fr", "de", "pt", "it", "ru", "ar", "hi", "th", "vi", "id", "tr", "nl", "pl"]
+tags:
+- sentence-transformers
+- multilingual
+- model-compression
+- layer-pruning
+- vocab-pruning
+- knowledge-distillation
+- me5-small
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+license: apache-2.0
+---
+
+# me5s_compressed_distilled (Distilled)
+
+Compact multilingual sentence encoder compressed from `intfloat/multilingual-e5-small` (9x compression).
+
+## Model Details
+
+| Property | Value |
+|---|---|
+| Base model | `intfloat/multilingual-e5-small` |
+| Architecture | bert (encoder) |
+| Hidden dim | 384 (from 384) |
+| Layers | 5 (from 12) |
+| Intermediate | 1536 |
+| Attention heads | 12 |
+| Vocab size | 8,675 (from 250,037) |
+| Parameters | ~12.4M |
+| Model size (FP32) | 47.9MB |
+| Compression | 9x |
+| Distilled | Yes |
+
+## Quick Start
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("me5s_compressed_distilled", trust_remote_code=True)
+
+sentences = [
+    "Hello, how are you?",
+    "안녕하세요, 잘 지내세요?",
+    "こんにちは、元気ですか？",
+    "你好，你好吗？",
+]
+
+embeddings = model.encode(sentences)
+print(embeddings.shape)  # (4, 384)
+```
+
+## MTEB Evaluation Results
+
+**Overall Average: 46.91%**
+
+| Task Group | Average |
+|---|---|
+| Classification | 51.82% |
+| Clustering | 28.72% |
+| STS | 58.72% |
+
+### Classification
+
+| Task | Average | Details |
+|---|---|---|
+| AmazonCounterfactualClassification | 62.93% | en: 69.51%, en-ext: 65.08%, ja: 59.78%, de: 57.34% |
+| Banking77Classification | 62.72% | default: 62.72% |
+| ImdbClassification | 50.74% | default: 50.74% |
+| MTOPDomainClassification | 68.85% | en: 79.79%, es: 71.9%, fr: 69.12%, th: 67.41%, hi: 64.79% |
+| MassiveIntentClassification | 26.86% | en: 57.38%, zh-CN: 48.33%, ja: 45.3%, de: 45.04%, fr: 44.92% |
+| MassiveScenarioClassification | 34.66% | en: 66.9%, de: 59.71%, zh-CN: 58.67%, nl: 56.51%, fr: 55.21% |
+| ToxicConversationsClassification | 54.97% | default: 54.97% |
+| TweetSentimentExtractionClassification | 52.8% | default: 52.8% |
+
+### Clustering
+
+| Task | Average | Details |
+|---|---|---|
+| ArXivHierarchicalClusteringP2P | 47.82% | default: 47.82% |
+| ArXivHierarchicalClusteringS2S | 43.73% | default: 43.73% |
+| BiorxivClusteringP2P.v2 | 13.34% | default: 13.34% |
+| MedrxivClusteringP2P.v2 | 20.89% | default: 20.89% |
+| MedrxivClusteringS2S.v2 | 20.47% | default: 20.47% |
+| StackExchangeClustering.v2 | 39.47% | default: 39.47% |
+| StackExchangeClusteringP2P.v2 | 31.1% | default: 31.1% |
+| TwentyNewsgroupsClustering.v2 | 12.96% | default: 12.96% |
+
+### STS
+
+| Task | Average | Details |
+|---|---|---|
+| BIOSSES | 44.17% | default: 44.17% |
+| SICK-R | 69.93% | default: 69.93% |
+| STS12 | 71.78% | default: 71.78% |
+| STS13 | 61.49% | default: 61.49% |
+| STS14 | 65.8% | default: 65.8% |
+| STS15 | 78.31% | default: 78.31% |
+| STS17 | 34.39% | en-en: 78.79%, es-es: 71.36%, ar-ar: 55.54%, ko-ko: 50.35%, nl-en: 28.55% |
+| STS22.v2 | 32.35% | fr-pl: 61.98%, zh: 52.61%, es: 52.17%, it: 48.4%, fr: 45.5% |
+| STSBenchmark | 70.22% | default: 70.22% |
+
+
+## Distillation Impact
+
+| Task | Before | After | Delta |
+|---|---|---|---|
+| AmazonCounterfactualClassification | 65.35% | 62.93% | -2.42%p |
+| ArXivHierarchicalClusteringP2P | 47.9% | 47.82% | -0.08%p |
+| ArXivHierarchicalClusteringS2S | 47.79% | 43.73% | -4.06%p |
+| BIOSSES | 56.57% | 44.17% | -12.4%p |
+| Banking77Classification | 59.84% | 62.72% | +2.88%p |
+| BiorxivClusteringP2P.v2 | 16.17% | 13.34% | -2.83%p |
+| ImdbClassification | 54.77% | 50.74% | -4.03%p |
+| MTOPDomainClassification | 63.66% | 68.85% | +5.19%p |
+| MassiveIntentClassification | 29.77% | 26.86% | -2.91%p |
+| MassiveScenarioClassification | 33.24% | 34.66% | +1.42%p |
+| MedrxivClusteringP2P.v2 | 22.33% | 20.89% | -1.44%p |
+| MedrxivClusteringS2S.v2 | 20.56% | 20.47% | -0.09%p |
+| SICK-R | 58.16% | 69.93% | +11.77%p |
+| STS12 | 53.44% | 71.78% | +18.34%p |
+| STS13 | 61.88% | 61.49% | -0.39%p |
+| STS14 | 58.82% | 65.8% | +6.98%p |
+| STS15 | 73.5% | 78.31% | +4.81%p |
+| STS17 | 36.19% | 34.39% | -1.8%p |
+| STS22.v2 | 29.53% | 32.35% | +2.82%p |
+| STSBenchmark | 59.49% | 70.22% | +10.73%p |
+| StackExchangeClustering.v2 | 39.48% | 39.47% | -0.01%p |
+| StackExchangeClusteringP2P.v2 | 30.91% | 31.1% | +0.19%p |
+| ToxicConversationsClassification | 55.55% | 54.97% | -0.58%p |
+| TweetSentimentExtractionClassification | 42.38% | 52.8% | +10.42%p |
+| TwentyNewsgroupsClustering.v2 | 12.68% | 12.96% | +0.28%p |
+
+
+## Training
+
+### Stage 1: Model Compression
+- **Teacher**: `intfloat/multilingual-e5-small` (12L, 384d)
+- **Compression**: Layer pruning + Vocab pruning
+- **Result**: 5L / 384d / 8,675 vocab
+
+### Stage 2: Knowledge Distillation
+- **Method**: MSE + Cosine Similarity loss
+- **Data**: MTEB Classification/Clustering/STS task datasets
+- **Optimizer**: AdamW (lr=2e-5, weight_decay=0.01)
+- **Schedule**: Cosine annealing over 3 epochs
+
+
+## Supported Languages (18)
+
+ko, en, ja, zh, es, fr, de, pt, it, ru, ar, hi, th, vi, id, tr, nl, pl

config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 384,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 5,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "transformers_version": "4.56.2",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 8675
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "model_type": "SentenceTransformer",
+  "__version__": {
+    "sentence_transformers": "5.3.0",
+    "transformers": "4.56.2",
+    "pytorch": "2.10.0+cu128"
+  },
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:504814e0b78cfde32f085153eda926e8e0b395756de301be97c064b26bf0c74e
+size 50207568

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+    "max_seq_length": 512,
+    "do_lower_case": false
+}

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
+size 5069051

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,63 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8674": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "max_length": 64,
+  "model_max_length": 512,
+  "pad_to_multiple_of": null,
+  "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "</s>",
+  "sp_model_kwargs": {},
+  "stride": 0,
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "<unk>"
+}