Upload gte_L4_uniform_distilled (distilled) from GTE-multilingual

3b478ea verified 4 months ago

10.1 kB

language:
  - ko
  - en
  - ja
  - zh
  - es
  - fr
  - de
  - pt
  - it
  - ru
  - ar
  - hi
  - th
  - vi
  - id
  - tr
  - nl
  - pl
tags:
  - sentence-transformers
  - multilingual
  - layer-pruning
  - vocab-pruning
  - knowledge-distillation
  - gte-multilingual
library_name: sentence-transformers
pipeline_tag: sentence-similarity
license: apache-2.0

gte_L4_uniform_distilled (Distilled)

Lightweight sentence encoder created from alibaba-NLP/gte-multilingual-base via layer pruning + vocabulary pruning + knowledge distillation.

Model Details

Property	Value
Teacher	alibaba-NLP/gte-multilingual-base
Architecture	GTE-multilingual (pruned)
Hidden dim	768
Layers	4 / 12
Layer indices	[0, 4, 7, 11]
Strategy	4 layers, evenly spaced from GTE-multilingual (12L)
Parameters	220,757,760
Model size (FP32)	277.7MB
Distilled	Yes

Architecture

==============================================================
  TEACHER: GTE-multilingual  →  STUDENT: 4L / 57,376 vocab
==============================================================

            TEACHER                        STUDENT          
  ───────────────────────────    ───────────────────────────

  ┌─────────────────────────┐    ┌─────────────────────────┐
  │   Input Tokens          │    │   Input Tokens          │
  └────────────┬────────────┘    └────────────┬────────────┘
               │                              │
  ┌────────────┴────────────┐    ┌────────────┴────────────┐
  │  Embeddings             │    │  Embeddings (pruned)    │
  │  vocab: 250,048         │    │  vocab:  57,376         │
  │  dim:  768              │    │  dim:  768              │
  └────────────┬────────────┘    └────────────┬────────────┘
               │                              │
  ┌─────────────────────────┐    ┌─────────────────────────┐
  │  Layer  0               │ ──►  │  Layer  0 ← L0         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  1               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer  2               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer  3               │  ╳   │                         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  4               │ ──►  │  Layer  1 ← L4         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  5               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer  6               │  ╳   │                         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  7               │ ──►  │  Layer  2 ← L7         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer  8               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer  9               │  ╳   │                         │
  ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤    │                         │
  │  Layer 10               │  ╳   │                         │
  ├─────────────────────────┤    ├─────────────────────────┤
  │  Layer 11               │ ──►  │  Layer  3 ← L11        │
  └────────────┬────────────┘    └────────────┬────────────┘
               │                              │
  ┌────────────┴────────────┐    ┌────────────┴────────────┐
  │  Mean Pooling           │    │  Mean Pooling           │
  │  → 768d embedding       │    │  → 768d embedding       │
  └─────────────────────────┘    └─────────────────────────┘

  Size: 1058.2MB (FP32)           →  277.7MB (FP32)
  Params: 277,405,440        →  72,785,664
  Reduction: 73.8%
==============================================================

Quick Start

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("gte_L4_uniform_distilled", trust_remote_code=True)

sentences = [
    "Hello, how are you?",
    "안녕하세요",
    "Bonjour, comment allez-vous?",
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (3, 768)

MTEB Evaluation Results

Overall Average: 56.97%

Task Group	Average
Classification	63.0%
Clustering	35.73%
STS	70.49%

Classification

Task	Average	Details
AmazonCounterfactualClassification	69.34%	en-ext: 74.06%, en: 72.1%, de: 69.15%
Banking77Classification	82.7%	default: 82.7%
ImdbClassification	60.94%	default: 60.94%
MTOPDomainClassification	81.66%	en: 89.15%, es: 84.21%, fr: 82.21%
MassiveIntentClassification	40.9%	en: 70.82%, zh-CN: 68.47%, ja: 66.5%
MassiveScenarioClassification	46.01%	en: 76.68%, zh-CN: 75.65%, ja: 72.94%
ToxicConversationsClassification	63.41%	default: 63.41%
TweetSentimentExtractionClassification	59.05%	default: 59.05%

Clustering

Task	Average	Details
ArXivHierarchicalClusteringP2P	52.87%	default: 52.87%
ArXivHierarchicalClusteringS2S	47.21%	default: 47.21%
BiorxivClusteringP2P.v2	25.97%	default: 25.97%
MedrxivClusteringP2P.v2	29.68%	default: 29.68%
MedrxivClusteringS2S.v2	24.9%	default: 24.9%
StackExchangeClustering.v2	43.5%	default: 43.5%
StackExchangeClusteringP2P.v2	34.78%	default: 34.78%
TwentyNewsgroupsClustering.v2	26.93%	default: 26.93%

STS

Task	Average	Details
BIOSSES	67.24%	default: 67.24%
SICK-R	73.92%	default: 73.92%
STS12	73.6%	default: 73.6%
STS13	76.98%	default: 76.98%
STS14	75.26%	default: 75.26%
STS15	84.9%	default: 84.9%
STS17	58.52%	en-en: 83.29%, es-es: 79.59%, ko-ko: 70.34%
STS22.v2	44.26%	zh: 68.37%, es: 61.22%, it: 60.83%
STSBenchmark	79.77%	default: 79.77%

Distillation Impact

Task	Before	After	Delta
AmazonCounterfactualClassification	65.24%	69.34%	+4.1%p
ArXivHierarchicalClusteringP2P	50.97%	52.87%	+1.9%p
ArXivHierarchicalClusteringS2S	43.38%	47.21%	+3.83%p
Banking77Classification	68.58%	82.7%	+14.12%p
BiorxivClusteringP2P.v2	20.78%	25.97%	+5.19%p
BIOSSES	42.61%	67.24%	+24.63%p
ImdbClassification	63.28%	60.94%	-2.34%p
MassiveIntentClassification	35.71%	40.9%	+5.19%p
MassiveScenarioClassification	37.58%	46.01%	+8.43%p
MedrxivClusteringP2P.v2	26.37%	29.68%	+3.31%p
MedrxivClusteringS2S.v2	20.98%	24.9%	+3.92%p
MTOPDomainClassification	68.67%	81.66%	+12.99%p
SICK-R	55.11%	73.92%	+18.81%p
StackExchangeClustering.v2	34.36%	43.5%	+9.14%p
StackExchangeClusteringP2P.v2	31.55%	34.78%	+3.23%p
STS12	47.97%	73.6%	+25.63%p
STS13	65.61%	76.98%	+11.37%p
STS14	57.02%	75.26%	+18.24%p
STS15	64.76%	84.9%	+20.14%p
STS17	17.95%	58.52%	+40.57%p
STS22.v2	40.55%	44.26%	+3.71%p
STSBenchmark	62.23%	79.77%	+17.54%p
ToxicConversationsClassification	57.84%	63.41%	+5.57%p
TweetSentimentExtractionClassification	48.1%	59.05%	+10.95%p
TwentyNewsgroupsClustering.v2	12.03%	26.93%	+14.9%p

Training

Stage 1: Layer Pruning

Teacher: alibaba-NLP/gte-multilingual-base (12 layers, 768d)
Selected layers: [0, 4, 7, 11] (4 layers, evenly spaced from GTE-multilingual (12L))
Vocabulary pruning applied

Stage 2: Knowledge Distillation

Method: MSE + Cosine Similarity loss
Data: MTEB Classification/Clustering/STS task datasets
Optimizer: AdamW (lr=2e-5, weight_decay=0.01)
Schedule: Cosine annealing over 3 epochs

Supported Languages (18)

ko, en, ja, zh, es, fr, de, pt, it, ru, ar, hi, th, vi, id, tr, nl, pl