sentence-transformers
Safetensors
bert
embeddings
retrieval
northeast-india
low-resource
multilingual
RAG
Instructions to use MWirelabs/ne-embed with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use MWirelabs/ne-embed with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("MWirelabs/ne-embed") sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -38,7 +38,7 @@ base_model: sentence-transformers/LaBSE
|
|
| 38 |
|
| 39 |
## What is NE-Embed?
|
| 40 |
|
| 41 |
-
NE-Embed
|
| 42 |
|
| 43 |
Trained on 201,738 balanced parallel pairs across 10 languages using bi-encoder fine-tuning with `MultipleNegativesRankingLoss`.
|
| 44 |
|
|
@@ -89,7 +89,7 @@ Evaluated on 500 samples per language. CLRI = Cross-Language Retrieval Interfere
|
|
| 89 |
```python
|
| 90 |
from sentence_transformers import SentenceTransformer
|
| 91 |
|
| 92 |
-
model = SentenceTransformer("MWirelabs/ne-embed
|
| 93 |
|
| 94 |
sentences = [
|
| 95 |
"Where is the nearest hospital?", # English
|
|
@@ -117,7 +117,17 @@ score = 0.7 * ne_embed_score + 0.3 * bm25_score
|
|
| 117 |
- **Loss:** `MultipleNegativesRankingLoss`
|
| 118 |
- **Data:** 201,738 English↔NE language parallel pairs, capped at 25k per language to prevent Assamese attractor bias
|
| 119 |
- **Epochs:** 3 · **Batch size:** 64 · **Max seq length:** 128
|
| 120 |
-
- **Hardware:** NVIDIA A40 · **Training time:** ~1.3 hours
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
|
| 122 |
---
|
| 123 |
|
|
|
|
| 38 |
|
| 39 |
## What is NE-Embed?
|
| 40 |
|
| 41 |
+
NE-Embed A multilingual text embedding model purpose-built for Northeast Indian languages. It enables **semantic search, retrieval, and RAG** across languages that general multilingual models fail on — languages like Garo, Meitei, Nyishi, and Pnar where raw LaBSE achieves under 15% retrieval accuracy.
|
| 42 |
|
| 43 |
Trained on 201,738 balanced parallel pairs across 10 languages using bi-encoder fine-tuning with `MultipleNegativesRankingLoss`.
|
| 44 |
|
|
|
|
| 89 |
```python
|
| 90 |
from sentence_transformers import SentenceTransformer
|
| 91 |
|
| 92 |
+
model = SentenceTransformer("MWirelabs/ne-embed")
|
| 93 |
|
| 94 |
sentences = [
|
| 95 |
"Where is the nearest hospital?", # English
|
|
|
|
| 117 |
- **Loss:** `MultipleNegativesRankingLoss`
|
| 118 |
- **Data:** 201,738 English↔NE language parallel pairs, capped at 25k per language to prevent Assamese attractor bias
|
| 119 |
- **Epochs:** 3 · **Batch size:** 64 · **Max seq length:** 128
|
| 120 |
+
- **Hardware:** 1× NVIDIA A40 (48 GB) · **Training time:** ~1.3 hours
|
| 121 |
+
|
| 122 |
+
---
|
| 123 |
+
|
| 124 |
+
## Intended Uses
|
| 125 |
+
|
| 126 |
+
- Semantic search
|
| 127 |
+
- Dense retrieval
|
| 128 |
+
- RAG
|
| 129 |
+
- Cross-lingual retrieval
|
| 130 |
+
- Clustering
|
| 131 |
|
| 132 |
---
|
| 133 |
|