---
license: apache-2.0
language:
  - sa
  - bo
  - zh
  - pi
  - en
tags:
  - embeddings
  - retrieval
  - buddhist-studies
  - sanskrit
  - tibetan
  - classical-chinese
  - pali
library_name: transformers
pipeline_tag: feature-extraction
---

# mitra-qwen35-embedder

A 9B dense retrieval embedder for classical Buddhist literature, built on
Qwen3.5-9B with large-scale domain continued-pretraining, SFT, and contrastive
finetuning on cross-lingual parallel data (Sanskrit, Tibetan, Buddhist Chinese,
Pāli, English), including hard-negative pairs mined from deep-research usage.

It is the embedding model behind the [Dharmamitra](https://dharmamitra.org)
search stack. Its defining capability is **asymmetric cross-lingual
retrieval**: matching a single sentence in one classical language to the
passage window containing its parallel in another — a task on which strong
general-purpose embedders (BGE-M3, Qwen3-Embedding) score near zero.

## Prompting template

**Queries** are wrapped in an instruction template using the two special
tokens `<instruct>` and `<query>` (added to the tokenizer vocabulary):

```
<instruct>{instruction}\n<query>{query text}
```

**Passages/documents are embedded as raw text — no template.**

The instruction used at training time:

```
Please find the semantically most similar text.
```

optionally with a target-language hint, which improves cross-lingual precision:

```
Please find the semantically most similar text in Tibetan.
Please find the semantically most similar text in Chinese.
```

Three conventions that materially affect quality:

1. **EOS is required** on every input (query and passage). The bundled
   tokenizer has `add_eos_token: true`, so `AutoTokenizer` handles this
   automatically — do not disable it.
2. **Left padding + last-token pooling.** The embedding is the final hidden
   state at the last (EOS) position, L2-normalized.
3. **Script conventions:** Tibetan works in **either native Tibetan script or
   Wylie transliteration** — we measure identical retrieval quality for both
   (the pretraining corpus is native script; the contrastive stage used
   Wylie). Sanskrit and Pāli should be **IAST** romanization; Chinese in
   Chinese script.

## Usage

```python
import torch
from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained("buddhist-nlp/mitra-qwen35-embedder")
tok.padding_side = "left"
model = AutoModel.from_pretrained(
    "buddhist-nlp/mitra-qwen35-embedder", dtype=torch.bfloat16, device_map="cuda"
).eval()

INSTR = "Please find the semantically most similar text."

def embed(texts, is_query=False, batch_size=16):
    if is_query:
        texts = [f"<instruct>{INSTR}\n<query>{t}" for t in texts]
    out = []
    with torch.no_grad():
        for i in range(0, len(texts), batch_size):
            enc = tok(texts[i:i+batch_size], padding=True, truncation=True,
                      max_length=512, return_tensors="pt").to(model.device)
            h = model(**enc).last_hidden_state[:, -1]          # last-token pooling
            out.append(torch.nn.functional.normalize(h.float(), dim=-1).cpu())
    return torch.cat(out)

q = embed(["evaṃ mayā śrutam ekasmin samaye"], is_query=True)
p = embed(["'di skad bdag gis thos pa dus gcig na", "如是我聞一時"])
print(q @ p.T)   # cosine similarities
```

## Evaluation

R@1 on the mitra retrieval benchmark, **hardened setting: 25,000 in-language
distractor passages added to every candidate pool** (this model's own
measurements; Tibetan given in native script):

| task | this model | gemma-2-mitra-e | BGE-M3 |
|---|---|---|---|
| Multilingual parallels (sa/bo/zh) | **0.846** | 0.841 | 0.001 |
| Multilingual asymmetric (sentence→window) | **0.788** | 0.581 | 0.000 |
| English → classical | 0.876 | 0.879 | 0.068 |
| Verse → commentary | **0.483** | 0.384 | 0.138 |
| English question → passage | **0.523** | 0.473 | 0.020 |

**This checkpoint's own results** on gold pools (per-subtask pools without
added distractors):

| task | this model | gemma-2-mitra-e | BGE-M3 |
|---|---|---|---|
| Multilingual parallels (sa/bo/zh) | **0.861** | 0.850 | 0.007 |
| Multilingual asymmetric (sentence→window) | **0.813** | 0.654 | 0.002 |
| English → classical | 0.885 | 0.886 | 0.204 |
| Verse → commentary | **0.506** | 0.413 | 0.298 |
| English question → passage | **0.574** | 0.534 | 0.077 |

This checkpoint additionally leads the sibling on question→passage retrieval
(its distinguishing strength from the deep-research continuation training).

## Model details

- Base: Qwen3.5-9B after ~30B tokens of continued pretraining on classical
  Buddhist corpora and a domain SFT stage
- Contrastive finetuning: LoRA (r=32), InfoNCE with cross-device negatives,
  temperature 0.02, group size 8, 512-token inputs, followed by a short
  continuation on hard-negative pairs mined from deep-research citation logs
- Embedding dimension: 4096
- Languages: Sanskrit, Tibetan (Wylie), Buddhist Chinese, Pāli, English

## Citation

If you use this model, please cite the Dharmamitra project
(https://dharmamitra.org). A technical report is in preparation.