| --- |
| title: Rabbinic Embedding Benchmark |
| emoji: 📚 |
| colorFrom: blue |
| colorTo: purple |
| sdk: gradio |
| sdk_version: 4.44.0 |
| app_file: app.py |
| pinned: false |
| license: mit |
| datasets: |
| - Sefaria/Rabbinic-Hebrew-English-Pairs |
| - Sefaria/Rabbinic-Embedding-Leaderboard |
| --- |
| |
| # Rabbinic Hebrew/Aramaic Embedding Benchmark |
|
|
| Evaluate embedding models on cross-lingual retrieval between Hebrew/Aramaic source texts and their English translations from Sefaria. |
|
|
| ## How It Works |
|
|
| Given a Hebrew/Aramaic text, can the model find its correct English translation from a pool of candidates? Models that excel at this task produce high-quality embeddings for Rabbinic literature. |
|
|
| ## Metrics |
|
|
| | Metric | Description | |
| |--------|-------------| |
| | **MRR** | Mean Reciprocal Rank (average of 1/rank of correct answer) | |
| | **Recall@k** | % of queries where correct translation is in top k results | |
| | **Bitext Accuracy** | True pair vs random pair classification | |
|
|
| ## Corpus |
|
|
| The benchmark uses the [Sefaria/Rabbinic-Hebrew-English-Pairs](https://huggingface.co/datasets/Sefaria/Rabbinic-Hebrew-English-Pairs) dataset, which includes diverse texts with English translations: |
|
|
| - **Talmud**: Bavli & Yerushalmi |
| - **Mishnah**: Selected tractates |
| - **Midrash**: Midrash Rabbah |
| - **Commentary**: Rashi, Ramban, Radak, Rabbeinu Behaye |
| - **Philosophy**: Guide for the Perplexed, Sefer HaIkkarim |
| - **Hasidic/Kabbalistic**: Likutei Moharan, Tomer Devorah, Kalach Pitchei Chokhmah |
| - **Mussar**: Chafetz Chaim, Kav HaYashar, Iggeret HaRamban |
| - **Halacha**: Sefer HaChinukh, Mishneh Torah |
|
|
| All texts sourced from [Sefaria](https://www.sefaria.org). |
|
|
| ## Leaderboard |
|
|
| Results are stored persistently in the [Sefaria/Rabbinic-Embedding-Leaderboard](https://huggingface.co/datasets/Sefaria/Rabbinic-Embedding-Leaderboard) dataset. |
|
|
| ## Configuration (Space Secrets) |
|
|
| The following environment variables can be set in Space settings: |
|
|
| ### Required for Leaderboard Persistence |
|
|
| | Secret | Description | |
| |--------|-------------| |
| | `HF_TOKEN` | HuggingFace token with write access to `Sefaria/Rabbinic-Embedding-Leaderboard`. Without this, evaluations will run but results won't be saved to the leaderboard. | |
|
|
| ### Optional for API-based Models |
|
|
| | Secret | Description | |
| |--------|-------------| |
| | `OPENAI_API_KEY` | For OpenAI embedding models | |
| | `VOYAGE_API_KEY` | For Voyage AI embedding models | |
| | `GEMINI_API_KEY` | For Google Gemini embedding models | |
|
|
| Users can also enter API keys directly in the interface (they are not stored). |
|
|
| ## Local Development |
|
|
| ```bash |
| # Clone and install dependencies |
| git clone https://huggingface.co/spaces/Sefaria/Rabbinic-Embedding-Benchmark |
| cd Rabbinic-Embedding-Benchmark |
| pip install -r requirements.txt |
| |
| # Run locally (leaderboard will be read-only without HF_TOKEN) |
| python app.py |
| |
| # Or with write access to leaderboard |
| export HF_TOKEN=your_token_here |
| python app.py |
| ``` |
|
|
| ## Related |
|
|
| - [Benchmark Dataset](https://huggingface.co/datasets/Sefaria/Rabbinic-Hebrew-English-Pairs) |
| - [Leaderboard Dataset](https://huggingface.co/datasets/Sefaria/Rabbinic-Embedding-Leaderboard) |
| - [Sefaria](https://www.sefaria.org) |
|
|
