README.md

---
license: afl-3.0
datasets:
- wikimedia/structured-wikipedia
- xlangai/spider
- birdsql/bird_sql_dev_20251106
- wenhu/tab_fact
- hfhgj/wikitables
- DongfuJiang/FeTaQA
- target-benchmark/ottqa-corpus
- target-benchmark/ottqa-corpus

tags:
- tabular
- strucutred
- dataset
- table_search
- https://wikidbs.github.io

---

![TABERTA_cover](https://cdn-uploads.huggingface.co/production/uploads/62f3b239279252f3f1145456/NlsYvpzEhgHH4wH6LGLzm.png)
>*TABERTA visualized: a wise owl learning a relevance function f(q,T) while riding a dragon that brute-forces the table space ... fast, fearless, and only slightly less chaotic thanks to structure-aware supervision (illustration generated with ChatGPT 5.2).
*

# TABERTA: Structure-Aware Table Retrieval with Bi-Encoders

TABERTA is a structure-aware fine-tuning framework for learning dense representations of relational tables.
Unlike approaches that flatten tables into unstructured text, TABERTA explicitly exposes **schema structure and table content** through controlled serialization views and retrieval-oriented training objectives.

The resulting table encoders support **accurate and generalizable table retrieval** across heterogeneous tasks, including:
- ad-hoc dataset discovery,
- question answering evidence selection,
- fact verification,
- and schema grounding for text-to-SQL.

This repository provides:
- the TABERTA codebase (training, serialization, evaluation),
- and **7 fine-tuned table encoders**, released via Hugging Face.

---

## Core Idea

Given a natural-language query \( q \) and a corpus of tables \( \mathcal{T} \), TABERTA learns an encoder
\( E(\cdot) \) such that relevant tables are ranked highly using standard similarity search.

Two design choices are central:
1. **Serialization View** — how table structure and content are exposed to the encoder.
2. **Fine-Tuning Objective** — how retrieval relevance is learned.

TABERTA systematically studies the interaction between these choices.

---

## Table Serialization Views

TABERTA supports three complementary serialization strategies:

### SchemaView
Encodes only schema-level information (table name, column names, types).

- Emphasizes structural and semantic intent.
- Robust to noise and large tables.
- Best suited for **ad-hoc table retrieval** and dataset discovery.

### RowView
Encodes individual rows paired with schema context.

- Grounds semantics in concrete values.
- Supports evidence-based retrieval.
- Useful when relevance depends on specific tuples.

### Hybrid / FullView
Combines schema information with sampled or aggregated table content.

- Balances abstraction and grounding.
- Most general and consistently effective across tasks.
- Used as the default in cross-benchmark evaluation.

---

## Released Models

All models are bi-encoders initialized from a sentence-transformer backbone and fine-tuned for table retrieval.
They differ in supervision signal and training objective.

### 1. Pairwise Contrastive (P/N Pair)
**Supervision:** Supervised  
**Objective:** Pairwise contrastive (positive vs. negative tables)  
**Serialization:** SchemaView / Hybrid  

Learns explicit relevance boundaries between matching and non-matching tables.
Strong baseline when labeled query–table pairs are available.

---

### 2. Triplet Contrastive (TC)
**Supervision:** Supervised  
**Objective:** Triplet loss (anchor, positive, negative)  
**Serialization:** SchemaView / Hybrid  

Encourages relative ranking rather than absolute separation.
More stable than pairwise training in heterogeneous corpora.

---

### 3. Optimized Triplet Contrastive (TC-opt)
**Supervision:** Supervised  
**Objective:** Triplet loss with hard-negative mining  
**Serialization:** SchemaView / Hybrid  

Improves discrimination in large repositories where many tables are semantically close.
Best suited for high-recall retrieval settings.

---

### 4. Self-Supervised Contrastive (SimCSE-style)
**Supervision:** Self-supervised  
**Objective:** Contrastive learning via stochastic dropout views  
**Serialization:** SchemaView / Hybrid  

Does not require relevance labels.
Captures structural regularities and semantic consistency across table representations.

---

### 5. Masked Language Modeling (MLM)
**Supervision:** Self-supervised  
**Objective:** Token-level reconstruction  
**Serialization:** FullView  

Focuses on contextual encoding of table text.
Useful as a pretraining signal but weaker alone for retrieval without contrastive supervision.

---

### 6. Hybrid (MLM → Contrastive)
**Supervision:** Self-supervised + Supervised  
**Objective:** Two-stage (MLM pretraining followed by contrastive fine-tuning)  
**Serialization:** Hybrid / FullView  

Combines representation quality with retrieval alignment.
Provides strong and stable performance across tasks.

---

### 7. Unified Hybrid (Recommended)
**Supervision:** Mixed  
**Objective:** Retrieval-oriented contrastive fine-tuning  
**Serialization:** Hybrid  

A single encoder trained to generalize across:
- table retrieval,
- question answering evidence selection,
- fact verification,
- and schema grounding.

This is the **default model** used in the paper.

---

## Dataset corpus 
### **Training TABERTA Corpus**
TABERTA is fine-tuned on WikiDBs, a large-scale corpus of relational databases automatically extracted from Wikidata, containing over 100K databases and 1.6M tables spanning diverse domains and realistic schema designs. WikiDBs is used only for representation learning no downstream benchmark queries or relevance labels are observed during training by making the learned embeddings reusable across tasks.


### **Experiencing and evaluating Corpus**
 To evaluate generalization, TABERTA is tested on heterogeneous table-retrieval benchmarks covering schema-driven search (WikiTables), value-grounded evidence retrieval (TabFact, FeTaQA, OTTQA), and schema/table grounding (Spider, BIRD). All datasets are publicly available: WikiDBs can be downloaded from https://github.com/DataManagementLab/WikiDBs, WikiTables from https://github.com/zhangshuo1014/WikiTable, TabFact from https://github.com/wenhuchen/TabFact, FeTaQA from https://github.com/czyssrs/FeTaQA, OTTQA from https://github.com/wenhuchen/OTT-QA, Spider from https://yale-lily.github.io/spider, and BIRD from https://bird-bench.github.io. These resources allow users to directly reproduce the retrieval setting described in the paper and experiment with TABERTA on realistic dataset discovery scenarios.


---

## Usage

### Load a TABERTA Encoder

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("TABERTA/7_hybrid_model_reg")
## you can try all 7 variances of the fine-tuned models and compare between all of them 

## Encode Tables


table_embeddings = model.encode(
    serialized_tables,
    normalize_embeddings=True,
    show_progress_bar=True
)




## Encode Queries and Retrieve
query_embedding = model.encode(query, normalize_embeddings=True)
scores = table_embeddings @ query_embedding
top_k = scores.argsort()[-k:][::-1]


##  TBA :)
@inproceedings{taberta,
  title     = {TABERTA: Structure-Aware Fine-Tuning of Bi-Encoders for Table Retrieval},
  author    = {…},
  booktitle = {…},
  year      = {2026}
}