---
license: mit
datasets:
- MhoOmm/Question_answering_using_retrival_based_context
- hotpotqa/hotpot_qa
language:
- en
base_model:
- microsoft/deberta-v3-base
pipeline_tag: question-answering
tags:
- question_answering
- debert
- hotpotqa
- extractive-question-answering
- open-domain-qa
- multi-hop-reasoning
---
# DeBERTa-v3-base for Open-Domain Multi-Hop Question Answering

A fine-tuned **DeBERTa-v3-base** model for **extractive question answering** on the **HotpotQA** dataset. The model is designed to work as the **reader component** of a Retrieval-Augmented Question Answering (QA) pipeline, where a dense retriever first retrieves relevant documents and the reader extracts the answer span from the retrieved context.

---

## Model Overview

- **Base Model:** `microsoft/deberta-v3-base`
- **Task:** Extractive Question Answering
- **Framework:** Hugging Face Transformers
- **Dataset:** HotpotQA (Distractor Setting)
- **Language:** English

Unlike generative language models, this model predicts the **exact answer span** from a retrieved context instead of generating free-form text.

---

# Pipeline Architecture

```
Question
      │
      ▼
Dense Retriever (BGE-base-en-v1.5 + FAISS)
      │
Top-10 Retrieved Documents
      │
      ▼
DeBERTa-v3-base Reader
      │
Predicted Answer Span
      │
      ▼
(Optional) LLM Explanation Layer
```

The retriever identifies relevant Wikipedia articles, while DeBERTa extracts the answer span from the retrieved passages.

---

# Dataset

The model was fine-tuned on the **HotpotQA** dataset.

### Data Preparation

For each question:

1. Retrieve the top-10 Wikipedia articles using a dense retriever.
2. Assemble the retrieved passages into a context.
3. If retrieval misses supporting documents, use the gold supporting passages from HotpotQA.
4. Remove:
   - Yes/No questions
   - Samples where the answer cannot be aligned within the context

### Final Dataset Size

| Split | Examples |
|--------|---------:|
| Training | 84,959 |
| Validation | 4,687 |

---

# Retriever

The reader depends on an external dense retriever.

### Embedding Model

- **BAAI/bge-base-en-v1.5**

### Vector Database

- FAISS Flat Index

### Corpus

- 507,494 Wikipedia articles

### Embedding Dimension

- 768

### Retrieval Performance

| Metric | Score |
|---------|------:|
| Recall@3 | 0.7100 |
| Recall@5 | 0.7650 |
| Recall@10 | **0.8150** |
| Recall@20 | 0.8200 |

---

# Training Configuration

| Parameter | Value |
|-----------|-------|
| Base Model | microsoft/deberta-v3-base |
| Optimizer | AdamW |
| Learning Rate | 2e-5 |
| Weight Decay | 0.01 |
| Epochs | 3 |
| Batch Size | 8 |
| Max Sequence Length | 512 |
| Document Stride | 128 |

---

# Tokenization

The model uses the official DeBERTa tokenizer.

Configuration:

- `truncation="only_second"`
- `padding="max_length"`
- `max_length=512`
- `stride=128`
- `return_overflowing_tokens=True`
- `return_offsets_mapping=True`

Answer spans are aligned using character-level offsets.

Windows that do not contain the answer are assigned start/end position 0.

---

# Evaluation

Performance on the HotpotQA validation set:

| Metric | Score |
|---------|------:|
| Exact Match (EM) | **60.53** |
| F1 Score | **74.21** |

Comparison with other transformer readers:

| Model | EM | F1 |
|-------|----:|----:|
| **DeBERTa-v3-base** | **60.53** | **74.21** |
| RoBERTa-base | 60.17 | 73.96 |
| ALBERT-base-v2 | 56.73 | 70.31 |

---

# Example Usage

```python
from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="YOUR_USERNAME/deberta-hotpotqa",
    tokenizer="YOUR_USERNAME/deberta-hotpotqa"
)

result = qa_pipeline(
    question="Which magazine was started first, Arthur's Magazine or First for Women?",
    context=context
)

print(result["answer"])
```

---

# Intended Use

This model is suitable for:

- Open-domain Question Answering
- Multi-hop Question Answering
- Educational Assistants
- Search Systems
- Retrieval-Augmented QA Pipelines
- Research in Extractive Question Answering

---

# Limitations

This model:

- Requires an external retrieval system.
- Cannot retrieve documents by itself.
- Produces extractive answer spans only.
- Does not generate explanations.
- Performance depends heavily on retrieval quality.
- Was trained only on English Wikipedia.
- Does not support Yes/No questions since they were removed during preprocessing.

---

# Future Improvements

Potential improvements include:

- Hybrid BM25 + Dense Retrieval
- Cross-Encoder Re-ranking
- Paragraph-level indexing
- Iterative Multi-hop Retrieval
- Confidence estimation
- Citation generation
- Integration with an instruction-tuned LLM for grounded answer explanations

---

---

# License

This project is intended for research and educational purposes. Please ensure compliance with the licenses of the original HotpotQA dataset and the DeBERTa-v3-base model.