---
base_model: meta-llama/Meta-Llama-3-8B
library_name: peft
license: llama3
tags:
- sparse-retrieval
- information-retrieval
- msmarco
---

# AdaSparse-8B (LLaMA-3-8B, MS MARCO)

AdaSparse sparse retriever built on `meta-llama/Meta-Llama-3-8B`, trained on MS MARCO with
contrastive + knowledge-distillation loss and a learned per-term threshold. This repository
contains the LoRA adapter (including the learned `q_thres`/`d_thres` thresholding modules),
the tokenizer, and the retriever config.

## Usage

Requires the [adasparse](https://github.com/YOUR_GITHUB/adasparse) codebase:

```python
import torch
from transformers import AutoTokenizer
from scaling_retriever.modeling.llm_encoder import LlamaBiSparse

model = LlamaBiSparse.load_from_lora("YOUR_HF_USERNAME/adasparse-8B")
tokenizer = AutoTokenizer.from_pretrained("YOUR_HF_USERNAME/adasparse-8B")

queries = ["What is the capital of France?"]
passages = ["Paris is the capital of France."]

tokenized_queries = tokenizer(queries, max_length=192, truncation=True,
                              padding="longest", return_tensors="pt")
tokenized_passages = tokenizer(passages, max_length=192, truncation=True,
                               padding="longest", return_tensors="pt")

query_embeds = model.query_encode(**tokenized_queries)
doc_embeds = model.doc_encode(**tokenized_passages)
scores = torch.matmul(query_embeds, doc_embeds.T)
```

Note: the base model `meta-llama/Meta-Llama-3-8B` is gated — request access on its model page first.