---
license: apache-2.0
language: en
base_model: lightonai/LateOn-unsupervised
pipeline_tag: feature-extraction
library_name: transformers
tags:
  - splade
  - sparse
  - retrieval
  - modernbert
  - sentence-transformers
  - sentence-similarity
---

<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/6273df31c3b822dad2d1eef2/pdQykzYo3cSUGnqm482Jj.png" width="500">
</p>

# SPARSEUP

> **TL;DR:** SPARSEUP is a strong and efficient sparse retriever trained on the same backbone family and fine-tuning data as DenseOn / LateOn (149M, ModernBERT). Strongest public vocabulary-based sparse encoder we know of under 150M on BEIR-13 (56.4 nDCG@10). Apache 2.0. Use it!

For the full story, see the **[blog post](https://www.linkup.so/blog/introducing-sparseup-by-linkup)**.

## About

LightOn released open data and a recipe to train **DenseOn** (single-vector) and **LateOn** (late interaction). **SPARSEUP** fills the missing slot: a learned sparse embedding model, so the three architectures can be compared with backbone and data held fixed. The model is initialized from `LateOn-unsupervised` with ModernBERT’s MLM `head` / `decoder.bias` grafted on (`decoder.weight` stays tied to LateOn embeddings). Contrastive fine-tuning on LightOn’s mixture (7 hard negatives sampled from 50, plus in-batch). No cross-encoder distillation.

Three extra knobs *vs* vanilla SPLADE-like model:
1. **`logit_shift=15`** — `log(1 + ReLU(x − 15))` so ReLU support starts sparse  
2. **`position_top_k=12`** — each token keeps 12 vocab dims max before pooling (expansion budget)  
3. **`vocab_fold=case_space`** — fold case / `Ġ` surface forms onto one id after pooling  (avoid duplication of surface forms = improve efficiency). Voc goes from 50*k* to ~34*k*.  

Prefixes `[Q] ` / `[D] ` are attended but not pooled. Scoring is **dot product**.

## BEIR (nDCG@10)

* *Controlled comparison against LateOn and DenseOn (same backbone / data). SPARSEUP uses Seismic (approximate); their numbers are exact search.*
| Model | Avg | Avg (w/o MS MARCO) |   
| --- | --- | --- |    
| LateOn | **57.9** | **58.9** |   
| DenseOn | 56.9 | 57.9 |  
| SPARSEUP | 55.4 | 56.4 |  

* *Comparison against competitive sparse encoders. Best model at <150M parameters.*
| Model	| Avg (wo/ MS MARCO) |
| --- | --- | 
| splade-v3	| 51.7 |
| granite-embedding-30m-sparse | 50.6 |
| ModernBERT-VT	| 52.4 |
| opensearch-neural-sparse-encoding-v1	| 52.44 |
| opensearch-neural-sparse-encoding-doc-v3-gte	| 54.6 |
| SPARSEUP	| **56.4** |
| LACONIC-1B**	| 58.7 |

## SPARSEUP is fast

Using dedicated infra (Seismic), it can reach sub-*ms* latency on MS MARCO (single-threaded).

<img src="https://cdn-uploads.huggingface.co/production/uploads/6273df31c3b822dad2d1eef2/Jjf8ZPgmiTkwi4hGz-gev.png" width="500">

## Model description

- **Model Type**: Sentence Transformer or bare Transformers (see below)
- **Encoder Type (sparse):** MLM → `log1p(ReLU(x − θ))` → per-position top-k → max-pool → vocab fold
- **Output Dimensionality**: ~34*k* (sparse)
- **Base:** ModernBERT-base (149M), from LateOn-unsupervised + grafted MLM head
- **Language:** English
- **Similarity:** dot
- **Query / doc prefixes:** `[Q] ` / `[D] `
- **Max lengths (eval):** query 128, document 512
- **License:** Apache 2.0


## Usage

### Transformers

```python
from transformers import AutoModel
model = AutoModel.from_pretrained("Linkup-Platform/linkup-sparseup-embed-v1", trust_remote_code=True)
```

#### Encoding

`encode_to_dict(texts, kind="document", top_k=None)` — encodes texts and returns one {token: weight} dict per text, sorted by descending weight. `kind` ("query"/"document") picks the prefix and max length; `top_k` caps how many terms you get back (`None` = all non-zero).

```python
queries = ["England football player highest paid", "NYC capital which country?"]
model.encode_to_dict(queries, kind="query", round_to=2, top_k=10)
[{'ĠEngland': 2.96,
  'Ġfootball': 2.68,
  'Ġpaid': 2.65,
  'Ġplayer': 2.57,
  'Ġhighest': 2.52,
  'Ġplayers': 2.19,
  'Ġpay': 2.04,
  'Ġenglish': 2.03,
  'Ġeng': 2.03,
  'Ġfootballer': 1.97},
 {'Ġcapital': 3.25,
  'ĠNYC': 3.12,
  'Ġcap': 3.12,
  'Ġcountry': 3.12,
  'Ġny': 3.09,
  'ĠYork': 2.87,
  'Ġcountries': 2.59,
  'Ġcapit': 2.55,
  'Ġnation': 2.47,
  'ĠCapitol': 2.45}]
```

#### Scoring

```python
documents = [
    "Harry Kane is the England captain and among the highest-paid footballers.",
    "New York City is in the United States, not a national capital.",
]
q = model.encode(queries, kind="query")
d = model.encode(documents, kind="document")
print(model.score(q, d))
#tensor([[113.2993,   0.0000],
#        [  7.0030,  99.1290]])
```

#### Diagnosis

We also provide utilities to better visually inspect the content of predictions.

* **prediction with attributions**: `render(texts, kind="document", top_k=25, width=36, color=None)` — returns a terminal bar chart string: one block per text, one line per expansion term with a heat-colored bar proportional to weight, plus the input subtoken that produced it (<- source@pos, <exp> for pure expansions). `width` is the bar length in characters; `color=None` auto-detects a TTY, so pass `color=True` in Jupyter.

```python
print(model.render(queries, kind="query", top_k=10))
```

```
query · England football player highest paid
   England █████████████████████████████████▓▒░   2.96  <- ·England@2
  football ██████████████████████████████▓▒░      2.68  <- football@3
      paid █████████████████████████████▓▒░       2.65  <- paid@6
    player ████████████████████████████▓▒░        2.57  <- player@4
   highest ████████████████████████████▓▒░        2.52  <- highest@5
   players ████████████████████████▓▒░            2.19  <- player@4 <exp>
       pay ██████████████████████▓▒░              2.04  <- paid@6 <exp>
   english ██████████████████████▓▒░              2.03  <- ·England@2 <exp>
       eng ██████████████████████▓▒░              2.03  <- ·England@2 <exp>
footballer █████████████████████▓▒░               1.97  <- player@4 <exp>

query · NYC capital which country?
  capital █████████████████████████████████▓▒░   3.25  <- capital@4
      NYC ████████████████████████████████▓▒░    3.12  <- ·C@3 <exp>
      cap ████████████████████████████████▓▒░    3.12  <- capital@4 <exp>
  country ████████████████████████████████▓▒░    3.12  <- country@6
       ny ███████████████████████████████▓▒░     3.09  <- ·NY@2
     York █████████████████████████████▓▒░       2.87  <- ·NY@2 <exp>
countries ██████████████████████████▓▒░          2.59  <- country@6 <exp>
    capit █████████████████████████▓▒░           2.55  <- capital@4 <exp>
   nation ████████████████████████▓▒░            2.47  <- country@6 <exp>
  Capitol ████████████████████████▓▒░            2.45  <- capital@4 <exp>
```

* **visual attribution**: `highlight(texts, kind="document", reduce="sum", color=None)` — returns the original texts with their firing words lit up, intensity being each word's share of the vector's total mass (`reduce="sum"`, default) or its single strongest term (`reduce="max"`). On a TTY it uses reverse-video heat colors; in plain mode it wraps words in tiered markers ⟦strong⟧ «mid» ‹weak› (same `color=True` caveat for Jupyter).

```python
document = ["The presence of communication amid scientific minds was equally important to the success of the Manhattan Project as scientific intellect was. The only cloud hanging over the impressive achievement of the atomic researchers and engineers is what their success truly meant; hundreds of thousands of innocent lives obliterated."]
print(model.highlight(document, kind="document", color=True))
# type: use color=True for a more visual rendering. ⟦ ⟧ / « » / ‹ › represent strong / mid / weak attributions.
```

```
‹The› ⟦presence⟧ «of» ⟦communication⟧ ⟦amid⟧ ⟦scientific⟧ ⟦minds⟧ «was» ⟦equally⟧ ⟦important⟧ «to» the ⟦success⟧ «of» «the» ⟦Manhattan⟧ ⟦Project⟧ «as» ‹scientific› ⟦intellect⟧ was«.» ‹The› «only» ⟦cloud⟧ ⟦hanging⟧ «over» the ⟦impressive⟧ ⟦achievement⟧ of the ⟦atomic⟧ ⟦researchers⟧ ‹and› ⟦engineers⟧ ‹is› what their «success» «truly» ⟦meant;⟧ ⟦hundreds⟧ «of» ⟦thousands⟧ «of» ⟦innocent⟧ ⟦lives⟧ ⟦obliterated⟧.
```
<img src="https://cdn-uploads.huggingface.co/production/uploads/6273df31c3b822dad2d1eef2/E05aAsyYwDzWCrha9EPhP.png" width="1000">

### Sentence Transformers

The model can be used with Sentence Transformers. First install the lib:  

```python
pip install -U sentence-transformers
```

```python
from sentence_transformers import SparseEncoder
model = SparseEncoder("Linkup-Platform/linkup-sparseup-embed-v1", trust_remote_code=True)
queries = [
    "England football player highest paid",
    "NYC capital which country?",
]
q = model.encode_query(queries)
print(model.decode(q, top_k=10))
# [[('ĠEngland', 2.96), ('Ġfootball', 2.68), ...],
#  [('Ġcapital', 3.25), ('ĠNYC', 3.12), ...]]
```

```python
documents = [
    "Harry Kane is the England captain and among the highest-paid footballers.",
    "New York City is in the United States, not a national capital.",
]
d = model.encode_document(documents)
print(model.similarity(q, d))
# tensor([[113.29,   0.00],
#         [  7.00,  99.13]])
```