---
base_model: lightonai/LateOn
library_name: litembeddings
license: apache-2.0
language:
- en
pipeline_tag: sentence-similarity
tags:
- gguf
- litembeddings
- modernbert
- colbert
- late-interaction
- pylate
- retrieval
- quantized
---

# LateOn Q4_0 for litembeddings

A compact Q4_0 GGUF conversion of [lightonai/LateOn](https://huggingface.co/lightonai/LateOn), a 149M-parameter ModernBERT ColBERT retrieval model. This speed/size variant is built for [litembeddings](https://github.com/alexandernicholson/litembeddings) and includes the model's complete three-stage PyLate projection folded into one 768-to-128 matrix.

Q4_0 is intended for high-throughput packed indexing and retrieval. It is approximately 40% smaller than the Q8_0 conversion and reached approximately 29.9k tokens/s at packed saturation on an Apple M4 Max, versus approximately 20.9k tokens/s for Q8_0 under the same benchmark. Very short single requests are dominated by dispatch overhead and are not consistently faster; use [Q8_0](https://huggingface.co/embedme/lightonai-lateon-Q8_0) when maximum numerical fidelity matters more than size or packed throughput.

## Files

| File | Purpose | SHA-256 |
|---|---|---|
| `LateOn-Q4_0.gguf` | Q4_0 ModernBERT encoder | `6562738e6ea23e59ad739c7c792afe0e2c64c57158dd65f08109a30bb987f1c6` |
| `LateOn-Q4_0.projection` | Folded 768-to-128 PyLate projection | `3e7fd24683cbcde958448d2d7120d7f12d4d0abcb67bed47e19e7a3170a5fa0a` |

Both files are required. The GGUF encoder alone produces 768-dimensional token vectors; litembeddings applies and normalizes the supplied projection to produce the model's intended 128-dimensional token vectors.

## litembeddings usage

```sql
.load ./litembeddings

SELECT lembed_model(
  '/path/to/LateOn-Q4_0.gguf',
  json_object(
    'colbert_projection', '/path/to/LateOn-Q4_0.projection',
    'ctx_size', 300,
    'batch_size', 300
  )
);

-- Full-precision token vectors.
SELECT lembed_tokens('What is SQLite WAL mode?');

-- Compact int8 token vectors for storage and MaxSim search.
SELECT lembed_tokens_quantized('What is SQLite WAL mode?');

SELECT lembed_maxsim(
  lembed_tokens('What is SQLite WAL mode?'),
  lembed_tokens('WAL mode lets SQLite readers coexist with one writer.')
);
```

LateOn was trained for queries up to 32 tokens and documents up to 300 tokens. Use the same preprocessing and retrieval conventions as the source model when comparing quality results.

## Validation

An end-to-end parity check against the source FP32 Transformers/PyLate pipeline over 331 output tokens produced per-token cosine similarity `min=0.994162`, `mean=0.998347`.

A 216-document FastAPI retrieval evaluation with 50 queries retained the same measured ranking metrics as exact search with this artifact: MRR `0.8284`, Hits@1/5/10 `74% / 94% / 96%`, and weighted score `92/100`. This is a small validation corpus, not a replacement for task-specific evaluation; Q4 quantization can affect other corpora or near-tied rankings.

## Conversion provenance

- Source: [`lightonai/LateOn`](https://huggingface.co/lightonai/LateOn)
- Source revision: `c01907b70557ee5c7753680d4819a5cce1674b83`
- Converted: 2026-07-14
- llama.cpp revision: `6eddde06a4f25d55d538b5d15628dcc2b6882147`
- Quantization: Q4_0 from the F16 GGUF conversion
- Projection: `W3 @ (W2 + R2) @ (W1 + R1)`, preserving both learned dimension-changing PyLate residuals

## License and attribution

The source model is released under Apache 2.0. See the [LateOn model card](https://huggingface.co/lightonai/LateOn) for training details, evaluation results, intended use, limitations, and citation information. This repository is an independent conversion and is not affiliated with LightOn.