File size: 3,709 Bytes
18bddbb | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | ---
base_model: lightonai/LateOn
library_name: litembeddings
license: apache-2.0
language:
- en
pipeline_tag: sentence-similarity
tags:
- gguf
- litembeddings
- modernbert
- colbert
- late-interaction
- pylate
- retrieval
- quantized
---
# LateOn Q4_0 for litembeddings
A compact Q4_0 GGUF conversion of [lightonai/LateOn](https://huggingface.co/lightonai/LateOn), a 149M-parameter ModernBERT ColBERT retrieval model. This speed/size variant is built for [litembeddings](https://github.com/alexandernicholson/litembeddings) and includes the model's complete three-stage PyLate projection folded into one 768-to-128 matrix.
Q4_0 is intended for high-throughput packed indexing and retrieval. It is approximately 40% smaller than the Q8_0 conversion and reached approximately 29.9k tokens/s at packed saturation on an Apple M4 Max, versus approximately 20.9k tokens/s for Q8_0 under the same benchmark. Very short single requests are dominated by dispatch overhead and are not consistently faster; use [Q8_0](https://huggingface.co/embedme/lightonai-lateon-Q8_0) when maximum numerical fidelity matters more than size or packed throughput.
## Files
| File | Purpose | SHA-256 |
|---|---|---|
| `LateOn-Q4_0.gguf` | Q4_0 ModernBERT encoder | `6562738e6ea23e59ad739c7c792afe0e2c64c57158dd65f08109a30bb987f1c6` |
| `LateOn-Q4_0.projection` | Folded 768-to-128 PyLate projection | `3e7fd24683cbcde958448d2d7120d7f12d4d0abcb67bed47e19e7a3170a5fa0a` |
Both files are required. The GGUF encoder alone produces 768-dimensional token vectors; litembeddings applies and normalizes the supplied projection to produce the model's intended 128-dimensional token vectors.
## litembeddings usage
```sql
.load ./litembeddings
SELECT lembed_model(
'/path/to/LateOn-Q4_0.gguf',
json_object(
'colbert_projection', '/path/to/LateOn-Q4_0.projection',
'ctx_size', 300,
'batch_size', 300
)
);
-- Full-precision token vectors.
SELECT lembed_tokens('What is SQLite WAL mode?');
-- Compact int8 token vectors for storage and MaxSim search.
SELECT lembed_tokens_quantized('What is SQLite WAL mode?');
SELECT lembed_maxsim(
lembed_tokens('What is SQLite WAL mode?'),
lembed_tokens('WAL mode lets SQLite readers coexist with one writer.')
);
```
LateOn was trained for queries up to 32 tokens and documents up to 300 tokens. Use the same preprocessing and retrieval conventions as the source model when comparing quality results.
## Validation
An end-to-end parity check against the source FP32 Transformers/PyLate pipeline over 331 output tokens produced per-token cosine similarity `min=0.994162`, `mean=0.998347`.
A 216-document FastAPI retrieval evaluation with 50 queries retained the same measured ranking metrics as exact search with this artifact: MRR `0.8284`, Hits@1/5/10 `74% / 94% / 96%`, and weighted score `92/100`. This is a small validation corpus, not a replacement for task-specific evaluation; Q4 quantization can affect other corpora or near-tied rankings.
## Conversion provenance
- Source: [`lightonai/LateOn`](https://huggingface.co/lightonai/LateOn)
- Source revision: `c01907b70557ee5c7753680d4819a5cce1674b83`
- Converted: 2026-07-14
- llama.cpp revision: `6eddde06a4f25d55d538b5d15628dcc2b6882147`
- Quantization: Q4_0 from the F16 GGUF conversion
- Projection: `W3 @ (W2 + R2) @ (W1 + R1)`, preserving both learned dimension-changing PyLate residuals
## License and attribution
The source model is released under Apache 2.0. See the [LateOn model card](https://huggingface.co/lightonai/LateOn) for training details, evaluation results, intended use, limitations, and citation information. This repository is an independent conversion and is not affiliated with LightOn.
|