jina-reranker-v3-GGUF

GGUF quantizations of jina-reranker-v3 using llama.cpp. A 0.6B parameter multilingual listwise reranker quantized for efficient inference.

Requirements

Python 3.8+
llama.cpp binaries (llama-embedding and llama-tokenize)
Hanxiao's llama.cpp fork recommended: https://github.com/hanxiao/llama.cpp

Installation

pip install numpy safetensors

Files

jina-reranker-v3-BF16.gguf - Quantized model weights (BF16, 1.1GB)
projector.safetensors - MLP projector weights (3MB)
rerank.py - Reranker implementation

Usage

from rerank import GGUFReranker

# Initialize reranker
reranker = GGUFReranker(
    model_path="jina-reranker-v3-BF16.gguf",
    projector_path="projector.safetensors",
    llama_embedding_path="/path/to/llama-embedding"
)

# Rerank documents
query = "What is the capital of France?"
documents = [
    "Paris is the capital and largest city of France.",
    "Berlin is the capital of Germany.",
    "The Eiffel Tower is located in Paris."
]

results = reranker.rerank(query, documents)

for result in results:
    print(f"Score: {result['relevance_score']:.4f}, Doc: {result['document'][:50]}...")

API

`GGUFReranker.rerank(query, documents, top_n=None, return_embeddings=False, instruction=None)`

Arguments:

query (str): Search query
documents (List[str]): Documents to rerank
top_n (int, optional): Return only top N results
return_embeddings (bool): Include embeddings in output
instruction (str, optional): Custom ranking instruction

Returns: List of dicts with keys: index, relevance_score, document, and optionally embedding

Citation

If you find jina-reranker-v3 useful in your research, please cite the original paper:

@misc{wang2025jinarerankerv3lateinteractiondocument,
      title={jina-reranker-v3: Last but Not Late Interaction for Document Reranking},
      author={Feng Wang and Yuqing Li and Han Xiao},
      year={2025},
      eprint={2509.25085},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.25085},
}